Speaker validation using a multitasking speech model

Abstract

The method (400) includes obtaining a speaker identification (SID) model (151) trained to predict speaker embeddings (155) from utterances spoken by different speakers, the SID model including a trained audio encoder (150) and a trained SID head (154). The method also includes receiving a plurality of synthetic speech detection (SSD) training utterances (330) including a set of human spoken speech samples and a set of synthetic speech samples. The method also includes training an SSD head (160) with the SSD training utterances using the trained audio encoder to learn to detect the presence of synthetic speech in audio encodings encoded by the trained audio encoder. The operations also include providing a multitasking neural network model (130) for execution on the computing device (102) to perform both SID and SSD tasks in parallel on the input audio data.

Description

Technical Field

[0001] This disclosure relates to automatic speaker verification using a multitask voice model.

Background Art

[0002] In voice-enabled environments such as homes or cars, users can use voice input to access information and / or control various functions. The information and / or functions may be personalized for a given user. Thus, it may be advantageous to identify a given speaker before enabling user access to virtual assistants and telephone voice interfaces. Speaker identification (e.g., speaker verification and voice authentication) provides an easy way for registered users of a user device to access the user device. By speaker identification, a registered user can unlock and access their device by speaking, without the need to manually enter a passcode (e.g., via typing or speaking) to access the user device. Speaker verification also enables a digital assistant to identify the registered user from spoken utterances without requiring the user to provide authentication credentials. In some cases, an attacker may use a voice synthesis algorithm to create a voice synthesized in the voice of a registered user and trick a speaker identification system into authenticating the synthesized voice to access the user's device or invoke a digital assistant. In additional examples, an attacker may record an audio of a registered user speaking a verification phrase and then play the recorded audio to access the user's device. As a result, systems that receive utterances in an environment need some way to distinguish between human voice utterances directed at the system and synthesized voice utterances.

Summary of the Invention

[0003] One aspect of the present disclosure provides a computer implementation method for causing data processing hardware to perform an operation when executed on data processing hardware. The operation includes obtaining a speaker identification (SID) model trained to predict speaker embeddings from utterances spoken by various speakers. The SID model includes a trained audio encoder and a trained SID head. The operation also includes receiving a plurality of synthetic speech detection (SSD) training utterances, including a set of human-spoken speech samples and a set of synthetic speech samples. The operation also includes training an SSD head with the SSD training utterances using the trained audio encoder to learn to detect the presence of synthetic speech of audio encoding encoded by the trained audio encoder, the SSD head being overlaid on the trained audio encoder. The operation also includes providing a multitask neural network model for performing both SID and SSD tasks in parallel on input audio data for execution on a computing device, wherein the multitask neural network model includes a trained audio encoder, a trained SID head overlaid on the trained audio encoder, and a trained SSD head overlaid on the trained audio encoder.

[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the SID model is trained on multiple SID training datasets, each containing multiple SID training utterances spoken by M different speakers. Here, one or more SID training utterances spoken by a particular speaker may be paired with a corresponding target speaker embedding for training an audio encoder and a SID head to learn how to predict a speaker embedding corresponding to a particular speaker. Furthermore, in these embodiments, the multiple SID training utterances in one or more SID training sets may include N utterances spoken per speaker among M different speakers. Optionally, the multiple SID training utterances in each SID training dataset may include human-generated, non-synthesized speech.

[0005] In some examples, the number of speech samples in a set of human-spoken speech samples is greater than the number of speech samples in a set of synthesized speech samples. In additional examples, the parameters / weights of the trained audio encoder are frozen while the SSD head is being trained with SSD training utterances. The audio encoder may include a multi-head attention model architecture. For example, the multi-head attention model architecture may incorporate a stack of conformer blocks or transformer blocks. The SSD head may include one or more fully connected layers.

[0006] In some embodiments, a multitask neural network model is configured to perform SID and SSD tasks in parallel on input audio data by receiving input audio data as input to a trained audio encoder, generating a shared audio encoding of the input audio data as output from the trained audio encoder, providing the shared audio encoding generated as output from the trained audio encoder as input to trained SID and SSD heads, generating an evaluator speaker embedding representing the voice features of the utterances characterized by the input audio data as output from the trained SID head, and generating an SSD classification score as output from the trained SSD head indicating whether the utterances characterized by the input audio data include human-generated genuine speech or synthesized speech. The audio input data features utterances captured by an array of one or more microphones. In these embodiments, the multitask neural network model may be configured to authorize access to one or more resources when an evaluator speaker embedding matches a reference speaker embedding associated with a registered user authorized to access the resource, and when the SSD classification score generated as output from the trained SSD head indicates that the utterance characterized by the input audio data contains human-generated authentic speech. Additionally or alternatively, in these embodiments, the multitask neural network model may be configured to restrict access to one or more resources when at least one of the evaluator speaker embeddings does not match any reference speaker embedding associated with a registered user authorized to access the resource, or when the SSD classification score generated as output from the trained SSD head indicates that the utterance characterized by the input audio data contains synthesized speech.

[0007] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware that communicates with the data processing hardware and stores executable instructions on the data processing hardware that cause the data processing hardware to perform actions. The operation includes obtaining a speaker identification (SID) model trained to predict speaker embeddings from utterances spoken by various speakers. The SID model includes a trained audio encoder and a trained SID head. The operation also includes receiving a plurality of synthetic speech detection (SSD) training utterances, which include a set of human-spoken speech samples and a set of synthesized speech samples. The operation also includes training an SSD head on the SSD training utterances using the trained audio encoder to learn to detect the presence of synthesized speech of audio encodings encoded by the trained audio encoder, the SSD head being overlaid on the trained audio encoder. The operation also includes providing a multitask neural network model for performing both SID and SSD tasks in parallel on input audio data for execution on a computing device. Here, the multitasking neural network model includes a trained audio encoder, a trained SID head overlaid on the trained audio encoder, and a trained SSD head overlaid on the trained audio encoder.

[0008] This aspect of the Disclosure may include one or more of the following optional features. In some embodiments, the SID model is trained on multiple SID training datasets, each containing multiple SID training utterances spoken by M different speakers. Here, one or more SID training utterances spoken by a particular speaker may be paired with a corresponding target speaker embedding for training an audio encoder and a SID head to learn how to predict speaker embeddings corresponding to a particular speaker. Furthermore, in these embodiments, the multiple SID training utterances in one or more SID training sets may include N utterances spoken per speaker among M different speakers. Optionally, the multiple SID training utterances in each SID training dataset may include human-generated, non-synthesized speech.

[0009] In some examples, the number of speech samples in a set of human-spoken speech samples is greater than the number of speech samples in a set of synthesized speech samples. In additional examples, the parameters / weights of the trained audio encoder are frozen while the SSD head is being trained with SSD training utterances. The audio encoder may include a multi-head attention model architecture. For example, the multi-head attention model architecture may incorporate a stack of conformer blocks or transformer blocks. The SSD head may include one or more fully connected layers.

[0010] In some embodiments, a multitask neural network model is configured to perform SID and SSD tasks in parallel on input audio data by receiving input audio data as input to a trained audio encoder, generating a shared audio encoding of the input audio data as output from the trained audio encoder, providing the shared audio encoding generated as output from the trained audio encoder as input to trained SID and SSD heads, generating an evaluator speaker embedding representing the voice features of the utterances characterized by the input audio data as output from the trained SID head, and generating an SSD classification score as output from the trained SSD head indicating whether the utterances characterized by the input audio data include human-generated genuine speech or synthesized speech. The audio input data features utterances captured by an array of one or more microphones. In these embodiments, the multitask neural network model may be configured to authorize access to one or more resources when an evaluator speaker embedding matches a reference speaker embedding associated with a registered user authorized to access the resource, and when the SSD classification score generated as output from the trained SSD head indicates that the utterance characterized by the input audio data contains human-generated authentic speech. Additionally or alternatively, in these embodiments, the multitask neural network model may be configured to restrict access to one or more resources when at least one of the evaluator speaker embeddings does not match any reference speaker embedding associated with a registered user authorized to access the resource, or when the SSD classification score generated as output from the trained SSD head indicates that the utterance characterized by the input audio data contains synthesized speech.

[0011] Details of one or more embodiments of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings and the claims. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of an exemplary speech environment having speaker identification and a synthesized speech detection system for speaker identification. [Figure 2] This is a schematic diagram of an example of a conformer block that can form a conformer layer stack implemented by an encoder in a speaker identification and synthesized speech detection system. [Figure 3A] This is a schematic diagram of the first training stage, in which an untrained audio encoder and speaker identification (SID) head are trained on the SID task to learn how to predict speaker embeddings / representations from input audio. [Figure 3B] This is a schematic diagram of the second training phase, in which a Synthetic Speech Detection (SSD) head is trained to learn how to predict whether the input audio corresponds to genuine human-generated speech or synthesized / synthesized speech. [Figure 4] This is an exemplary flowchart of the operation configuration for training a multitask neural network model to perform both SID and SSD tasks in parallel on input audio data. [Figure 5] This is a schematic diagram of an exemplary computing device that may be used to implement the systems and methods described herein. [Modes for carrying out the invention]

[0013] Similar reference symbols in various drawings refer to the same elements.

[0014] In voice-enabled environments such as homes, cars, workplaces, or schools, users may speak queries or commands, and a digital assistant may respond to those queries and / or execute those commands. Such voice-enabled environments can be implemented using a network of connected microphone devices distributed across various rooms or areas of the environment. Users can query the digital assistant through spoken utterances via the network of microphones, without needing a computer or other interface in front of them. In some cases, a voice-enabled environment is associated with multiple registered users, for example, people living in a household. These cases can occur when a single device is shared by multiple users, such as a smartphone, smart speaker, smart display, tablet device, smart TV, smart appliance, or vehicle infotainment system. Therefore, it is desirable to determine the identity of the specific user speaking the query. The process of determining the identity of a specific speaker / user may be called speaker verification, speaker recognition, speaker identification, or voice recognition.

[0015] Speaker verification / identification can enable users to issue queries that function on behalf of a specific user and / or trigger personalized responses in a multi-user environment. Speaker verification / identification (e.g., voice authentication) provides a simple way for a user of a user device to gain access to it. For example, a user may unlock and access a user device by speaking an utterance, without the user having to manually enter a passcode (e.g., by typing or speaking) to gain access to the user device.

[0016] In some scenarios, a user queries a digital assistant that requires access to resources from a set of personal resources related to and / or associated with the user's personal information. For example, a particular user (e.g., a user registered with a digital assistant) might ask the digital assistant, "When is my meeting with Matt?" or query the digital assistant, "Play my music playlist." Here, the user may be one of one or more registered users, each with permission to access their own set of personal resources (e.g., calendar, music player, email, messaging, contact list, etc.) and restricted from accessing the personal resources of other registered users. For example, if John is a registered user of the digital assistant, the digital assistant needs to access John's calendar for the appropriate registered user to determine when his meeting with Matt is scheduled and distinguish whether John said the utterance, "When is my meeting with Matt?" in order to respond with meeting details for the scheduled meeting with Matt. Similarly, the digital assistant needs to distinguish whether John said "Play my music playlist" in order to access the music player and ultimately output tracks from John's music playlist audibly.

[0017] To determine which user is speaking in a multi-user voice-enabled environment, a voice-enabled system may include a speaker identification system (e.g., a speaker verification system or voice authentication system). An automated speaker verification (ASV) system may employ a speaker identification (SID) model trained to extract evaluation vectors (e.g., speaker embeddings) from audio data corresponding to utterances spoken by a particular user, so that the extracted evaluation vectors represent the voice characteristics of that particular user. The evaluation vectors may include d-vectors. The SID model may be a neural network model trained under machine or human supervision to output d-vectors. To resolve the identity of a particular user, a comparator determines whether the evaluation vectors match any reference vectors of any registered and / or authorized users of the user device, where each reference vector may correspond to a voiceprint or unique identifier representing the voice characteristics of the respective registered / authorized user. Each registered and / or approved user may speak multiple registration utterances, and for each registration phrase, the SID model may generate a corresponding criterion d vector, which may be combined, for example, averaged, or otherwise accumulated to form the respective criterion d vector for the registered / approved user.

[0018] With the advent of synthetic media (e.g., “deepfakes”), it is crucial for these systems to accurately determine when speech utterances include synthesized speech (i.e., computer-generated audio output that resembles human speech). For example, state-of-the-art text-to-speech (TTS) and voice-to-speech (VC) systems can now closely mimic human speakers, providing a means to attack and impersonate ASV systems. Thus, state-of-the-art TTS and VC systems can be used to generate synthesized / synthesized speech representations in the voice of an authorized / registered user, impersonating the SID model to verify that the synthesized speech representation was spoken by an authorized / registered user. Where used herein, the terms “synthesized speech,” “synthesized voice,” “synthesized utterance,” and “synthesized utterance” are used interchangeably. Imposters may also impersonate ASV systems by playing pre-recorded voice audio of a registered user to gain unauthorized access to the registered user’s locked user device and / or resources associated with the registered user. As a result, the user device's microphone can capture synthesized or played audio of the registered user's pre-recorded voice, and the ASV system can extract evaluation vectors that match the registered user's reference vector, thus allowing the ASV system to be impersonated and enabling an imposter to gain unauthorized access to the user device.

[0019] To detect these attempts at impersonation, ASV systems may include one or more countermeasures for Synthetic Speech Detection (SSD) and / or Replay Attack Detection (RAD). Since SID models are typically neural network-based, traditional anti-impersonation systems for SSD and RAD are also typically based on neural network models trained via machine learning techniques. Thus, employing additional countermeasures in conjunction with SID models can significantly increase computation / storage requirements, as it requires running multiple separate neural network models in parallel, i.e., separate neural network models for SID, SSD, and RAD on the input audio signal. Minimizing latency is critical to ensuring a positive user experience, as the primary use case for ASV systems is authenticating users "in real time" to launch a virtual assistant and / or gain access to the phone's voice interface. Furthermore, since ASV system deployments typically occur on resource-constrained systems (e.g., smartphones), minimizing the use of computation and storage resources is crucial.

[0020] Machine learning (ML) algorithms, such as neural networks, are the primary driving force behind the proliferation of ASV systems and other speech recognition technologies. However, these algorithms traditionally require a large number of training samples, and consequently, the lack of sufficiently large and high-quality datasets is often a major bottleneck in training accurate models. For example, large datasets containing human speech are readily available, but similar datasets containing synthesized speech are not. Therefore, training models that can accurately identify synthesized speech without conventional training sets presents a significant challenge in the development of synthesized speech detection systems.

[0021] Embodiments of this specification relate to multitask neural network models that can perform both speaker identification and synthesized speech detection tasks simultaneously without requiring separate neural network models for performing SID and SSD tasks. In some embodiments, the multitask neural network model performs replay attack detection (RAD) in addition to, or instead of, the SSD task. In particular, the multitask neural network model includes, as input, an audio encoder that receives an input audio signal characterizing an utterance and, as output, generates a shared output embedding for use in at least one of the SID head and SSD head or RAD head. Here, each of the SID head, SSD head and RAD head may be superimposed on the audio encoder and may be configured to compute outputs associated with the SID task, SSD task and RAD task, respectively. For example, the SID head may include a fully connected network that receives the shared output embedding generated by the audio encoder and computes a speaker embedding that characterizes the voice characteristics of the speaker of the utterance. Thus, the audio encoder and SID head can collectively provide an SID model. The SSD head also receives the same shared output embedding generated by the audio encoder, but instead may include a fully connected network that computes a synthesized speech score indicating whether the input audio signal corresponds to genuine speech (e.g., unsynthesized / real / human speech) or synthesized speech. Thus, the audio encoder and SSD head can collectively provide an SSD model. Where applicable, the RAD head may include a fully connected network that also receives the same shared output embedding generated by the audio encoder and computes a RAD score indicating whether the input audio signal corresponds to pre-recorded speech played from an audio output device or unrecorded speech audibly spoken by a human. Thus, the audio encoder and SSD head can collectively provide a RAD model.

[0022] As will be described in more detail below, the audio encoder and the SID head may be initially trained on SID training utterances spoken by different speakers to learn a method for predicting speaker embeddings for input audio data. Thereafter, the SSD head and / or the RAD head may be trained to learn a method for predicting the aforementioned synthetic voice and / or RAD score using the trained audio encoder. The initial training process may first train a multi-task neural network model on the SID task to teach the multi-task neural network model to learn to generate speaker embeddings from SID training utterances spoken by different speakers.

[0023] Referring to FIG. 1, in some embodiments, an exemplary system 100 includes a user device 102 associated with one or more users 10 and communicating with a remote system 111 via a network 104. The user device 102 may correspond to a computing device such as a mobile phone, a computer (laptop or desktop), a tablet, a smart speaker / display, a smart appliance, smart headphones, a wearable, a vehicle infotainment system, etc., and includes data processing hardware 103 and memory hardware 107. The user device 102 may include one or more microphones 106 for capturing utterances from each user 10 or may communicate with one or more microphones 106. The remote system 111 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources 113 (e.g., data processing hardware) and / or storage resources 115 (e.g., memory hardware).

[0024] The user device 102 may include a hotword detector (not shown) configured to detect the presence of hotwords in the streaming audio 118 without performing semantic analysis or speech recognition processing on the streaming audio 118. The user device 102 may also include an acoustic feature extractor (not shown) which may be implemented as part of the hotword detector or as a separate component for extracting audio data 120 from an utterance 119. For example, referring to Figure 1, the acoustic feature extractor may receive streaming audio 118 captured by one or more microphones 106 of the user device 102, corresponding to an utterance 119 spoken by user 10, and extract audio data 120. The audio data 120 may include acoustic features such as Mel-frequency sepstrum coefficients (MFCCs) or filter bank energy calculated across a window of the audio signal. In the illustrated example, the utterance 119 spoken by user 10 includes "Ok Google, play my music playlist."

[0025] The hotword detector may receive audio data 120 and determine whether the utterance 119 includes a specific hotword (e.g., Ok Google) spoken by the user 10. That is, the hotword detector 110 detects the presence of a hotword (e.g., Ok Google) in the audio data 120 or one or more variations of the hotword (e.g., Hey Google), wakes up the user device 102 from a sleep state or a standby state, triggers the automatic speech recognition (ASR) system 180, and may be trained to perform speech recognition on the hotword and / or one or more other terms, following the hotword, for example, followed by a voice query specifying an action to be performed. In the illustrated example, the query following the hotword of the captured utterance 119 in the streaming audio includes "Play my music playlist" which specifies an action for the digital assistant to access a music playlist associated with a specific user (e.g., John) 10, and provides a response 190 including an audio track from John's music playlist to the user device 102 (and / or one or more selected audio output devices) to reproduce audible output from the speaker.

[0026] System 100 includes a multitask neural network model 130, which is configured to process audio data 120 to determine the identity of user 10 speaking utterance 119 and to determine whether the audio data 120 contains disguised speech. As used herein, the multitask neural network model 130 may include detecting that the audio data 120 contains synthesized speech and / or optionally, detecting that the audio data 120 contains pre-recorded speech played from an audio output device, thereby determining whether the audio data 120 contains disguised speech. Advantageously, the multitask neural network model 130 includes a shared audio encoder 150 that receives the audio data 120 as input and produces, as output, an audio encoding 152 corresponding to a hidden feature representation of the audio data 120 encoded by an audio encoder 150. Each audio encoding 152 output from the audio encoder 150 can be input to the SID head 154, SSD head 160, and RAD head 161, which branch off from the audio encoder 150. Thus, the audio encoder 150 can function as the "main body" of a multitasking neural network model 130, where the audio encoder 150 and the SID head 154 form an SID branch corresponding to the SID model 151 (Figure 3A), the audio encoder 150 and the SSD head 160 form an SSD branch corresponding to the SSD model 153 (Figure 3B), and the audio encoder 150 and the RAD head 161 form a RAD branch corresponding to the RAD model.

[0027] The multitasking neural network model 130 may run on a computing device including data processing hardware 103 of a user device 102 associated with user 10, or it may run on data processing hardware 113 of a remote system 111. When running on data processing hardware 113 of the remote system 111, user device 102 may transmit audio data 120 corresponding to utterances 119 captured by user device 102 to the multitasking neural network model 130 running on the remote system 111 via the network 104. In some embodiments, the components of the multitasking neural network model 130 are distributed across data processing hardware 103 of user device 102 and data processing hardware 113 of the remote system 111 for execution. For example, the audio encoder 150 may run on either the user device 102 or the remote system 111, while the SID head, SSD head, and RAD heads 154, 160, and 161 may run on either the user device 102 or the remote system 111. Such a configuration may be beneficial when processing / storage / memory resources are limited on the user device 102. The multitasking neural network model 130 may employ all three of the SID head, SSD head, and RAD heads 154, 160, and 161, or any combination of two of the heads. For example, the multitasking neural network model 130 may employ the SID head 154 and the SSD head 160, but not the RAD head 161.

[0028] In some embodiments, the audio encoder 150 includes a multi-head attention model architecture. For example, the audio encoder 150 may include a stack of conformer blocks. The multi-head attention model architecture of the audio encoder 150 does not have to be limited to conformer blocks and may include, but is not limited to, any type of multi-head attention model architecture including a stack of transformer blocks or lightweight convolutional blocks.

[0029] Figure 2 provides an example of a conformer block 200 from the conformer layer stack of encoder 150. The conformer block 200 includes a first half-feedforward layer 210, a second half-feedforward layer 240, and a concatenation operator 205, with a multi-head self-attention block 220 and a convolutional layer 230 positioned between the first half-feedforward layer 210 and the second half-feedforward layer 240. The first half-feedforward layer 210 processes audio features 121 extracted from input audio data 120. For example, an acoustic front-end (not shown) may extract audio features 121 from input audio data 120 for input to the audio encoder. The audio features 121 may include a logmel filter bank calculated and stacked every four frames having three strides. Subsequently, the multi-head self-attention block 220 receives the input audio data 120 concatenated with the output of the first half-feedforward layer 210. In particular, the multi-head self-attention block 220 integrates a relative sinusoidal position coding scheme that enables the self-attention block 220 to generalize better to different input lengths, making the resulting encoder 150 more robust to variations in the length of the input audio data 120 characterizing the utterance 119. Intuitively, the role of the multi-head self-attention block 220 is to separately summarize the acoustic context for each input frame being encoded. The convolutional layer 230 subsamples the output of the multi-head self-attention block 220, which is concatenated with the output of the first half-feedforward layer 210. Subsequently, the second half-feedforward layer 240 receives the concatenation of the output of the convolutional layer 230 and the multi-head self-attention block 220. The Layernorm module 250 processes the output from the second half-feedforward layer 240. Mathematically, the conformer block 200 generates the output feature y by transforming the input feature x using the modulation feature m, as follows:

number

[0030] Referring back to Figure 1, in some examples, the multitasking neural network model 130 is deployed in a multi-user voice-enabled environment where multiple different users 10 are each registered on a user device 102 and have permission to access their respective sets of personal resources associated with them (e.g., calendar, music player, email, messaging, contact list, etc.). Registered users 10 are restricted from accessing personal resources from the respective sets of personal resources associated with other registered users. Each registered user 10 may have their own user profile linked to each set of personal resources associated with them, as well as other relevant information associated with that user 10 (e.g., user-specified preference settings).

[0031] The audio encoder 150 and the SID head 154 may be initially trained on the SID task to determine each evaluation speaker embedding / vector 155 for the utterance 119 that conveys the voice characteristics of user 10. The reference speaker embedding / vector 155 represents the voice characteristics of the utterance 119 captured by the user device 102. The verifier 170 may then compare the speaker embedding 155 to one or more reference speaker embeddings 171 associated with registered / authorized users who have permission to access each set of personal resources associated with that user (e.g., calendar, music player, email, messaging, contact list). The evaluation and reference speaker embedding / vector 155 may include a d vector or an i vector. When the speaker embedding / vector 155 matches one of the reference speaker embeddings 171, the verifier 170 may verify the identity of user 10 who spoke the utterance 119 as one of the registered / authorized users. Advantageously, SID / ASV allows user 10 to unlock and access user device 102 by speaking an utterance, without having to manually enter (e.g., via typing) or speak a passcode to gain access to user device 102, or provide any other means of verification (e.g., answering a challenge question, providing biometric verification data, etc.). Similarly, in a multi-user voice-enabled environment, SID / ASV performed by the audio encoder 150, SID head 154, and verifier 170 may be used to determine which registered user 10 is speaking the utterance 119. For example, in Figure 1, John may be a registered user 10 of user device 102 (or an interface for a digital assistant 101 running on the user device), and the digital assistant 101 needs to distinguish whether John spoke the utterance 119, "Ok Google, play my music list," in order to access the music player and ultimately output a track audibly from the appropriate music playlist associated with John.

[0032] The audio encoder 150, initially trained in the SID task, may then be used to train the SSD head 160 in the SSD task to determine whether the input audio data 120 characterizing the utterance 119 contains synthesized speech. For example, the utterance 119 captured by the user device 102 of the streaming audio 118 may correspond to synthesized speech generated by a text-to-speech (TTS) system 11 or a voice conversion (VC) model 13. Here, the TTS system 11 may convert the input text corresponding to the utterance 119 into synthesized speech having the voice characteristics of an authorized / registered user 10 who has permission to access the user device 102. Similarly, the VC model 13 may receive audio of the utterance 119 spoken by a false user not authorized to access the user device 102 and convert the audio into converted audio having the voice characteristics of an authorized / registered user 10. Therefore, a false user / entity may generate a speaker embedding of the authorized / registered user 10 by forging the SID head 154, employing the TTS system 11 and / or VC model 13 to generate a synthesized speech with the voice characteristics of the authorized / registered user 10, thereby gaining unauthorized access to the user device 102 and / or the set of personal resources associated with the authorized / registered user 10. As a countermeasure to prevent forgery of the verifier 170, the SSD head 160 of the multitask neural network model 130 is configured to receive, as input, an audio encoding 152 produced as output by the audio encoder 150, and as output, an SSD classification score 162 indicating whether the audio data 120 characterizing the utterance 119 contains human-generated genuine speech or synthesized / forged speech. In some examples, the SSD classification score 162 includes a probability score indicating the presence of synthesized speech when the probability score 162 satisfies the SSD threshold score. In these examples, the disguised speech detector 172 may compare the probability score transmitted by the SSD classification score 162 with the SSD threshold score to determine whether the utterance 119 is genuine speech or synthesized / disguised speech.In other examples, an SSD classification score of 162 includes a binary / boolean value that indicates the presence of genuine human-generated speech when the value is equal to "0", or synthesized speech when the value is equal to "1", and vice versa.

[0033] Similar to the SSD head 160, the RAD head 161 may be trained in the RAD task using the audio encoder 150, which was initially trained in the SID task, to determine whether the input audio data 120 characterizing the utterance 119 includes pre-recorded voice played from an audio output device. As previously mentioned, the registered user's pre-recorded voice may be played from the audio output device, and the microphone of the user device 102 may capture the played audio of the registered user's pre-recorded voice. The SID head 154 then extracts an evaluation vector 155, which the verifier 170 validates as a match to a reference vector 171 for the registered user, thereby potentially allowing an imposter to gain unauthorized access to the user device by impersonating the verifier 170. As a measure to prevent spoofing of the verifier 170, the RAD head 161 of the multitask neural network model 130 is configured to receive, as input, audio encoding 152 generated as output by the audio encoder 150, and as output, generate a RAD classification score 163 indicating whether the audio data 120 characterizing the utterance 119 includes genuine speech associated with unrecorded speech audibly spoken by a human, or spoofed speech associated with pre-recorded speech played back from an audio output device. In some examples, the RAD classification score 163 includes a probability score indicating the presence of pre-recorded speech when the probability score 163 satisfies a RAD threshold score. In these examples, the spoofed speech detector 172 may compare the probability score conveyed by the RAD classification score 163 with a RAD threshold score to determine an output indicating whether the utterance 119 is genuine speech or pre-recorded / spoofed speech. In other examples, a RAD classification score of 163 includes a binary / boolean value that, when equal to "0", indicates the presence of genuine human-generated speech that was not recorded, or when equal to "1", indicates the presence of pre-recorded speech, and vice versa.

[0034] In some embodiments, the multitasking neural network model 130 is configured to restrict access to one or more resources when the verifier 170 determines that the evaluator speaker embedding 155 does not match any of the reference speaker embeddings 171 associated with registered users authorized to access the resources, or when the spoofed speech detector 172 determines that the SSD classification score 162 and / or the RAD classification score 163 include synthesized and / or pre-recorded speech. In these embodiments, the multitasking neural network model 130 prevents the ASr system 180 from performing speech recognition on the audio data 120, or, if the ASR system 180 has already started performing speech recognition, it suppresses speech recognition performed on the audio data 120.

[0035] In some additional embodiments, the multitask neural network model 130 is configured to authorize access to one or more resources when the Validator 170 determines that the evaluator speaker embedding 155 matches a reference speaker embedding 171 associated with a registered user authorized to access the resource, and when the spoofed voice detector 172 determines that the SSD classification score 162 and / or RAD classification score 163 contain genuine human-generated speech. In these embodiments, the Validator 170 and the spoofed voice detector 172 may collectively output a Confirmation 175, which, when received by the ASR system 180, causes the ASR system 180 to perform speech recognition on the audio data 120 characterizing the utterance 119. If the ASR system 180 has already started performing speech recognition, the Confirmation 175 causes the ASR system 180 to continue processing the audio data 120 to recognize the terms spoken by user 10 in the utterance 119.

[0036] The ASR system 180 may include an ASR model, which is configured to perform speech recognition on audio data 120 characterizing a query, for example, "Play my music playlist." The ASR system 180 may also include a natural language understanding (NLU) module configured to perform query interpretation on the speech recognition results output by the ASR model. Generally, the NLU module may perform semantic analysis on the speech recognition results to identify the action to be performed, as specified by the query. In the illustrated example, the NLU module may determine that performing the action specified by the query "Play my music playlist" requires access to each set of personal resources associated with each registered user 10 of the user device 102. Therefore, the NLU module determines that the action specified by the query lacks a necessary parameter to perform the action, namely the user's identity. Therefore, the NLU module may use the identity of a specific registered user (e.g., John) 10, verified by the verifier 170, to determine which resources to access in order to perform an action, and thus initiate the execution of the query by providing an output command 185 to perform the action specified by the query. In the illustrated example, the output command 185 may instruct a music streaming service to stream a music track from the music playlist of the registered user John. The digital assistant 101 may provide a response to the query that includes the music track for audible output from the user device 102 and / or one or more other devices communicating with the user device 102.

[0037] Figures 3A and 3B illustrate an exemplary training process 300 for training a multitask neural network model 130 to perform both speaker identification (SID) and synthesized speech detection (SSD) tasks simultaneously, without the need to employ separate neural network models for the SID and SSD tasks. The training process 300 can be performed on the remote system 111 shown in Figure 1. The training process 300 includes a first training stage 300a (Figure 3A) in which an untrained audio encoder 150 and a SID head 154 superimposed on the untrained audio encoder 150 learn how to predict speaker embeddings / representations 155 from input audio, which have been trained on the SID task. After the first training stage 300a is completed and the audio encoder 150 and SID head 154 are fully trained to perform an SID task (e.g., automatic speaker verification), the training process 300 includes a second training stage 300b (Figure 3B) in which the SSD head 160 is trained to learn how to predict whether the input audio corresponds to a human-generated authentic speech or a synthesized / synthesized speech, using the audio encoder 150 trained on the SID task by the first training stage 300a.

[0038] Referring to Figure 3A, the first training stage 300a (i.e., the SID training stage) retrieves one or more SID training datasets 310 stored in the data storage 301 and trains an untrained audio encoder 150 and SID head 154 with the training datasets 310. Once trained by the first training stage 300a, the audio encoder 150 and SID head 154 can collectively form an SID model 151. The data storage 301 may reside on the memory hardware 115 of the remote system 111. Each training dataset 310 contains multiple training utterances 320, 320a~n spoken by M different speakers, allowing the training dataset 310 to contain multiple negative training utterances 320 from different speakers in a given training batch. Furthermore, each training dataset 310 may contain N utterances 320 per speaker among the M different speakers, allowing the training dataset 310 to also contain positive training utterances 320 from the same speaker in a given training batch. Each corresponding training utterance 320 may include a text-dependent portion 321 and / or a text-independent portion 322. The text-dependent portion 321 includes an audio segment that characterizes a given word spoken in the training utterance 320 (e.g., "Hey Google") or a variation of a given hotword (e.g., "Ok Google"). In additional embodiments, the text-dependent portion 321 of some training utterances 320 may include an audio segment that characterizes other terms / phrases instead of the given word or a variation of it, such as a custom hotword, a verification phrase (e.g., "This is Meg"), or a commonly used voice command (e.g., play, pause, volume up / down, call, message, navigate / direct, etc.). The text-dependent portion 321 is optional, so that only a portion of the training utterance 320 may include a text-dependent portion, or none of the training utterances 320 may include a text-dependent portion 321.

[0039] In some examples, the text-independent portion 322 of each training utterance 320 includes an audio segment that characterizes the query statement spoken in the training utterance 320, following a predetermined word characterized by a text-dependent portion 321. For example, a corresponding training utterance 320 may include "Ok Google, what's the weather like outside?", thereby including a text-dependent portion 321 that characterizes a predetermined "Ok Google" and a text-independent portion 322 that characterizes the query statement "What's the weather like outside?". The text-dependent portion 321 within each training utterance 320 is phonetically constrained by the same predetermined word or its variation, but the lexicon of the query statement characterized by each text-independent portion 322 is not constrained, such that the duration and phonemes associated with each query statement are variable.

[0040] Continuing to refer to Figure 3A, the first stage 300a of the training process 300 trains an untrained audio encoder 150 on training utterances 320, 320a-n, to generate audio encodings 152 for the training utterances 320 as output from the audio encoder 150. That is, the audio encoder 150 is configured to encode the audio data characterizing the training utterances into a hidden feature representation (e.g., a series of encodings / vectors 152). The SID head 154 receives the audio encodings 152 output from the audio encoder 150 as input and generates a speaker embedding 155 for each utterance 320 as output. In some examples, the speaker embedding 155 output by the SID head 154 for each utterance includes a d vector. However, the first training stage 300a may also train the audio encoder 150 and the SID head 154 on the SID task to generate other types of speaker embeddings 155, such as an i vector. During training, additional information about each utterance 320 may be provided as input to a SID model 151, which consists of an audio encoder 150 and a SID head 154 superimposed on the audio encoder 150. For example, SID targets 324, such as SID target vectors / speaker embeddings, corresponding to ground truth output labels, may be provided as input to the neural network during training with utterances 320, for training the SID model 151 to learn how to predict speaker embeddings 155 for a particular speaker. Thus, one or more utterances 320 from each particular speaker may be paired with a particular SID target vector 324.

[0041] As described above, the audio encoder 150 includes a multi-head attention model architecture that may incorporate a stack of conformer blocks. Optionally, but not limited to, other types of multi-head attention model architectures, such as transformer blocks or lightweight convolutional blocks, may be employed in the audio encoder 150 instead of conformer blocks. The SID head 154 stacked / superimposed on the audio encoder 150 may include one or more fully connected network layers selected to learn to generate speaker-related information specific to the speaker embedding 155.

[0042] Training an untrained audio encoder 150 and SID head 154 can be initiated by using a pairwise training technique, where a sequence of training utterances 320, each paired with a specific SID target vector 324, is input to the audio encoder 150, and the corresponding audio encoding 152 generated by the audio encoder 150 is processed by the SID head 154 to generate speaker embeddings 155 for each of the first and second training utterances 320. Next, the SID loss module 156 can compare the respective speaker embeddings 155 to determine whether the two speaker embeddings 155 were derived from training utterances 320 spoken by the same speaker. That is, the SID loss module 156 can calculate a SID loss 158 indicating the likelihood that the two speaker embeddings 155 are identical. The SID loss 158 output from the SID loss module 156 provides an indication of whether the training utterances of the first and second training utterances 320 were spoken by the same speaker. In one example, the SID loss 158 may simply contain a binary value of "0" or "1", where "0" indicates that the utterances were spoken by different speakers, and "1" indicates that the utterances were spoken by the same speaker. The parameters of the audio encoder 150 and the SID head 154 can then be adjusted based on the SID loss 158. Multiple sets of paired training utterances 320 can be processed in this manner.

[0043] Referring to Figure 3B, the second training stage 300b (i.e., the SSD training stage) receives / acquires the SID model 151 trained by the first training stage 300a and trains the SSD head 160 to learn how to predict whether the input audio corresponds to human-generated authentic speech or synthesized / conjugated speech using the trained audio encoder 150. During the second training stage 300b, the parameters / weights of the trained audio encoder 150 are fixed and the SSD head 160 is trained (or fine-tuned) to learn how to project an SSD classification score 162 indicating whether the input audio contains human-generated authentic speech or synthesized speech. As will become clear, the SSD head 160 receives, as input, the audio encoding 152 produced by the trained audio encoder 150 for the SSD training utterances 330 and, as output, generates the respective SSD classification score 162 for each SSD training utterance 330. In some examples, the SSD classification score includes a probability score of 162 indicating the presence of synthesized speech when the probability score of 162 satisfies the SSD threshold. In other examples, the SSD classification score includes a binary / boolean value that, when equal to "0", indicates genuine human-generated speech, or when equal to "1", indicates synthesized speech, and vice versa.

[0044] Similar to the SID head 154 employed in the SID model 151, the SSD head 160, stacked / superimposed on the trained audio encoder 150, may include one or more fully connected network layers. However, in contrast to the fully connected network layers employed in the SID head 154, the fully connected network layers of the SSD head 160 are trained / selected to learn to project the SSD classification score 162. In some examples, a second training stage 300b trains one or more fully connected network layers of the SSD head 160 from scratch using SSD training utterances 330. In other examples, one or more fully connected network layers of the SSD head 160 include the same fully connected network layers of the SID head 154 trained by the first training stage 300a, and the second training stage 300b fine-tunes one or more fully connected network layers to learn how to project speaker embeddings 155 into the SSD classification score 162 using SSD training utterances 330 for use by the SSD head 160. In particular, regardless of whether the SSD head 160 is fine-tuned or trained from scratch during the second training phase 300b, the parameters / weights of the trained audio encoder 150 remain fixed while the SSD head 160 is trained on the SSD task using the SSD training utterances 330.

[0045] The second training stage 300b receives / acquires multiple SSD training utterances 330 stored in the data storage 301 and uses the trained audio encoder 150 to train the SSD head 160 with the SSD training utterances 330. Once trained by the second training stage 300b, the audio encoder 150 and the SSD head 160 can collectively form an SSD model 153. The data storage 301 may reside on the memory hardware 115 of the remote system 111. The SSD training utterances 330 may include a set of human-spoken voice samples 330a and a set of synthesized voice samples 330b. In some examples, the number of voice samples in the set of human-spoken voice samples 330a is greater than the number of voice samples in the set of synthesized voice samples 330b. In other examples, the number of voice samples in the set of human-spoken voice samples 330a is less than the number of voice samples in the set of synthesized voice samples 330b. Each human-spoken voice sample 330a corresponds to a real / human-spoken non-synthesized speech utterance, and each synthesized voice sample 330b corresponds to a machine-generated / synthesized speech. That is, one or more synthesized voice samples 330b may include synthesized speech generated by a text-to-speech (TTS) system from the corresponding input text. Additionally or alternatively, one or more synthesized voice samples 330b may include synthesized speech generated by a speech conversion model from the corresponding input utterance having different vocal characteristics from the generated synthesized speech.

[0046] In some embodiments, a second training stage 300b trains the SSD head 160 using supervised learning, where each SSD training utterance 330 is paired with a corresponding ground truth target 165 indicating whether the SSD training utterance 330 contains a human-generated speech sample 330a or a synthesized speech sample 330b. Here, an SSD loss module 168 may apply a loss function to the classification score 162 predicted by the SSD head 160 to generate a corresponding SSD loss 169 based on the ground truth target 165. In some examples, the loss function includes a softmax cross-entropy loss, where a softmax layer applies softmax to the classification score 162 (e.g., a Boolean value) output from the SSD head 160, which is fed into the cross-entropy loss function to generate the SSD loss 169.

[0047] In other embodiments, the second training stage 300b trains the SSD head 160 using self-supervised learning (also called “unsupervised learning”), where the labels are a natural part of the training utterances 330 and do not include separate external ground truth labels; that is, each human-generated speech sample 330a does not include a separate external ground truth label indicating that the sample contains genuine speech, and each synthesized speech sample 330b does not include a separate external ground truth label indicating that the sample contains synthesized speech. Here, the human-generated speech samples 330a may correspond to positive training samples, and the synthesized speech samples 330b may correspond to negative training samples in which the SSD model 153 learns to look for patterns without existing annotated labels and with minimal human supervision.

[0048] A third training stage may similarly train the RAD head 161 using the trained audio encoder 150, which was trained during the first training stage 300a in Figure 3A. The third training stage trains the RAD head 161 to learn how to predict whether the input audio corresponds to genuine, human-generated speech that is not pre-recorded or pre-recorded speech played from an audio output device. Similar to the second training stage 300b, the parameters / weights of the trained audio encoder 150 are fixed, and the RAD head 161 is trained (or fine-tuned) to learn how to project a RAD classification score 163 indicating whether the input audio contains genuine speech or disguised / pre-recorded speech. The RAD classification score 163 may include a probability distribution indicating the likelihood that the audio data contains pre-recorded speech, or a Boolean / binary value indicating the presence of genuine, human-generated speech that is not pre-recorded when the value is equal to "0", or the presence of pre-recorded speech when the value is equal to "1", or vice versa.

[0049] In some examples, the SSD head 160 is also trained to detect the presence of pre-recorded speech in the input audio, thereby reducing the need for a separate RAD head 161. Thus, the SSD head 160 can be trained to output a classification score indicating the input audio containing either genuine or disguised speech when either synthesized speech or pre-recorded speech is detected.

[0050] Figure 4 provides an exemplary flowchart of the configuration of an exemplary operation of Method 400 for training a multitask neural network model 130 to perform both SID and SSD tasks in parallel on input audio data. Method 400 may be executed on data processing hardware 113 of a remote system 111 and / or on data processing hardware 103 of a user device 102. In operation 402, Method includes obtaining a Speaker Identification (SID) model 151 trained to predict speaker embeddings 155 from utterances spoken by different speakers. The SID model 151 includes a trained audio encoder 150 and a trained SID head 154. The trained audio encoder 150 may include a multi-head attention model architecture. For example, the multi-head attention model architecture may incorporate a stack of conformer blocks 200. In other configurations, the multi-head attention model architecture may incorporate a stack of transformer blocks or a stack of lightweight convolutional blocks.

[0051] The acquired SID model 151 can be trained on multiple SID training datasets 320, each containing multiple SID training utterances spoken by M different speakers. Here, one or more SID training utterances spoken by a particular speaker may be paired with a corresponding target speaker embedding 324 for training the audio encoder 150 and SID head 154 to learn how to predict the corresponding speaker embedding 155 for that particular speaker. Additionally or alternatively, one or more SID training utterances in one or more of the SID training sets 320 may contain N utterances spoken by each of the M different speakers. Furthermore, the multiple SID training utterances in each SID training set 320 may contain non-synthesized human speech.

[0052] In operation 404, method 400 includes receiving a plurality of SSD training utterances 330, where the SSD training utterances 330 include a set of human-spoken voice samples 330a and a set of synthesized voice samples 330b. In some examples, the number of voice samples in the set of human-spoken voice samples 330a is greater than the number of voice samples in the set of synthesized voice samples 330b.

[0053] In operation 406, method 400 includes training an SSD head 160 with an SSD training utterance 330 to learn to detect the presence of a synthesized speech of an audio encoding 152 encoded by the trained audio encoder 150, using the trained audio encoder 150. Here, the SSD head 160 is superimposed on the trained audio encoder 150.

[0054] In operation 408, method 400 includes providing a multitask neural network model 130 for performing both SID tasks and SSD tasks in parallel on input audio data 120 for execution on computing devices 102, 111. The multitask neural network model 130 includes a trained audio encoder 150, a trained SID head 154 superimposed on the trained audio encoder 150, and a trained SSD head 160 superimposed on the trained audio encoder 150.

[0055] A software application (i.e., a software resource) may refer to computer software that enables a computing device to perform a task. In some examples, a software application may be called an “application,” “app,” or “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0056] Figure 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0057] The computing device 500 includes a processor 510, memory 520, a storage device 530, a high-speed interface / controller 540 connected to memory 520 (i.e., memory hardware) and a high-speed expansion port 550, and a low-speed bus 570 and a low-speed interface / controller 560 connected to storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and may be mounted on a common motherboard or otherwise present as needed. The processor (i.e., data processing hardware) 510 processes instructions stored in the memory hardware (i.e., memory 520) for execution within the computing device 500, including instructions stored on memory 520 or storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 580 coupled to the high-speed interface 540. The data processing hardware 510 and memory hardware 520 may include the data processing hardware 113 and memory hardware 115 of the remote system, the data processing hardware 103 and memory hardware 115 of the AED 102, or a combination thereof. In other embodiments, multiple processors and / or multiple buses may be used as needed, along with multiple memories and multiple types of memory. Multiple computing devices 500 may also be connected, with each device performing some of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0058] Memory 520 stores information non-temporarily within the computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporarily stored memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disk or tape.

[0059] The storage device 530 can provide high-capacity storage to the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different embodiments, the storage device 530 may be an array of devices including a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory or other similar solid-state memory device, or a storage area network or other configuration device. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that perform one or more of the methods described above at runtime. The information carrier is a computer-readable medium or machine-readable medium such as memory 520, the storage device 530, or memory on the processor 510.

[0060] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500 more effectively, while the low-speed controller 560 manages the bandwidth-intensive operations less effectively. Such job assignments are merely examples. In some embodiments, the high-speed controller 540 is coupled to memory 520, a display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 560 is coupled to a storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (such as USB, Bluetooth, Ethernet, or wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as switches or routers, for example, via a network adapter.

[0061] The computing device 500 can be implemented in many different forms, as shown in the figure. For example, it may be implemented as a standard server 500a, or multiple times within a group of servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0062] Various embodiments of the systems and technologies described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be specialized or general-purpose and may include embodiments in one or more computer programs that are executable and / or interpretable on a programmable system comprising at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to a storage system.

[0063] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transient computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor that includes a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0064] The processes and logical flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to act on input data and produce outputs. Processes and logical flows may also be performed by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, for example, one or more processors from both general-purpose and special-purpose processors, and from either type of digital computer. Generally, processors receive instructions and data from read-only memory, random-access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to them to receive data from or transmit data to them, or both. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into dedicated logic circuits.

[0065] To interact with a user, aspects of the present invention can be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touchscreen, and optionally a keyboard and pointing device (e.g., mouse or trackball) through which the user can input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user can be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending and receiving documents to and from the device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0066] Several embodiments have been described. Nevertheless, it is understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer implementation method (400), which is executed by data processing hardware (510), wherein the data processing hardware provides: The acquisition of a speaker identification (SID) model (151) trained to predict speaker embeddings (155) from utterances spoken by different speakers, wherein the SID model (151) includes a trained audio encoder (150) and a trained SID head (154). The receiving of multiple synthesized speech detection (SSD) training utterances (330), wherein the multiple SSD training utterances (330) include a set of human-spoken voice samples and a set of synthesized voice samples (330b), Training an SSD head (160) with the SSD training utterance (330) using the trained audio encoder (150) in order to learn to detect the presence of synthesized speech of an audio encoding (152) encoded by the trained audio encoder (150), wherein the parameters / weights of the trained audio encoder (150) are frozen while the SSD head (160) is being trained with the SSD training utterance (330), and the SSD head (160) is overlaid on the trained audio encoder (150). To provide a multitask neural network model (130) for performing both SID and SSD tasks in parallel on input audio data (120) for execution on a computing device (102), wherein the multitask neural network model (130) includes the trained audio encoder (150), the trained SID head (154) overlaid on the trained audio encoder (150), and the trained SSD head (160) overlaid on the trained audio encoder (150), A computer implementation method (400) that includes performing an operation.

2. The method according to claim 1 (400), wherein the SID model (151) is trained on a plurality of SID training datasets (320), each SID training dataset comprising a plurality of SID training utterances spoken by M different speakers.

3. The method according to claim 2 (400), wherein one or more SID training utterances spoken by a specific speaker are paired with a corresponding target speaker embedding (324) for training the audio encoder (150) and the SID head (154) to learn how to predict a corresponding speaker embedding (155) of the specific speaker.

4. The method (400) of claim 2, wherein the plurality of SID training utterances of one or more SID training datasets (320) include N utterances (320) spoken by each speaker among the M different speakers.

5. The method according to claim 2 (400), wherein the plurality of SID training utterances in each SID training dataset include human-generated non-synthesized speech.

6. The method according to claim 1 (400), wherein the number of voice samples in the set of human-spoken voice samples is greater than the number of voice samples in the set of synthesized voice samples (330b).

7. The aforementioned multitasking neural network model (130) is The process involves receiving the input audio data (120) as input to the trained audio encoder (150), wherein the input audio data (120) characterizes speech captured by an array of one or more microphones (106). The output from the trained audio encoder (150) is to generate a shared audio encoding (152) that encodes the input audio data (120), To provide the shared audio encoding (152) generated as the output from the trained audio encoder (150) as input to the trained SID head (154) and the SSD head (160), The output from the trained SID head (154) is to generate an evaluator speaker embedding (155) that represents the voice characteristics of the utterance characterized by the input audio data (120), The trained SSD head (160) generates an SSD classification score (162) as output, indicating whether the utterance characterized by the input audio data (120) includes human-generated genuine speech or synthesized speech. The method according to claim 1 (400), wherein the SID task and the SSD task are performed in parallel on the input audio data (120).

8. The aforementioned multitasking neural network model (130) When the evaluator speaker embedding (155) matches a reference speaker embedding (155) associated with a registered user (10) authorized to access the resource, When the SSD classification score (162) generated as output from the trained SSD head (160) indicates that the utterance characterized by the input audio data (120) includes genuine human-generated speech, The method according to claim 7 (400), configured to authorize access to one or more of the aforementioned resources.

9. The aforementioned multitasking neural network model (130) The evaluator speaker embedding (155) does not match any reference speaker embedding (155) associated with a registered user (10) authorized to access the resource, or The SSD classification score (162) generated as output from the trained SSD head (160) indicates that the utterance characterized by the input audio data (120) includes synthesized speech. The method according to claim 7 (400), configured to restrict access to one or more of the resources when at least one of the conditions is met.

10. The method according to claim 1 (400), wherein the audio encoder (150) includes a multi-head attention model architecture.

11. The multi-head attention model architecture according to claim 10 (400), which incorporates a stack of conformer blocks.

12. The method according to any one of claims 1 to 11 (400), wherein the SSD head (160) includes one or more fully bonded layers.

13. System (100), Data processing hardware (510) and Memory hardware (520) that communicates with the data processing hardware (510), wherein the memory hardware (520) stores instructions, and when an instruction is executed on the data processing hardware (510), it communicates to the data processing hardware (510). Obtaining a speaker identification (SID) model trained to predict speaker embeddings (155) from utterances spoken by different speakers, wherein the SID model (151) includes a trained audio encoder (150) and a trained SID head (154), The receiving of multiple synthesized speech detection (SSD) training utterances, wherein the multiple SSD training utterances (330) include a set of human-spoken voice samples and a set of synthesized voice samples (330b), Training an SSD head (160) with the SSD training utterance (330) using the trained audio encoder (150) in order to learn to detect the presence of synthesized speech of an audio encoding (152) encoded by the trained audio encoder (150), wherein the parameters / weights of the trained audio encoder (150) are frozen while the SSD head (160) is being trained with the SSD training utterance (330), and the SSD head (160) is overlaid on the trained audio encoder (150). To provide a multitasking neural network model (130) for performing both SID and SSD tasks in parallel on input audio data (120) for execution on a computing device, wherein the multitasking neural network model includes the trained audio encoder (150), the trained SID head (154) overlaid on the trained audio encoder (150), and the trained SSD head (160) overlaid on the trained audio encoder (150), The memory hardware (520) that enables the operation, A system (100) including this.

14. The system (100) according to claim 13, wherein the SID model (151) is trained on a plurality of SID training datasets (320), each SID training dataset comprising a plurality of SID training utterances spoken by M different speakers.

15. The system (100) according to claim 14, wherein one or more SID training utterances spoken by a specific speaker are paired with corresponding target speaker embeddings (324) for training the audio encoder (150) and the SID head (154) to learn how to predict speaker embeddings (155) corresponding to the specific speaker.

16. The system (100) according to claim 14, wherein the plurality of SID training utterances of one or more SID training datasets (320) include N utterances (320) spoken by each speaker among the M different speakers.

17. The system (100) according to claim 14, wherein the plurality of SID training utterances in each SID training dataset include human-generated non-synthesized speech.

18. The system (100) according to claim 13, wherein the number of voice samples in the set of human-spoken voice samples is greater than the number of voice samples in the set of synthesized voice samples (330b).

19. The aforementioned multitasking neural network model (130) is The process involves receiving the input audio data (120) as input to the trained audio encoder (150), wherein the input audio data (120) characterizes speech captured by an array of one or more microphones (106). The output from the trained audio encoder (150) is to generate a shared audio encoding (152) that encodes the input audio data (120), To provide the shared audio encoding (152) generated as the output from the trained audio encoder (150) as input to the trained SID head (154) and the SSD head (160), The output from the trained SID head (154) is to generate an evaluator speaker embedding (155) that represents the voice characteristics of the utterance characterized by the input audio data (120), The trained SSD head (160) generates an SSD classification score (162) as output, indicating whether the utterance characterized by the input audio data (120) includes human-generated genuine speech or synthesized speech. The system (100) according to claim 13, configured to perform the SID task and the SSD task in parallel on the input audio data (120).

20. The aforementioned multitasking neural network model (130) When the evaluator speaker embedding (155) matches a reference speaker embedding (155) associated with a registered user (10) authorized to access the resource, When the SSD classification score (162) generated as output from the trained SSD head (160) indicates that the utterance characterized by the input audio data (120) includes genuine human-generated speech, The system (100) according to claim 19, configured to authorize access to one or more of the aforementioned resources.

21. The aforementioned multitasking neural network model (130) The evaluator speaker embedding (155) does not match any reference speaker embedding (155) associated with a registered user (10) authorized to access the resource, or The SSD classification score (162) generated as output from the trained SSD head (160) indicates that the utterance characterized by the input audio data (120) includes synthesized speech. The system (100) according to claim 19, configured to restrict access to one or more of the resources when at least one of the conditions is met.

22. The system (100) according to claim 13, wherein the audio encoder (150) includes a multi-head attention model architecture.

23. The multi-head attention model architecture incorporates a stack of conformer blocks, according to the system (100) of claim 22.

24. The system (100) according to any one of claims 13 to 23, wherein the SSD head (160) includes one or more fully coupled layers.

Images