Neural Networks for Speaker Verification

Abstract

To provide computer-executed methods for training and using neural networks or other types of models for speaker verification.SOLUTION: A method for verifying an utterance includes the steps of: receiving, at a computing device, data characterizing an utterance of a user of the computing device; generating, using a neural network on the computing device, a speaker representation for the utterance; and comparing the speaker representation with or evaluating the speaker representation with respect to the speaker model. The neural network is trained based on a plurality of training samples that include data characterizing a first utterance and data characterizing one or more second utterances and that are labeled as matching speaker samples or non-matching speaker samples.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The subject matter herein generally relates to neural networks and other models used in speaker verification tasks. [Background technology]

[0002] Speaker verification generally relates to verifying a person's identity based on characteristics of their voice. Some computing devices allow a user to "subscribe" to the device by providing the device with one or more samples of speech spoken by the user. From the samples, a speaker model is determined that represents the user's voice. Subsequent speech samples received at the device may then be processed and evaluated with respect to the speaker model to verify the user's identity. Summary of the Invention [Means for solving the problem]

[0003] Generally, systems, methods, devices, and other techniques for training and using neural networks, or other types of models, for speaker verification are described herein. In some implementations, the neural network may be a component of a speaker verification model accessible by a computing device that performs speaker verification. In general, the neural network may be trained with iterations that simulate the addition of a speaker and the verification of an utterance, respectively. For example, in each training iteration, a speaker representation generated by the neural network for a given utterance may be evaluated with respect to the speaker model. Based on a comparison of the speaker representation for the simulated verification utterance with a combination (e.g., an average) of the speaker representations for one or more simulated addition utterances, parameters of the neural network may be updated to optimize the ability of the speaker verification model to classify a given utterance as being spoken by the same person or a person different from the addition person. It will be appreciated that this has advantages in terms of increasing the reliability of the system. The neural network may further be configured to process data characterizing an entire utterance through the neural network in a single pass, rather than processing frames of the utterance independently or sequentially. These and other implementations are more fully described below and illustrated in the drawings.

[0004] Some implementations of the subject matter described herein include a computer-implemented method that can include selecting, at a computing system, a plurality of different subsets of training data for training a neural network, each subset of the training data including a plurality of first components that classify each utterance of a first speaker and a second component that characterizes an utterance of the first speaker or a second speaker. For each selected subset of training data, the method may include inputting each of the first components into a neural network to generate a respective first speaker expression corresponding to each of the first components; inputting the second components into the neural network to generate a respective first speaker expression corresponding to the second component; determining a simulated speaker model for the first speaker based on an average of the respective first speaker expressions for the plurality of first components; comparing the second speaker expression to the simulated speaker model to classify an utterance characterized by the second component as an utterance of the first speaker or as an utterance of a speaker different from the first speaker; and adjusting the neural network based on whether the utterance characterized by the second component was correctly classified as an utterance of the first speaker or as an utterance of a speaker different from the first speaker.

[0005] These and other implementations may include one or more of the following features.

[0006] In response to inputting the second component into the neural network, a second speaker representation can be generated by the neural network by processing data for the entire utterance characterized by the second component through the neural network in a single pass.

[0007] The step of selecting a first subset of training data from the multiple different subsets of training data may include selecting a first utterance group corresponding to a first speaker from a multiple utterance groups each corresponding to a different speaker, such that each utterance group is composed of data characterizing only utterances of the corresponding speaker for each utterance group; selecting a second utterance group from the multiple groups of utterances corresponding to the first speaker or a second speaker; determining the multiple first components from the data characterizing utterances in the first utterance group; and determining a second component from the data characterizing utterances in the second utterance group.

[0008] At least one of the first group of utterances and the second group of utterances may be randomly selected from the multiple groups of utterances.

[0009] The first speakers can differ from one another in at least some of the different subsets of training data, and the second speakers can differ from one another in at least some of the different subsets of training data.

[0010] A total number of first components in a first subset of training data among the plurality of different subsets of training data can be different from a total number of first components in a second subset of training data among the plurality of different subsets of training data.

[0011] Comparing a second speaker expression to the simulated speaker model to classify the utterance characterized by the second component as an utterance of the first speaker or as an utterance of a speaker different from the first speaker may include determining a distance between a value from the second speaker expression and a value from the simulated speaker model, and applying a logistic regression function to the distance.

[0012] The neural network may include multiple hidden layers, and may not have a softmax output layer.

[0013] The neural network may include a deep neural network with a locally connected hidden layer followed by multiple fully connected hidden layers.

[0014] The utterance characterized by the plurality of first components and the utterance characterized by the second component may all have a fixed length.

[0015] The neural network can be a long short-term memory recurrent neural network configured to be trained with data characterizing utterances having variable lengths.

[0016] The utterances characterized by each of a plurality of first components across the plurality of different subsets of training data and the utterances characterized by each of a plurality of second components across the plurality of different subsets of training data can all be utterances of the same word or phrase.

[0017] At least some of the utterances characterized by each of a plurality of first components across the plurality of different subsets of training data and at least some of the utterances characterized by each of a plurality of second components across the plurality of different subsets of training data may be utterances consisting of different words or different phrases.

[0018] The trained neural network may be provided to one or more computing devices separate from the computing system for use in performing speaker verification on the one or more computing devices.

[0019] Each of the first speaker expressions may be generated by a neural network based on a respective first component input to the neural network corresponding to the first speaker expression, and a second speaker expression may be generated by a neural network based on a second component input to the neural network corresponding to the second speaker expression.

[0020] Some implementations of the subject matter described herein may include a computing device. The computing device may include one or more computer processors and one or more computer-readable media having instructions stored thereon that, when executed by the one or more processors, cause operations to be performed. The operations may include receiving, at the computing device, data characterizing an utterance of a user of the computing device; generating, at the computing device, a speaker representation for the utterance using a neural network on the computing device, the neural network being trained based on a plurality of training samples, each of which includes (i) data characterizing a first utterance and data characterizing one or more second utterances, and (ii) is labeled as a matching sample or a non-matching sample according to whether a speaker of the first utterance is the same as a speaker of the one or more second utterances; accessing, at the computing device, a speaker model for an authenticated user of the computing device; and evaluating, at the computing device, the speaker representation for the utterance for the speaker model to determine whether the utterance could have been spoken by the authenticated user of the computing device.

[0021] Each of the plurality of training samples can be generated by selecting a first utterance and the one or more second utterances from utterance groups corresponding to different speakers, such that each utterance group is composed of only utterances of the corresponding speaker for that respective utterance group.

[0022] The operations may further include obtaining a set of utterances of an authenticated user of the computing device, inputting each utterance from the set of utterances into a neural network to generate a respective speaker representation for the utterance, and generating a speaker model for the authenticated user of the computing device based on an average of each speaker representation for the utterance in the set of utterances of the authenticated user.

[0023] None of the multiple training samples upon which the neural network is trained may include data characterizing the vocalizations of a user of the computing device.

[0024] Generating, at the computing device, a speaker representation for the utterance may include processing data characterizing the entire utterance through a neural network in a single pass.

[0025] The neural network can be a recurrent neural network. An utterance of a user can have a first length of time. The speaker representation for the utterance can include processing data characterizing the utterance for an entirety of the first length of time of the utterance. The operations can further include receiving another utterance of a user of a computing device, the other utterance having a second length of time different from the first length of time, and generating a second speaker representation for the other utterance of the user by processing data characterizing the other utterance for an entirety of the second length of time of the other utterance.

[0026] The operations may further include performing a function at the computing device in response to determining that the user utterance was likely spoken by an authenticated user of the computing device. The function may include changing a state of the computing device from a locked state to an unlocked state, where the computing device is configured to block access to one or more capabilities of the computing device in the locked state and where the computing device is configured to allow access to the one or more capabilities of the computing device in the unlocked state. The speaker representation may include an output of a neural network that indicates distinctive features of the user's voice determined based on the determined utterances.

[0027] Some implementations of the subject matter described herein may include a computer-implemented method. The method may include receiving, at a computing device, data characterizing an utterance of a user of the computing device. A speaker representation may be generated at the computing device for the utterance using a neural network on the computing device. The neural network may be trained based on a plurality of training samples, each of which includes (i) data characterizing a first utterance and data characterizing one or more second utterances, and (ii) is labeled as a matching sample or a non-matching sample according to whether a speaker of the first utterance is the same as a speaker of the one or more second utterances. A speaker model may be accessed at the computing device for an authenticated user of the computing device. A speaker representation for the utterance may be evaluated at the computing device with respect to the speaker model to determine whether the utterance could have been spoken by an authenticated user of the computing device.

[0028] Some implementations of the subject matter described herein can include a computer-implemented method that can include determining a speaker model for a particular speaker based on output of a neural network for a first set of utterances, the first set including a plurality of different particular speaker utterances, determining a speaker representation based on output of the neural network for a particular utterance not within the first set of utterances, comparing the speaker representation to the speaker model for the particular speaker to classify the particular utterance as an utterance of the particular speaker or as an utterance of a speaker different from the particular speaker, and adjusting the neural network based on whether the classification of the utterance as an utterance of the particular speaker or as an utterance of a speaker different from the particular speaker was correct.

[0029] These and other implementations may include one or more of the following features: A set of different utterances may be selected as training data for training the neural network. Each set of utterances may include (i) a first speaker for each set of utterances, and (ii) a second utterance of either the first speaker for each set of utterances or a second speaker for each set of utterances different from the first speaker. The set of different utterances may be used to repeatedly train the neural network. Each set of utterances from the set of different utterances may be used in different training iterations to train the neural network. A first set of utterances may be selected from the set of different utterances. The particular speaker is the first speaker for the first set of utterances.

[0030] Further implementations of the subject matter described herein may include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods and encoded on computer storage devices. One or more computer systems may be configured with software, firmware, hardware, or combinations thereof installed on the systems that, during operation, cause the systems to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by one or more data processing devices, cause the devices to perform the actions.

[0031] Some implementations of the subject matter described herein may achieve one or more of the following advantages: A neural network may be trained to generate speaker representations for use in a speaker verification model that can more accurately verify the identity of a speaker based on characteristics of a user's voice. A target performance level may be achieved with a compact neural network that may be stored and used on a mobile computing device with finite computing resources. Furthermore, the neural network may be trained in a manner that simulates the verification and joining phases of a speaker verification process. Thus, the neural network may achieve good performance due to the symmetry between the training phase and the verification and joining phases of the speaker verification process. A benefit of training a neural network according to the approach described herein is that a larger number of different speakers may be used to train the network, in contrast to other approaches that involve training a neural network to classify inputs as belonging to a particular speaker among a preselected number of speakers. Furthermore, there may be no minimum number of training utterances required per training speaker to ensure reliable training. Further, the neural network may be configured to process data characterizing an entire utterance through the neural network in a single pass, without having to process frames of that utterance through the neural network in independent passes. The above-referenced aspects may contribute to assessing whether an utterance was likely spoken by an authenticated user of a computing device. The aspects may make such assessment particularly reliable. This may result in a more effective speech recognition system, at least by reducing errors in assessing whether an utterance was spoken by a particular person, such as an authenticated user of a computing device. The reliability may increase the security of a wide range of systems associated with the above-referenced aspects.For example, when the aspects are used to recognize an authorized user and, in response, change the state of a computing device from a locked state to an unlocked state, the reduction in errors referenced above makes unlocking a computing device more secure. Indeed, the high reliability of the system may make the computing device less vulnerable to fraudulent unlocking attempts from unauthorized users. The reduction in errors may also reduce the need for error correction, which may reduce the need to allocate computing resources to error correction. This is particularly advantageous in mobile computing devices where computer resources may be more limited. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a schematic diagram of an exemplary client device and computing system that performs operations of (i) training a neural network, (ii) enrolling a user at a computing device, and (iii) verifying the utterances of a user of the computing device based on distinctive features of the user's voice. [Diagram 2] FIG. 1 is a block diagram for training a neural network to determine speaker expressions for use in speaker verification tasks. [Diagram 3] 1 is a flow diagram of an exemplary process for training a neural network to determine speaker expressions indicative of distinctive characteristics of speech. [Figure 4A] FIG. 2 is a conceptual diagram illustrating an exemplary selection of a subset of training utterances from a group of utterances for different speakers in a utterance pool. [Figure 4B] FIG. 2 is a conceptual diagram illustrating an exemplary selection of a batch of training data from a pool of utterances for training a neural network. [Figure 5A]FIG. 1 is a block diagram of an example deep neural network configured to process data characterizing at least a portion of an utterance and generate a speaker representation based on the data characterizing the at least a portion of the utterance. [Figure 5B] FIG. 1 is a block diagram of an example recurrent neural network with long and short-term memory layers configured for use in a speaker verification model. [Figure 6] 1 is a flow diagram of an example process for verifying a user's identity based on characteristics of the user's voice determined from the user's vocalizations using a neural network on a computing device. [Figure 7] FIG. 1 illustrates an example of a computing device and a mobile computing device that can be used to execute the computer-implemented methods and other techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] 1 is a schematic diagram of an exemplary system 100 for training a neural network against a speaker verification model and performing a process of speaker verification using the model. In general, speaker verification is the task of accepting or rejecting a speaker identity claim based on characteristics of the speaker's voice determined from one or more utterances of the speaker. As shown in FIG. 1, speaker verification can generally include three stages: (i) training a neural network against the speaker verification model, (ii) adding a new speaker, and (iii) verifying the added speaker.

[0034] The system 100 includes a client device 110, a computing system 120, and a network 130. In some implementations, the computing system 120 may provide the client device 110 with a speaker verification model 144 based on the trained neural network 140. In some implementations, the speaker verification model 144 may be pre-installed on the client device 110, for example, as a component of an operating system or application. In other implementations, the speaker verification model 144 may be received over the network 130. The client device 110 may use the speaker verification model 144 to enroll the user 102 in the speaker verification process. When the identity of the user 102 needs to be verified at a later point in time, the client device 110 may receive the conversational utterances of the user 102 to verify the identity of the user 102 using the speaker verification model 144. The speaker verification model 144 may be stored locally on the client device 110, so that the client device 110 may be able to make a speaker verification decision without communication over the network 130.

[0035] Although not shown in FIG. 1 , in some implementations, the computing system 120 may store a speaker verification model 144 based on the trained neural network 140, rather than or in addition to the neural network 140 stored on the client device 110. In these implementations, the client device 110 may communicate with the computing system 120 over the network 130 to remotely access the speaker verification model 144 and use it for the enrollment of the user 102. When the identity of the user 102 needs to be verified at a later point in time, the client device 110 may receive the conversational utterances of the user 102 and communicate with the computing system 120 over the network 130 to verify the identity of the user 102 using the remotely located speaker verification model 144. The computing system 120 and the computing device 110 may be distinct from each other and may be physically separate.

[0036] In system 100, client device 110 can be, for example, a desktop computer, a laptop computer, a tablet computer, a watch, a wearable computer, a cellular phone, a smartphone, a music player, an e-book reader, a navigation system, or any other suitable computing device with which a user may interact. In some implementations, client device 110 may be a mobile computing device. Computing system 120 can include one or more computers, and functions may be performed on independent ones of the computers, or the functions may be distributed for performance across multiple computers. Network 130 can be wired or wireless, or a combination of both, and can include the Internet.

[0037] In some implementations, a client device 110, such as a user's phone, may store the speaker verification model 144 locally on the client device 110, allowing the client device 110 to verify the identity of a user without relying on a model at a remote server (e.g., the computing system 120) for the joining or verification process, thus saving communication bandwidth and time. Furthermore, in some implementations, when one or more new users join, the speaker verification model 144 described herein does not require any retraining of the speaker verification model 144 with the new users, which may also be computationally efficient. In other implementations, utterances of a given user provided for joining, verification, or both may be provided to the computing system 120 and added to the training data, such that the neural network (and thus the speaker verification model) can be constantly updated based on the use of newly collected training data.

[0038] The size of the speaker verification model 144 including the trained neural network 140 may desirably be compact since storage and memory space of the client device 110 may be limited. As described below, the speaker verification model 144 is based on the trained neural network 140. The speaker verification model 144 may include the neural network 140 for generating a speaker representation indicative of distinctive features of the voice of a speaker of the utterance based on data characterizing the utterance. The speaker verification model 144 may include further components for processing the speaker representation and determining whether the voice of the speaker of the utterance is sufficiently similar to the voice of a subscribed user so that the identity claim of the speaker of the utterance can be verified.

[0039] In some implementations, the neural network may be trained using a large set of training data. Various techniques may be applied during pre-processing of the training data, during the training itself, or during a post-training stage to constrain and / or reduce the size of the neural network to achieve a compact model size. For example, the speaker verification model 144 may be constructed by selecting only certain layers of the neural network 140. This may result in a compact speaker verification model suitable for storage on the client device 110. Joining may be performed without a softmax or other classification layer in generating speaker representations for the speaker model.

[0040] 1 also illustrates an exemplary flow of data, indicated as steps (A) through (F). Steps (A) through (F) may occur in the sequence shown, or may occur in a different sequence than the sequence shown. In some implementations, one or more of steps (A) through (F) may occur offline. Computing system 120 may perform computations when client device 110 is not connected to network 130. Steps (A) and (B) typically occur during the training phase referenced above. Step (D) typically occurs during an enrollment phase. Steps (E) through (G) typically occur during a validation phase.

[0041] In step (A), the computing system 120 selects a sample of training utterances to provide supervised training of the neural network 140 to the neural network 140. In some implementations, the utterances in the training sample 122 may each consist of one or more predefined words spoken by a number of different training speakers. The utterances have been previously recorded and made accessible for use by the computing system 120. Each training speaker may speak the predefined utterance to a computing device, which may record an audio signal including the utterance. For example, each training speaker may be prompted to speak the training phrase "Hello Phone." In some implementations, each training speaker may be prompted to speak the same training phrase multiple times. The recorded audio signal of each training speaker may be transmitted to the computing system 120, which may collect the recorded audio signals from a number of different computing devices and a number of different training speakers. In some implementations, neural network 140 may be optimized for text-dependent speaker verification, in that a user's identity may be verified based on characteristics of the user's voice determined from utterances of the predefined training phrases. In such implementations, neural network 140 may be trained with all, or substantially all, utterances that include the predefined training phrases. In other implementations, neural network 140 may be trained to enable text-independent speaker verification, in that a user's identity may be verified based on characteristics of the user's voice determined from utterances of a wide variety of words or phrases, which may not be predefined.For example, a user can independently determine what words or phrases he or she wants to speak to verify their identity, and a speaker verification model based on the trained neural network 140 can then authenticate the user given those spoken words or phrases. To enable text-independent speaker verification, the neural network 140 may be trained with a wide variety of utterances of words or phrases spoken by a number of different training speakers.

[0042] In step (B), the neural network 140 may be trained in a parallel manner with the enrollment and verification of users at the client device. Thus, the computing system 120 may select a set of simulated enrollment utterances 122b and simulated verification utterances 122a for each training sample 122. The simulated enrollment utterances 122b may be all utterances of the same training speaker, such that a simulated speaker model may be determined for each training sample 122. The simulated verification utterances 122a may be utterances of the same speaker as the speaker of the simulated enrollment utterances 122b, or may be utterances of a different speaker. The training sample 122 may then be provided to the neural network 140, and a classification may be made based on the output of the neural network 140 as to whether the simulated verification utterances 122a were spoken by the same speaker as the speaker of the simulated enrollment utterances 122b, or by a different speaker than the speaker of the simulated enrollment utterances 122b. The neural network 140 may then be updated based on whether the speaker determination was correct. In some implementations, each training sample 122 may be labeled as belonging to one of two classes: a matching speaker class 141a (for samples where the speakers of the simulated verification utterance and the added utterance are the same) and a non-matching speaker class 141b (for samples where the speakers of the simulated verification utterance and the added utterance are different). These labels indicate the ground truth of whether the same speaker spoke the utterance 122a and the utterance 122b. The accuracy of the classification of the training sample 122 may be determined based on the label of the sample. In some implementations, adjustments to the neural network may not be based strictly on the accuracy of the classification of the input sample, but may generally be based on one or more metrics determined from a comparison of the speaker representations generated by the neural network for the simulated verification utterance 122a and the simulated added utterance 122b.In some implementations, the training samples 122 may be selected from a repository of training data. The repository of training data may be organized into utterance pools 121. Each of the utterance pools 121 may include training utterances grouped by training speakers of that utterance.

[0043] The neural network 140 may include an input layer for inputting information about the utterances in the training samples 122, and several hidden layers for processing the samples 122. The weights or other parameters of one or more hidden layers may be adjusted such that the trained neural network 140 produces an output that causes the speaker verification model 144 to produce a desired classification of the training samples 122 as having either a matching or non-matching speaker among the simulated verification utterances and the enrolled utterances of the samples 122. In some implementations, the parameters of the neural network 140 may be adjusted automatically by the computing system 120. In some other implementations, the parameters of the neural network 140 may be adjusted manually by an operator of the computing system 120. The training phase of the neural network is described in more detail below, for example in the description of FIGS. 2, 3, 4A-B, and 5A-B.

[0044] In step (C), once the neural network 140 is trained, a speaker verification model 144 based on the trained neural network 140 is transmitted, for example, from the computing system 120 to the client device 110 via the network 130. In some implementations, the trained neural network 140, or a portion thereof, may be a component of the speaker verification model 144. The speaker verification model 144 may be configured to verify the identity of the user 102 based on characteristics of the user's voice determined from one or more user 102 utterances. The model 144 may be configured to provide data characterizing the user's utterances as input to the trained neural network 140 to generate a speaker representation for the user 102 that indicates distinctive features of the user's voice. The speaker representation may then be compared to a previously determined model of the user's voice. If the speaker representation is sufficiently similar to the user's speaker model, the speaker verification model 144 may output an indication that the identity of the user 102 is valid. In contrast, if the speaker expressions are not sufficiently similar to the user's speaker model, the speaker verification model 144 may output an indication that the identity of the user 102 is invalid (not verified).

[0045] In step (D), a user 102 who wishes to have his or her voice enlisted on the client device 110 provides one or more enlistment utterances 152 to the client device 110 in an enlistment step. Typically, the user 102 is not one of the training speakers whose voice was not used in training the neural network 140. In some implementations, the client device 110 may prompt the user 102 to speak an enlistment phrase that is the same phrase spoken by the set of training speakers in the utterances of the training samples 122. In some implementations, the client device 110 may prompt the user to speak the enlistment phrase a number of times and record the audio signal for the spoken enlistment utterance as the enlistment utterance 152.

[0046] The client device 110 uses the enrollment utterances 152 to enroll the user 102 in the speaker verification system of the client device 110. Generally, enrollment of the user 102 occurs without retraining the neural network 140. Respective instances of the same speaker verification model 144 may be used in many different client devices to enroll many different speakers without having to change weight values ​​or other parameters in the neural network 140. Because the speaker verification model 144 can be used to enroll any user without retraining the neural network 140, enrollment may be implemented in the client device 110 with limited processing requirements.

[0047] In some implementations, information about the joining utterances 152 may be input to a speaker verification model 144, which may output a reference vector or other set of values ​​corresponding to the user 102. The reference vector or other set of values ​​may constitute a speaker model that characterizes distinctive features of the user's voice. The speaker model may be stored on the client device 110 or on a computing system remote from the client device 110, so that speaker representations generated based on utterances subsequently received by the client device 110 may be compared to the speaker model to verify whether each speaker of the subsequently received utterance is the user 102 or another speaker.

[0048] In step (E), the user 102 attempts to gain access to the client device 110 using voice authentication. The user 102 provides a verification utterance 154 to the client device 110 during the verification phase. In some implementations, the verification utterance 154 is an utterance of a spoken phrase that is identical to the enrollment utterance 152. The verification utterance 154 is used as an input to the speaker verification model 144.

[0049] In step (F), the client device 110 determines whether the user's voice matches that of the joined user. In some implementations, the neural network 140 may process data characterizing the verification utterance 154 and output a speaker representation for the user 102 based on the verification utterance 154. In some implementations, the client device 110 may compare the speaker representation for the user 102 to the speaker models for the joined users to determine whether the verification utterance 154 was spoken by the joined user. The verification stage of the neural network is described in more detail below, for example with respect to FIG. 6.

[0050] In step (G), the client device 110 provides an indication to the user 102 representing the verification result 156. In some implementations, if the client device 110 accepts the identity of the user 102, the client device 110 may send a visual or audio indication to the user 102 that the verification was successful. In some other implementations, if the client device 110 accepts the identity of the user 102, the client device 110 may prompt the user 102 for further input. For example, the client device 110 may output a message "Device enabled. Please enter your search" on its display. In some other implementations, if the client device 110 accepts the identity of the user 102, the client device 110 may perform a subsequent action without waiting for further input from the user 102. For example, the user 102 may speak "Hello Phone, search the nearest coffee shop" to the client device 110 during the verification phase. The client device 110 may verify the identity of the user 102 using the verification phrase "Hello Phone". If the identity of the user 102 is accepted, the client device 110 may perform a search for the nearest coffee shop without prompting the user 102 for further input. In general, in some implementations, if the client device 110 accepts the identity of the user 102, the client device 110 may respond by transitioning from a locked state to an unlocked state, in which one or more capabilities of the client device 110 are disabled or blocked, and in which the capabilities are enabled or made available for the user 102 to access. Similarly, the client device 110 may "activate" or transition to a more fully characterized state in response to a successful verification.

[0051] In some implementations, if the client device 110 rejects the identity of the user 102, the client device 110 may send a visual or audio indication to the user 102 that verification was rejected. In some implementations, if the client device 110 rejects the identity of the user 102, the client device 110 may prompt the user 102 for another speech attempt. In some implementations, if the number of attempts exceeds a threshold, the client device 110 may block the user 102 from further attempts to verify his or her identity.

[0052] Referring to Figure 2, a block diagram of an exemplary system 200 for training a neural network 206 is shown. Upon completion of the training phase illustrated by Figure 2, the trained neural network 206 can process data characterizing a speaker's utterances and generate a speaker representation for the speaker that indicates distinctive features of the speaker's voice. The speaker representation may then be used to generate a speaker model for the speaker during an enrollment phase or by a speaker verification model in verifying the identity of the speaker during a verification phase.

[0053] In general, FIG. 2 shows that the neural network 206 may be trained in a parallel manner with the enrollment and verification phases occurring at a client device that subsequently performs the speaker verification task. Unlike some approaches that train the neural network 206 to classify training utterances from a finite number of speakers into corresponding classes for each of the speakers, the neural network 206 of FIG. 2 is not trained to determine the specific speaker of a given utterance. Instead, the neural network 206 is trained to generate speaker representations that are distinctive and can be used to determine whether the speaker of a given utterance is the same as the speaker of another set of utterances, without necessarily matching any of the utterances to a specific speaker identity. In this way, the loss function optimized during training is the same function utilized by the speaker verification model during the verification phase. In other words, during verification, the speaker representations based on the verification utterances are compared to the speaker model for the enrolled user. If the speaker representations are sufficiently similar to the speaker model, the identity of the speaker of the verification utterance is verified. The approach shown in FIG. 2 uses similar techniques during training. That is, a simulated speaker model 214 is generated based on speaker representations for one or more added utterances, and a speaker representation 208 is also generated for the simulated verification utterances 202. Weights and other parameters of the neural network 206 are adjusted during training to minimize errors in classifying the simulated verification utterances 202 as being spoken by the same or a different speaker as the simulated added utterances 204a-n.

[0054] FIG. 2 illustrates a forward pass of a single training iteration based on a sample of training data including data characterizing a simulated verification utterance 202 and data characterizing one or more simulated addition utterances 204a-n. In practice, the neural network 206 is trained over many iterations and many different samples of training data. With each iteration, the neural network 206 may be adjusted based on the results of processing the corresponding sample of training data for that iteration. FIGS. 4A and 4B illustrate an exemplary technique by which the simulated verification utterance 202 and the simulated addition utterances 204a-n may be selected, as described further below. The simulated addition utterances 204a-n for a particular sample are generally all utterances spoken by the same training speaker. Although the speaker of the simulated addition utterances 204a-n may vary between different samples of training data for different training iterations, in a given training sample for a given training iteration, all of the addition utterances 204a-n are generally spoken by the same training speaker. The simulated verification utterance 202 may be spoken by the same training speaker as the speaker of the simulated add-on utterances 204a-n, or may be spoken by a different training speaker than the speaker of the simulated add-on utterances 204a-n. For training data samples whose speaker is the same between the simulated verification utterance 202 and the simulated add-on utterances 204a-n, the samples may be labeled as "matching" samples. For training data samples whose speaker is different between the simulated verification utterance 202 and the simulated add-on utterances 204a-n, the samples may be labeled as "non-matching" samples. The labels may represent the true classification of the training samples, and may be determined in a pre-processing stage prior to training.In some implementations, the estimated classification of a training sample based on the output of the neural network 206 may be compared to a true classification indicated by the label for the training sample to determine whether to adjust the neural network 206.

[0055] In some implementations, the data in the training samples may not be the raw audio signals for the simulated verification and added utterances 202, 204a-n. Instead, the utterances 202, 204a-n may have been processed and converted into a suitable format for processing by the neural network 206. For example, the data in the training samples may characterize features of the simulated verification and added utterances 202, 204a-n, rather than the raw audio signals themselves. In some implementations, the data representing each of the simulated utterances 202, 204a-n in the training samples may include one or more log filter banks for each utterance. In some implementations, each utterance may be divided into multiple frames for the utterance, and a separate log filter bank may be generated for each frame of the utterance. For example, each frame of the utterance may be represented by, for example, 40 log filter banks.

[0056] In some implementations, data characterizing the simulated verification utterance 202 and data characterizing each of the simulated joining utterances 204a-n may be processed once (i.e., in a single pass) through the neural network 206. Thus, while the training data for a given utterance has been split into multiple frames, each represented by a set of log filter banks, data characterizing all of the frames for the entire utterance may be input to the neural network 206 (e.g., as an 80x40 feature vector for 80 frames with 40 log filter banks each) for processing in a single pass through the neural network. This is in contrast to inputting data for each frame of the utterance independently into the neural network 206 for processing the frames separately. In other implementations, rather than training the neural network 206 to process data characterizing the entirety of each utterance 202, 204a-n through the neural network 206 in a single pass, data characterizing independent frames of the utterances 202, 204a-n can be provided as input to the neural network 206.

[0057] In some implementations, the simulated validation and enrollment utterances 202, 204a-n may be pre-processed according to one or more additional techniques. For example, the structure of the neural network 206 may require that the training utterances all have a fixed length (e.g., 0.8 seconds of audio). At least some of the utterances 202, 204a-n may thus be the result of pruning long utterances to a uniform length and / or padding some short utterances to create long clips. In other implementations, however, the neural network 206 may be able to handle variable length utterances, in which case the utterances 202, 204a-n in the training data may be pruned or padded to a fixed length. The audio for the utterances 202, 204a-n may also be equalized, and noise may be added to or removed from the training utterances 202, 204a-n to ensure that the neural network is trained to perform robustly in the presence of noise.

[0058] The portion of the system 200 within the dotted box 215 simulates the joining stage of the speaker verification process in that data characterizing a plurality of simulated joining utterances 204a-n is used to generate a simulated speaker model 214 for a particular training speaker of the simulated joining utterances 204a-n. Each piece of data characterizing each of the simulated joining utterances 204a-n is input separately to an input layer of the neural network 206. The neural network 206 processes the data through one or more hidden layers to generate a respective speaker representation 210a-n for each of the simulated joining utterances 204a-n. For example, as shown in FIG. 2, speaker representation 1 (210a) is generated by the neural network 206 based on the simulated joining utterance 1 (204a). Similarly, speaker representation 2 (210b) is generated by the neural network 206 based on the simulated joining utterance 2 (204b). The speaker representations may thus be generated by the neural network 206 for each of the simulated recruited utterances 204a-n. In some implementations, the speaker representations 210a-n may be generated by sequentially processing each of the simulated recruited utterances 204a-n through the neural network 206. In some implementations, the speaker representations 210a-n may be generated in parallel by processing data characterizing the utterances 204a-n in parallel with each instance of the neural network 206 for each of the simulated recruited utterances 204a-n. Each of the speaker representations 210a-n typically includes a set of values ​​that represent distinctive characteristics of the voice of that simulated recruited training speaker as determined by the neural network 206 based on a corresponding one of the simulated recruited utterances 204a-n. In some implementations, the speaker representations 210a-n may represent weights or other parameters of the final hidden layer of the neural network 206. In some implementations, the speaker expressions 210a-n may be the final output of the neural network 206 when the neural network 206 is configured without a softmax output layer.

[0059] To generate the simulated speaker model 214, the speaker expressions 210a-n may be averaged as shown in box 212 of FIG. 2. The simulated speaker model 214 may thus define a set of values ​​that represent distinctive characteristics of the speech of the training speaker of the simulated added utterances 204a-n. Averaging multiple speaker expressions 210a-n to determine the simulated speaker model 214 may smooth out variations in the speaker's speech between the different simulated added utterances 204a-n. The simulated speaker model 214 may be a more reliable representation of the speaker's speech than any of the independent speaker expressions 210a-n. It may independently reflect the qualities of a given simulated added utterance 204a-n.

[0060] In some implementations, the total number of simulated added utterances 204a-n in each sample of training data may vary for each training iteration. For example, a first training sample for a first training iteration may include nine simulated added utterances 204a-n. A second training sample for a second training iteration may include only four simulated added utterances 204a-n. In other implementations, the total number of simulated added utterances 204a-n in each sample of training data may be fixed for each training iteration. For example, the neural network 206 may be trained for a series of iterations in which the set of training data includes a total of five simulated added utterances 204a-n for each iteration. In some implementations, one, some, or all of the training iterations may be performed with training samples that include only a single simulated added utterance 204a-n.

[0061] In the same manner that the speaker representations 210a-n were generated from data characterizing the simulated addition utterances 204a-n, the speaker representations 208 may be generated from data characterizing the simulated verification utterances 202. The data characterizing the simulated verification utterances 202 (e.g., log filter bank values ​​characterizing features of the verification utterances 202) may be provided to an input layer of the neural network 206. The neural network 206 then processes the input through one or more hidden layers of the network. The output of the neural network 206 is a speaker representation 208 that defines a set of values ​​indicative of distinctive characteristics of the voice of the speaker who spoke the simulated verification utterances 202.

[0062] To further parallelize the verification phase during the training of the neural network 206, the speaker expressions 208 based on the simulated verification utterances 202 can be compared to the simulated speaker model 214 in the same manner as is done on the client device by the speaker verification model during the verification phase, for example. In some implementations, the comparison can be performed by taking the cosine distance (as shown in block 216) of (1) a first vector defining a set of values ​​for the simulated speaker expressions 208 and (2) a second vector defining a set of values ​​for the simulated speaker model 214. A logistic regression 218 can then be applied to the distance to estimate whether the training speaker who spoke the simulated verification utterances 202 is the same or different from the training speaker who spoke the simulated joining utterances 204a-n. This is represented in FIG. 2 by a first block 220a for the matching speaker class and a second block 220b for the non-matching speaker class. In some implementations, a classification technique different from logistic regression 218 may be applied to make a determination as to whether the training speaker who spoke the simulated verification utterance 202 is the same or different from the training speaker who spoke the simulated addition utterances 204a-n. For example, a hinge layer or a softmax layer may be used for some alternative classifications. In a two class model such as that shown in FIG. 2, the softmax and logistic regression techniques may use the same or similar optimization functions.

[0063] Weights or other parameters of the neural network 206 may then be adjusted based on the results of the comparison of the speaker representations 208 for the simulated verification utterances 202 with the simulated speaker model 214, as represented by block 222. For example, if the training sample was labeled as having a truly non-matching speaker and an incorrectly classified training sample was classified as having a matching speaker, the neural network 206 may be automatically adjusted to correct the error. More generally, the neural network 206 may be optimized to maximize the similarity score for matching speaker samples or optimize the score output by logistic regression, and the neural network 206 may also be optimized to minimize the similarity score for non-matching speaker samples or optimize the score output by logistic regression. In some implementations, adjustments to the neural network 206 may be made in response to the results of each training sample at each training iteration, or the neural network 206 may be adjusted based on the results of only some of the training iterations. In some implementations, the neural network 206 may be tuned to maximize the distance (i.e., maximize the variance) between the speaker representations 208 and the simulated speaker models 214 for the unmatched speakers and minimize the distance (i.e., minimize the variance) between the speaker representations 208 and the simulated speaker models 214 for the matching speakers. Note that in some implementations, hard decisions to classify training samples as belonging to either the matching speaker class 220a or the unmatched speaker class 220b may not be made during the training phase. Rather, the neural network 206 may be tuned in a manner that optimizes the score output by the logistic regression layer 218 or optimizes one or more other metrics.

[0064] 3, a flow diagram of an example process 300 for training a neural network that may be used in a speaker verification model is shown. In some implementations, the process 300 may be performed by a computing system described herein, such as computing system 120 from FIG. 1 and computing system 200 from FIG.

[0065] Process 300 begins at step 302, where a first set of training data is selected (i.e., first training samples). The first set of training data may include data characterizing simulated verification utterances and data characterizing one or more simulated recruitment utterances. The utterances in the training set are "simulated" in that they are used in the training process in a manner that parallels or "simulates" the recruitment and verification phases of speaker verification during the training phase. However, the utterances themselves are generally actual snippets of recorded conversations spoken by a training speaker. The training speaker is generally not the same speaker who provided utterances during the actual recruitment and verification phases of the speaker verification process. Figures 4A and 4B, described further below, show an exemplary technique for selecting the simulated verification utterances and the recruitment utterances.

[0066] The selected set of training data (i.e., the selected samples) may be labeled according to whether it represents a matching speaker's speech or a sample for a non-matching speaker. If the speaker of the simulated verification utterance is the same as the speaker of the simulated addition utterance, the set of training data is labeled as a matching speaker sample. If the speaker of the simulated verification utterance is different from the speaker of the simulated addition utterance, the set of training data is labeled as a non-matching speaker sample. In some implementations, the label can be used later in the training process 300 to determine whether the estimated classification of the set of training data as either a matching or non-matching sample is accurate.

[0067] In some implementations, the selected set of training data may not be raw audio signals for the simulated verification and enrollment utterances, but may instead include data characterizing features of the utterance. For example, each utterance represented in the set of training data may be characterized by a set of log filter banks determined for fixed length frames of the utterance. The log filter banks for each frame of the utterance may then be concatenated into a single set of input values ​​that are provided as input to a neural network to classify the entire utterance.

[0068] At steps 304 and 306 of the process 300, a speaker representation is determined for each of the utterances classified in the first set of training data. Each of the speaker representations can be a set of values ​​indicative of distinctive features of the speech of the training speaker who spoke the corresponding utterance for each of the speaker representations. For example, a first speaker representation may be generated based on the simulated verification utterances, and each of the second speaker representations may be generated based on each of the simulated joining utterances. To generate the speaker representations, data characterizing utterances is provided to an input layer of a neural network being trained. The neural network then processes the input data through one or more hidden layers of the network. The speaker representation is then the output of the neural network. In some implementations, the output is output at an output layer that is not a softmax layer. The final layer providing the output may be a fully connected linear layer. In some implementations, the speaker representation may include values ​​or activations generated at the last hidden layer of the neural network, rather than the output of a softmax output layer. In some implementations, the neural network may be constructed without a softmax output layer.

[0069] At step 308, the speaker expressions corresponding to the simulated recruited utterances are combined to generate a simulated speaker model. The simulated speaker model can be an average of the speaker expressions for the simulated recruited utterances. The speaker expressions can be averaged to determine a reliable model that characterizes the speech of the training speaker. For example, variations in the way the speaker spoke each of the simulated recruited utterances may be smoothed so that the speaker model can be used with a robust baseline against which the speaker expressions for the simulated verification utterances are compared. In some implementations, the process 300 may select only a subset of the speaker expressions for the simulated recruited utterances to combine in generating the simulated speaker model. For example, a measure of quality of each of the simulated recruited utterances or the corresponding simulated recruited utterance may be determined. The process 300 may then select only those speaker expressions that meet a threshold quality score, or those speaker expressions whose corresponding utterances meet a threshold quality score, for inclusion in the set of expressions used to generate the simulated speaker model.

[0070] At step 310, the speaker expressions for the simulated verification utterances are compared to the simulated speaker model. In some implementations, a binary classifier is used to classify the data samples as representing or not representing a matching speaker. In some implementations, the comparison may include determining a measure of similarity between the speaker expressions for the simulated verification utterances and the simulated speaker model. For example, the measure of similarity may be a cosine distance between a vector of values ​​for the speaker expressions and a vector of values ​​for the simulated speaker model. The measure of similarity may then be used to estimate a classification of the first set of training data as either a matching speaker sample or a non-matching speaker sample. For example, if the measure of similarity is sufficiently high (e.g., meets a threshold similarity score), logistic regression may be used to map the set of training data to a class of matching speakers. On the other hand, if the measure of similarity is too low (e.g., does not meet a threshold similarity score), logistic regression may be used to map the set of training data to a class of non-matching speakers.

[0071] Next, in step 312, one or more parameters of the neural network may be adjusted based on the results of the comparison in step 310 between the speaker representations and the simulated speaker model for the simulated verification utterances. For example, weights of various nodes in the hidden layer, or other parameters of the neural network, may be adjusted to increase the distance (reduce the similarity score) between the speaker representations and the simulated speaker model when the training data is labeled as a non-matching speaker sample. Furthermore, weights or other parameters of the neural network may be adjusted to reduce the distance (increase the similarity score) between the speaker representations and the simulated speaker model when the training data is labeled as a matching speaker sample. In general, since each iteration of the training process 300 is intended to simulate each recruitment stage and each verification stage, the neural network may be adjusted to optimize the same loss function that was applied to the actual recruitment and verification stages during speaker verification. One benefit of this approach is that the neural network is trained to better generate speaker representations that can be used in a speaker verification model for more accurate verification speaker identities. For example, in some implementations, no additional post-processing steps are performed during the actual validation of utterances that are not considered when training the neural network. These techniques may be considered as "end-to-end" approaches to training neural networks.

[0072] Finally, at step 314, a next set of training data is selected for another iteration of the training neural network. Again, the set of training data selected at this step may include data characterizing a simulated verification utterance and data characterizing one or more simulated addition utterances. Process 300 may then repeat steps 304 through 312, continuing to select additional sets of training data for additional training iterations until a limit is reached. In some implementations, the limit may result from all of the available training data expiring. In some implementations, process 300 may continue until a target performance level is reached. For example, after a number of training iterations, the neural network may be tested against a set of submitted data that was not used during the training process 300. Training may continue until testing on the submitted set indicates that the neural network has achieved at least the target performance level.

[0073] 4A and 4B, diagrams illustrating example techniques for selecting a set of training data for use in training a neural network for a speaker verification model are shown. In some implementations, the techniques described with respect to FIGs. 4A and 4B can ensure diversity of the selected training utterances over multiple training iterations. This can result in the neural network performing better for a given number of training utterances.

[0074] In some implementations, all or a portion of the available training utterances may be clustered into groups 410a-n. The groups 410a-n may be further organized into an utterance pool 408 that includes a collection of groups of training utterances. The training utterances may be grouped by speaker in some implementations. For example, group 410a may include a number of utterances all spoken by a first speaker, and group 410n may include a number of utterances all spoken by another speaker. Thus, each of the groups 410a-n may correspond to a different speaker. The groups 410a-n may all include the same number of training utterances, or the number of training utterances may vary among different ones of the groups 410a-n.

[0075] For each training iteration, the utterance pool 408 may be accessed and a particular utterance may be selected for the sample of training data to be applied as input in each training iteration. For example, FIG. 4A shows a set of training data randomly selected from the utterance pool 408 for a training iteration as the input sample 402. A first group of utterances corresponding to a first speaker may be selected from groups 410a-n in the utterance pool 408 for use in generating the simulated speaker model. The groups may be selected randomly or in another manner. From the selected group, e.g., group 410a in FIG. 4A, a subset of the first speaker's utterances is selected as the simulated joining utterances 406 in the input sample 402. This subset typically includes multiple utterances and may include the same or different number of utterances from one training iteration to another. Utterances from the selected group, e.g., group 410a, may be randomly selected so that different combinations of the utterances are used to generate different simulated speaker models for the first speaker in different training iterations.

[0076] An utterance 404 is also selected as a simulated verification utterance. The utterance 404 may be an utterance of the first speaker or a different speaker, depending on whether the training iteration is an example of a match or a non-match with the joining utterance 406. Both matching and non-matching examples are used in training. As a result, for some training iterations, the utterance 404 is an utterance of the first speaker, e.g., an utterance from group 410a. For other training iterations, the utterance 404 is an utterance of a second speaker different from the first speaker, as shown in FIG. 4A, so that the input sample 402 does not represent a match between the simulated verification utterance 404 and the simulated joining utterance 406.

[0077] In the example of FIG. 4A, a particular utterance is selected (e.g., randomly selected) from the second utterance group 410n as the simulated verification utterance 404. In some implementations, the second utterance group (from which the utterance 404 is selected) may be selected from the groups 410a-n in the utterance pool 408 randomly or according to a pattern of varying selection of the groups 410a-n. In other implementations, a random selection may be made as to whether another utterance from the same speaker as the speaker of the simulated joining utterance should be applied as the simulated verification utterance. Thus, perhaps the random selection is biased, so that there is a 50 percent chance that the simulated verification utterance 404 is an utterance of the same speaker as the speaker 406 of the simulated joining utterance. If the result of the random selection is that the input sample 402 is a matching speaker sample, the simulated verification utterance 404 may be selected from the same group of utterances 410 as the group of utterances from which the simulated joining utterance 406 was selected. However, if the result of the random selection is that the input sample 402 is a non-matching speaker sample, the simulated verification utterance 404 may be selected from a different group of utterances 410 corresponding to a different speaker than the group of utterances from which the simulated addition utterance 406 was selected.

[0078] In general, the selection technique illustrated by FIG. 4A can be applied in different training iterations to utterances from different combinations of speakers. For example, in a first training iteration, the simulated addition utterances may be spoken by a first speaker, and the simulated verification utterances may also be spoken by the first speaker. In a second training iteration, the simulated addition utterances may be spoken by a second speaker, and the simulated verification utterances may be spoken by a third speaker. Then, in a third training iteration, the simulated addition utterances may be spoken by a first speaker, and the simulated verification utterances may be spoken by a second speaker. In some implementations, a selection algorithm may be used that does not randomly select a group of utterances 410a-n, but instead deterministically selects a group of utterances 410a-n in a manner that generates different permutations or maximizes the number of permutations in the input samples 402 between the speakers of the simulated verification utterances and the addition utterances. As a simple example, if three groups A, B, and C of utterances from three different training speakers were available in the utterance pool 408, nine different input samples 402 may be generated for nine training iterations, namely (A,A), (A,B), (A,C), (B,A), (B,B), (B,C), (C,A), (C,B), and (C,C). Training iterations may also occur with pairs of these same groups, but with different utterances selected within the groups.

[0079] One history of the training approach described herein, in contrast to other approaches that involve training a neural network to classify inputs as belonging to a particular speaker among a preselected number of speakers, may be to use a larger number of different speakers to train the network. Furthermore, there is no minimum number of training utterances required per training speaker (other than the one or more utterances actually used per training speaker) to ensure reliable training, because the network is not trained for a particular speaker, but instead is trained based on whether a given input sample 402 has a matching or non-matching speaker among the simulated verification and enrollment utterances.

[0080] FIG. 4B shows a schematic diagram 400b of a shuffling technique for selecting utterances for input samples during neural network training. As shown in this figure, the samples in a batch of training samples can all come from different pools to obtain good shuffling and diversity of utterances among the training samples in the batch. The shuffling technique can result in more robust and reliable neural network training.

[0081] 5A and 5B, block diagrams of example neural networks 502, 512 that may be used in a speaker verification model are shown. In some implementations, either of the neural networks 502, 512 may be used to implement the techniques described with respect to Figures 1-4B and 6, including the training techniques described with respect to Figures 2-4B.

[0082] The architecture of the deep neural network 502 in FIG. 5A includes a locally connected layer 504 followed by one or more fully connected hidden layers 506a-n. The locally connected layer 504 and the fully connected layers 506a-n may have rectified linear units (ReLUs). The last layer of the network 502 is a fully connected linear layer 508, which outputs a speaker representation 510a based on an input utterance (or frame of utterance) 503a. The last layer 502 before the representation 510a is, in some implementations, a linear layer for mapping non-negative activations to a full space and determining a projection. The full space refers to the notion that ReLu activations can be functions such as y=max(x,0). Thus, the activations (y) that form the speaker representation may always be positive vectors. If such activation functions are modified by a linear activation function y=x, the speaker representation can be generated as a vector with potentially positive and negative values. The latter, for example, when followed by a cosine distance comparison function, may be a more appropriate representation of the speaker.

[0083] The configuration of the neural network 502 is generally capable of processing training utterances of a fixed length, or a fixed number of frames of an utterance. When the neural network 502 is trained and later used during run-time in the joining and validation phases, the utterance may be appropriately pruned or padded to ensure that the utterance has the required fixed length for processing by the neural network 502. As a result, the neural network 502 can compute a speaker representation in a single pass, e.g., a single forward propagation through the deep neural network 502. This allows the speaker representation to be generated with lower latency than techniques that involve sequential processing of different parts of an utterance.

[0084] Next, the neural network 512 shown in FIG. 5B is a recurrent neural network. Unlike the architecture of the neural network 502, the neural network 512 can process input utterances of variable length. For example, the utterance 503b may be a training utterance, an enrollment utterance, or a validation utterance, depending on the context in which the neural network 512 is used. The utterance 503b may be divided into multiple frames, which may have a fixed length. The number of frames input to the neural network 512 may be a function of the overall length of the utterance 503b. In other words, a long utterance may have more frames, and a short utterance may have fewer frames. The frames of the utterance 503b are input to a long short-term memory (LSTM) layer 516. One or more additional hidden layers may follow the LSTM layer 516. The last layer of the network 512 is again a fully connected linear layer 518. In some cases, a fully connected linear layer 518 may map non-negative activations into the entire space and output the speaker representation 510b by determining a projection. Because the neural network 512 can handle variable length utterances, it may be well suited for text-independent speaker verification, where words or phrases of an utterance are not predefined and may vary between different utterances.

[0085] Although the neural networks 502 and 512 shown in Figures 5A and 5B are shown as having particular configurations, the neural networks that may be used in the techniques described herein are not limited by these examples. For example, the hidden topology of the neural network may have different numbers and arrangements of layers, which may or may not include fully connected layers, locally connected layers, or any recurrent layers such as long short-term memory layers. In some implementations, the neural network may be a conventional neural network.

[0086] Figure 6 is a flow diagram of an example process 600 for verifying an utterance using a speaker verification model and a neural network trained according to the techniques described herein. Process 600 generally corresponds to the verification stages (stages E-G) shown in Figure 1. The neural network referenced in Figure 6 may, in some implementations, be trained according to the techniques described with respect to Figures 2-4B and may have a structure as shown in Figures 5A or 5B.

[0087] At step 602, a vocal utterance can be received from a user of the computing device. For example, the user may want to unlock their smartphone or perform some other function on the computing device. However, the smartphone may require the user to authenticate the user before it can be unlocked or the desired function can be performed. In some implementations, this may be performed based on characteristics of the user's voice using a speaker verification model of the smartphone. The smartphone may prompt the user to speak a verification utterance, which may be received and recorded by the smartphone at step 602.

[0088] At step 604, the smartphone accesses a neural network to generate a speaker representation based on the received utterance. The neural network may be stored locally on the smartphone or may be accessed on a remote computing system, for example, via an application programming interface (API). The neural network may be trained according to techniques described herein and may have been trained based on samples of data including a simulated verification utterance and a plurality of simulated joining utterances. The neural network may be configured to process data characterizing an entire utterance through the neural network in a single pass. At step 606, the data characterizing the received utterance is provided as an input to the neural network. The neural network processes the input and generates a speaker representation indicative of distinctive characteristics of the user's voice.

[0089] At step 608, a speaker model is accessed at the smartphone. The speaker model may represent distinctive features of the subscribed user's voice. In some implementations, the speaker model may be based on an average of multiple speaker expressions generated by a neural network from data characterizing each utterance of the subscribed user. At step 610, the speaker expressions generated at step 606 based on the verification utterances are compared to or otherwise evaluated with respect to the speaker model. In some implementations, the comparison or other evaluation is performed by a speaker verification model on the user's smartphone. The speaker verification model may determine a similarity distance or other measure between the speaker model and the speaker expressions for the verification utterances. Based on the similarity distance or other measure, the speaker verification model may authenticate the user if the user's voice is sufficiently similar to the subscribed user's voice. Otherwise, if the similarity of the user's voice does not meet at least a threshold similarity score with respect to the subscribed user's voice, the speaker verification model may generate an indication that the user is not authenticated.

[0090] In some implementations, if the speaker verification model determines with sufficient confidence that the verification utterance was spoken by the subscribed speaker, then the speaker model for the subscribed user may be updated based on the verification utterance. For example, consider how the device may respond to the following three verification utterances: The similarity score for the first of the three verification utterances is less than a first threshold such that the speaker verification model rejects the identity of the user who spoke the first verification utterance (e.g., the device may reject an unlock in response to the first verification utterance). The similarity score for the second of the three verification utterances may meet the first threshold such that the identity of the user who spoke the second verification utterance is accepted. However, the similarity score for the second verification utterance is high enough to update the speaker model for the subscribed user based on the second verification utterance. Finally, the similarity score for a third one of the verification utterances meets a first threshold such that the identity of the user who spoke the third verification utterance is accepted (e.g., may perform a first set of actions, such as unlocking the device), and a higher second threshold such that the speaker model for the enrolled user may be updated based on the third verification utterance. The speaker model may be updated by combining (e.g., averaging) the speaker representation generated by the neural network for the third verification utterance with other speaker representations from the enrolled utterances of the user that were used to generate the speaker model in the first instance.

[0091] At step 612, the smartphone may then take an action based on whether the user has been authenticated. For example, in response to determining that the user who provided the utterance is a subscribed user, the smartphone may be activated or unlocked. However, if it is determined that the user who provided the utterance is not a subscribed user or is not one of a plurality of subscribed users, the smartphone may remain locked or may be blocked from performing one or more functions that the user would otherwise have selected to perform. In another application, the speaker verification techniques described herein may be used by a user device (e.g., a smartphone, a notebook computer, a wearable device) to reject speech input detected by the device from an unauthenticated user (e.g., a voice user not subscribed to the device). For example, when the device is in an unlocked state, the device may listen for a voice command spoken by an authenticated user of the device indicating an action the user would like the device to perform (e.g., "Navigate to the football game" or "Play my music collection"). In some implementations, the device may only perform the requested action indicated by the voice command if it can determine that the voice command was spoken by the authenticated user, and thus may, for example, reject chatter from other non-authenticated users.

[0092] 7 illustrates an example of a computing device 700 and a mobile computing device that can be used to implement the techniques described herein. The computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular smartphones, smartphones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0093] The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connecting to the memory 704 and multiple high-speed expansion ports 710, and a low-speed interface 712 connecting to a low-speed expansion port 714 and the storage device 706. Each of the processor 702, the memory 704, the storage device 706, the high-speed interface 708, the high-speed expansion port 710, and the low-speed interface 712 are interconnected using various buses, which may be mounted on a common motherboard or in other manners as desired. The processor 702 is capable of processing instructions for execution within the computing device 700, including instructions stored in the memory 704 or the storage device 706 for displaying graphical information for a GUI on an external input / output device, such as a display 716 connected to the high-speed interface 708. In other implementations, multiple processors and / or multiple buses may be used along with multiple memories and types of memory as desired. Multiple computing devices may also be connected (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system) with each device providing a portion of the required operations.

[0094] The memory 704 stores information within the computing device 700. In some implementations, the memory 704 is a volatile memory unit or units. In some implementations, the memory 704 is a non-volatile memory unit or units. The memory 704 may also be another form of computer readable medium, such as a magnetic or optical disk.

[0095] The storage device 706 can provide mass storage to the computing device 700. In some implementations, the storage device 706 can be or include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The computer program product can also be tangibly embodied in a computer or machine-readable medium, such as the memory 704, the storage device 706, or a memory on the processor 702.

[0096] The high-speed interface 708 manages bandwidth-intensive operations for the computing device 700, and the low-speed interface 712 manages less bandwidth-intensive operations. Such functional allocations are merely exemplary. In some implementations, the high-speed interface 708 is connected to the memory 704 (e.g., through a graphics processor or accelerator), the display 716, and to a high-speed expansion port 710. The high-speed expansion port 710 may accept various expansion cards (not shown). In such implementations, the low-speed interface 712 is connected to the storage device 706 and to a low-speed expansion port 714. The low-speed expansion port 714 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may connect to one or more input / output devices such as a keyboard, pointing device, scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0097] Computing device 700 may be implemented in a number of different forms as shown in this figure. For example, it may be implemented as a standard server 720 or multiple times in a group of such servers. It may also be implemented in a personal computer such as a laptop computer 722. It may also be implemented as part of a rack server system 724. Alternatively, components from computing device 700 may be combined with other components in a mobile device (not shown), such as mobile computing device 750. Each such device may include one or more of computing device 700 and mobile computing device 750, and the entire system may be made up of multiple computing devices in communication with each other.

[0098] Mobile computing device 750 includes a processor 752, memory 764, and input / output devices such as a display 754, a communication interface 766, and a transceiver 768, among other components. Mobile computing device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the processor 752, memory 764, display 754, communication interface 766, and transceiver 768 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other manners as desired.

[0099] The processor 752 may execute instructions within the mobile computing device 750, including instructions stored in the memory 764. The processor 752 may be implemented as a chip set consisting of chips including separate and multiple analog and digital processors. The processor 752 may provide coordination of other components of the mobile computing device 750, such as control of the user interface, applications executed by the mobile computing device 750, and wireless communications by the mobile computing device 750.

[0100] The processor 752 may communicate with a user through a control interface 758 and a display interface 756 connected to a display 754. The display 754 may be, for example, a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 756 may comprise appropriate circuitry for driving the display 754 to provide graphical and other information to a user. The control interface 758 may receive commands from a user and translate the commands for transmission to the processor 752. Additionally, an external interface 762 may provide communication with the processor 752 to enable near-field communication of the mobile computing device 750 with other devices. The external interface 762 may, for example, provide wired communication in some implementations or wireless communication in other implementations, and multiple interfaces may also be used.

[0101] The memory 764 is stored within the mobile computing device 750. The memory 764 may be implemented as one or more of a computer readable medium or media, a volatile memory unit or unit, or a non-volatile memory unit or unit. An expansion memory 774 may also be provided and connected to the mobile computing device 750 through an expansion interface 772. The expansion interface 772 may include, for example, a Single In Line Memory Module (SIMM) card interface. The expansion memory 774 may provide additional storage space for the mobile computing device 750 or may store applications or other information for the mobile computing device 750. In particular, the expansion memory 774 may include instructions that perform or complement the processes described above and may include secure information. Thus, for example, the expansion memory 774 may be provided as a security module for the mobile computing device 750 and may be programmed with instructions that allow secure use of the mobile computing device 750. Furthermore, secure applications may be provided with additional information via a SIMM card, such as placing identifying information on the SIMM card in an unhackable manner.

[0102] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as described below. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The computer program product may be a computer or machine readable medium, such as memory 764, expansion memory 774, or memory on processor 752. In some implementations, the computer program product may be received in a propagated signal, for example, on transceiver 768 or external interface 762.

[0103] The mobile computing device 750 may communicate wirelessly through the communication interface 766. The communication interface 766 may include digital signal processing circuitry, if necessary. The communications interface 766 may provide communications under various modes or protocols, such as Global System for Mobile communications (GSM), Short Message Service (SMS), Enhanced Messaging Service (EMS), or Multimedia Messaging Service (MMS), code division multiple access (CDMA), time division multiple access (TDMA), Personal Digital Cellular (PDC), Wideband Code Division Multiple Access (WCDMA), CDMA2000, or General Packet Radio Service (GPRS), among others. Such communications may be performed using radio frequencies, for example, via the transceiver 768. Additionally, shortwave communications may be performed using, for example, Bluetooth, WiFi, or other such transceivers (not shown). Additionally, a Global Positioning System (GPS) receiver module 770 may provide additional navigation position related wireless data to the mobile computing device 750, which may be used as needed by applications executing on the mobile computing device 750.

[0104] The mobile computing device 750 may also communicate audibly using an audio codec 760, which may receive spoken information from a user and convert it into usable digital information. The audio codec 760 may also generate audible sounds for a user through a speaker, for example, at a handset of the mobile computing device 750. Such sounds may include sounds from a voice call, may include recorded sounds (e.g., voice messages, music files, etc.), and may include sounds generated by applications running on the mobile computing device 750.

[0105] The mobile computing device 750 may be implemented in a number of different forms, as shown in this figure, such as a cellular smartphone 780, or as part of a smartphone 782, personal digital assistant, or other similar mobile device.

[0106] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor. The at least one programmable processor may be special purpose or general purpose and may be coupled to transmit and receive data and instructions from a storage system, at least one input device, and at least one output device.

[0107] These computer programs (also known as programs, software, software applications or codes) include machine instructions for a programmable processor and may be implemented in high level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0108] To provide for interaction with a user, the systems and techniques described herein can be implemented in a computer having a display device (e.g., a cathode ray tube (CRT) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can be used to provide for interaction with a user. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, audible feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0109] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein), or includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wideband network (WAN), and the Internet.

[0110] The computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0111] Although various implementations have been described in detail above, other modifications are possible. Additionally, the logic flow depicted in the figures does not require the particular order shown, or sequential order, to achieve desirable results. Additionally, other steps may be provided or steps may be removed from the described flow, and other components may be added to or removed from the described system. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0112] 121 Vocal Pool 122 training samples 122a Simulated verification utterances 122b Simulated recruitment vocalizations 130 Network 140 trained neural networks 141a Matching Speakers 141b Unmatched Speakers 144 Speaker Verification Model 152 Joining utterance(s) 154 Verification Utterance 156 Verification Results

Claims

1. A computer-implemented method executed on data processing hardware, the method, when executed by the data processing hardware, causing the data processing hardware to: Receiving a plurality of training samples, each training sample comprising: Training verification vocalization, training utterances spoken by corresponding speakers; a first classification for the training samples indicating whether a speaker of the training verification utterances is the same as or different from the corresponding speaker who spoke the training enrollment utterances; and training a neural network with the plurality of training samples, For each training sample, processing audio signals characterizing the training verification utterances to generate first training speaker representations for the training verification utterances using the neural network; processing the audio signals characterizing the training enrollment utterances to generate second training speaker representations using the neural network; determining a second classification for the training sample based on the first training speaker expressions and the second training speaker expressions, the second classification for the training sample indicating whether the speaker of the training verification utterance is the same as or different from the corresponding speaker who spoke the training enrollment utterance; adjusting parameters of the neural network based on the first classification of the training samples and the second classification determined for the training samples; and transmitting the trained neural network over a network to a user device; 23. A computer-implemented method for causing a computer to perform the operations comprising:

2. The user device, receiving an audio signal, the audio signal representing a user of the user device speaking a plurality of training enrollment utterances recorded by the user device; generating a reference speaker model associated with the user, the reference speaker model characterizing distinctive features of the user's voice, using the trained neural network configured to receive as input the audio signal representative of the user speaking the plurality of training enrollment utterances; storing said reference speaker model on memory hardware of said user device; The computer-implemented method of claim 1 , configured to:

3. 3. The computer-implemented method of claim 2, wherein the trained neural network comprises a trained neural network comprising a long short-term memory (LSTM) layer configured to receive as input the audio signal representative of the user speaking the plurality of training enrollment utterances.

4. The user device, obtaining a plurality of audio frames representing a first utterance; generating a speaker representation for the first utterance using the trained neural network, the speaker representation representing distinctive characteristics of a speaker of the first utterance; and determining whether a similarity score between the speaker representation for the first utterance and the reference speaker model stored on the memory hardware of the user device satisfies a similarity score threshold; authenticating the speaker of the first utterance as the user associated with the reference speaker model based on determining that the similarity score satisfies the similarity score threshold. The computer-implemented method of claim 3 , configured to:

5. The trained neural network comprises: receiving as input the output of said LSTM layer; As an output, generate the speaker representation for the first utterance. The computer-implemented method of claim 4 , further comprising a fully connected layer configured to:

6. 5. The computer-implemented method of claim 4, wherein the user device is configured to transition operation of the user device from a lower power state to a more fully characterized state in response to authenticating the speaker of the first utterance as the user associated with the reference speaker model.

7. The computer-implemented method of claim 1 , wherein the training verification utterances and the training enrollment utterances comprise the same predefined phrase.

8. 2. The computer-implemented method of claim 1, wherein the comparison of the first classification of the training sample to the second classification determined for the training sample comprises a cosine distance between a vector of values ​​for the first classification and a vector of values ​​for the second classification.

9. The computer-implemented method of claim 1 , wherein the user device comprises a smartphone.

10. The computer-implemented method of claim 1 , wherein the data processing hardware is located on a remote computing device.

11. 1. A system comprising: Data processing hardware; memory hardware in communication with the data processing hardware and storing instructions that, when executed by the data processing hardware, cause the data processing hardware to: receiving a plurality of training samples, each training sample comprising: Training verification vocalization, training utterances spoken by corresponding speakers; a first classification for the training samples indicating whether a speaker of the training verification utterances is the same as or different from the corresponding speaker who spoke the training enrollment utterances; and training a neural network with the plurality of training samples, For each training sample, processing audio signals characterizing the training verification utterances to generate first training speaker representations for the training verification utterances using the neural network; processing the audio signals characterizing the training enrollment utterances to generate second training speaker representations using the neural network; determining a second classification for the training sample based on the first training speaker expressions and the second training speaker expressions, the second classification for the training sample indicating whether the speaker of the training verification utterance is the same as or different from the corresponding speaker who spoke the training enrollment utterance; adjusting parameters of the neural network based on the first classification of the training samples and the second classification determined for the training samples; and transmitting the trained neural network over a network to a user device; and a memory hardware for causing the memory hardware to perform operations comprising: A system comprising:

12. The user device, receiving an audio signal, the audio signal representing a user of the user device speaking a plurality of training enrollment utterances recorded by the user device; generating a reference speaker model associated with the user, the reference speaker model characterizing distinctive features of the user's voice, using the trained neural network configured to receive as input the audio signal representative of the user speaking the plurality of training enrollment utterances; storing said reference speaker model on memory hardware of said user device; The system of claim 11 configured to:

13. 13. The system of claim 12, wherein the trained neural network comprises a trained neural network comprising a long short-term memory (LSTM) layer configured to receive as input the audio signal representative of the user speaking the plurality of training enrollment utterances.

14. The user device, obtaining a plurality of audio frames representing a first utterance; generating a speaker representation for the first utterance using the trained neural network, the speaker representation representing distinctive characteristics of a speaker of the first utterance; and determining whether a similarity score between the speaker representation for the first utterance and the reference speaker model stored on the memory hardware of the user device satisfies a similarity score threshold; authenticating the speaker of the first utterance as the user associated with the reference speaker model based on determining that the similarity score satisfies the similarity score threshold. The system of claim 13 configured to:

15. The trained neural network comprises: receiving as input the output of said LSTM layer; As an output, generate the speaker representation for the first utterance. The system of claim 14 , further comprising a fully connected layer configured to:

16. 15. The system of claim 14, wherein the user device is configured to transition operation of the user device from a low power state to a more fully characterized state in response to authenticating the speaker of the first utterance as the user associated with the reference speaker model.

17. The system of claim 11 , wherein the training verification utterances and the training enrollment utterances comprise the same predetermined phrase.

18. 12. The system of claim 11, wherein the comparison of the first classification of the training sample to the second classification determined for the training sample comprises a cosine distance between a vector of values ​​for the first classification and a vector of values ​​for the second classification.

19. The system of claim 11 , wherein the user device comprises a smartphone.

20. The system of claim 11 , wherein the data processing hardware is located on a remote computing device.

Images