Speaker recognition performance assessment

Abstract

To provide a method for evaluating performance of a verification model.SOLUTION: A method (300) for evaluating a verification model (146) includes a step of receiving first and second sets of verification results (148), in which each verification result indicates whether a primary or alternative model confirms identity of a user (10) as a registered user. The method includes the steps of: identifying each verification result in the first and second sets, including a performance metric (212); determining a first score (222) for the primary model based on the number of verification results identified in the first set, including a performance metric; and further determining a second score for the alternative model based on the number of identified verification results in the second set, including the performance metric. The method further includes a step of determining, based on the first score and the second score, whether verification capability of the alternative model is better than the verification capability of the primary model.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] This disclosure relates to assessing speaker recognition performance. [Background technology]

[0002] In recent years, computing devices that offer multiple user input modalities have become increasingly popular. For example, smartphones and other user devices include speech recognition services that allow users to provide voice input to the device as an alternative to typing or pointing input. Voice-based input can be more convenient in some situations as a hands-free means of interacting with a computing device. To protect against privacy and security breaches, some devices require that a user's identity be verified before performing an action based on the voice input. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a method for evaluating performance of a verification model. The method includes receiving, at data processing hardware, a first set of verification results, each verification result of the first set indicating whether a primary verification model executing on a respective user device verified the identity of the respective user as one of one or more registered users of the respective user device during a respective interaction between the respective user and the respective user device. The method also includes receiving, at the data processing hardware, a second set of verification results, each verification result of the second set indicating whether an alternate verification model executing on the respective user device verified the identity of the respective user as one of one or more registered users of the respective user device during the respective interaction between the respective user and the respective user device. The method further includes identifying, by the data processing hardware, each verification result in the first and second sets of verification results, the performance metric indicating one of the success or failure of each of the primary or alternate verification models to verify the identity of the respective user as one of the one or more registered users during the respective interaction between the respective user and the respective user device. The method further includes determining, by the data processing hardware, a first performance score for the primary verification model based on a number of verification results identified in the first set of verification results, the number including the performance metric. The method also includes determining, by the data processing hardware, a second performance score for the alternative primary verification model based on a number of verification results identified in the second set of verification results, the number including the performance metric. The method further includes determining, by the data processing hardware, whether the verification ability of the alternative verification model is superior to the verification ability of the primary verification model based on the first performance score and the second performance score.The method also includes replacing, by data processing hardware, the primary verification model executed on at least one respective user device with the alternative verification model when the verification capability of the alternative verification model is superior to the verification capability of the primary verification model.

[0004] Implementations of the present disclosure may include one or more of the following features. In some implementations, the method also includes receiving, at the data processing hardware, a third set of verification results, where each verification result in the third set of verification results indicates whether a control verification model executing on the respective user device confirmed the identity of the respective user as one of the one or more registered users of the respective user device during a respective interaction between the respective user and the respective user device. In these implementations, the method further includes identifying, by the data processing hardware, each verification result in the third set of verification results, the control verification model including a performance metric indicating one of the success or failure of the control verification model to confirm the identity of the respective user during a respective interaction between the respective user and the respective user device. In these implementations, the method further includes determining, by the data processing hardware, a third performance score for the control verification model based on the number of verification results identified in the third set of verification results, the performance metric included, and determining, by the data processing hardware, whether the verification capability of the alternative verification model is superior to the verification capability of the control verification model based on the second performance score and the third performance score. In these implementations, replacing the primary verification model executed on each user device with an alternative verification model includes replacing the primary verification model executed on each user device with the alternative verification model when the verification capability of the alternative verification model is superior to the verification capabilities of both the primary verification model and the control verification model.

[0005] In some examples, the method includes initially allocating, by data processing hardware, a primary verification model to execute on a first plurality of user devices and allocating an alternate verification model to execute on a second plurality of user devices. In these examples, replacing the primary verification model to execute on at least one respective user device includes reallocating the alternate verification model to execute on at least one respective user device among the first plurality of user devices in place of the primary verification model. The first plurality of user devices may be more than the second plurality of user devices.

[0006] Another aspect of the present disclosure provides a system for evaluating performance of a verification model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a first set of verification results, each verification result of the first set of verification results indicating whether a primary verification model executing on a respective user device confirms the identity of the respective user as one of one or more registered users of the respective user device during a respective interaction between the respective user and the respective user device. The operations also include receiving a second set of verification results, each verification result of the second set of verification results indicating whether an alternate verification model executing on the respective user device confirms the identity of the respective user as one of one or more registered users of the respective user device during an interaction between the respective user and the respective user device. The operations further include identifying each verification result in the first set and the second set of verification results, the each including a performance metric indicating whether the primary verification model or the alternate verification model was able to verify the identity of the respective user as one of the one or more registered users during a respective interaction between the respective user and the respective user device. The operations further include determining a first performance score for the primary verification model based on the number of verification results identified in the first set of verification results that include the performance metric. The operations also include determining a second performance score for the alternate primary verification model based on the number of verification results identified in the second set of verification results that include the performance metric. The operations further include determining whether the verification ability of the alternate verification model is superior to the verification ability of the primary verification model based on the first performance score and the second performance score.The operations also include replacing the primary verification model executed on at least one respective user device with the alternate verification model when the verification capability of the alternate verification model is superior to the verification capability of the primary verification model.

[0007] In some implementations, the operations also include receiving a third set of verification results, each verification result in the third set of verification results indicating whether a control verification model executing on the respective user device confirmed the identity of the respective user as one of the one or more registered users of the respective user device during a respective interaction between the respective user and the respective user device. In these implementations, the operations further include identifying each verification result in the third set of verification results including a performance metric indicating one of the success or failure of the control verification model to confirm the identity of the respective user during a respective interaction between the respective user and the respective user device. In these implementations, the operations further include determining, by data processing hardware, a third performance score for the control verification model based on the number of verification results identified in the third set of verification results that include the performance metric, and determining, by data processing hardware, whether the verification capability of the alternative verification model is superior to the verification capability of the control verification model based on the second performance score and the third performance score. In these implementations, replacing a primary verification model executed on each user device with an alternative verification model includes replacing a primary verification model executed on each user device with an alternative verification model when the verification capability of the alternative verification model is superior to the verification capabilities of both the primary verification model and the control verification model.

[0008] In some examples, the operations include initially allocating a primary verification model to execute on a first plurality of user devices and allocating an alternate verification model to execute on a second plurality of user devices. In these examples, the operation of replacing the primary verification model to execute on at least one respective user device includes the operation of reallocating the alternate verification model to execute on at least one respective user device among the first plurality of user devices in place of the primary verification model. The first plurality of user devices may be more than the second plurality of user devices.

[0009] Implementations of the system or method may include any one or more of the following features: In some implementations, none of the verification results received in the first set and the second set of verification results include a user identifier identifying the respective user; In some configurations, none of the verification results received in the first set and the second set of verification results include audio data associated with a respective interaction between the respective user and the respective device; In operation, a primary verification model is trained with a first set of training data, and an alternate verification model is trained with a second set of training data that is different from the first set of training data; The primary verification model may include a first neural network, and the alternate verification model may include a second neural network having a neural network architecture different from the first neural network.

[0010] In some examples, the performance metrics include a false reject metric that indicates that each of the primary or alternate verification models falsely rejected identifying the respective user as one of one or more registered users of the respective user device. The false denial metric may include one of: a punt metric indicating that each of the primary or alternate verification models authorized the respective user for guest privilege during a respective interaction with the respective user device; a double punt metric indicating that each of the primary or alternate verification models authorized the same respective user for guest privilege during a respective interaction with the respective user device immediately after the primary or alternate verification model authorized the same respective user for guest privilege during a previous respective interaction with the respective user device; and a punt and re-query metric indicating that each of the primary or alternate verification models authorized the respective user for guest privilege during a respective interaction with the respective user device immediately after the primary or alternate verification model authorized the respective user for guest privilege during a previous respective interaction with the respective user device, when the respective interaction corresponds to the same respective user requesting authorized privilege.

[0011] In some implementations, the performance metrics include a false accept metric indicating that each of the primary or alternate verification models falsely accepted a respective user as one of one or more registered users of the respective user device, where the false accept metric may include a surrogate imposter acceptance metric indicating that each of the primary or alternate verification models determined respective verification scores associated with at least two registered users of the respective user device that met a verification threshold.

[0012] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0013] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary speech environment for a confirmation analyzer. [Figure 1B] FIG. 1 is a schematic diagram of an exemplary speech environment for a confirmation analyzer. [Figure 1C] FIG. 1 is a schematic diagram of an exemplary speech environment for a confirmation analyzer. [Figure 1D] FIG. 1 is a schematic diagram of an exemplary speech environment for a confirmation analyzer. [Figure 2A] FIG. 1 is a schematic diagram of an exemplary confirmation analyzer for evaluating the performance of a confirmation model. [Figure 2B] FIG. 1 is a schematic diagram of an exemplary confirmation analyzer for evaluating the performance of a confirmation model. [Figure 3] 1 is a flow diagram of an exemplary sequence of operations for a method of evaluating the performance of a validation model. [Figure 4] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. [Figure 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0014] Like reference symbols in the various drawings indicate like elements.

[0015] Generally, a verification process refers to a process that identifies whether an entity is authorized to perform some task or action based on one or more characteristics associated with the entity. When the entity is a person, the characteristics included in the verification process are often the person's biomarkers. For example, the verification process uses phonetic features extracted from speech or audio spoken by the person to verify that the person is authorized to perform a particular task. Besides speech, other examples of biomarkers that a verification process may use include fingerprints, retinal features, facial features (e.g., in facial recognition), and / or handwriting (e.g., signature). With any of these biomarkers, the verification process typically compares the current biomarker input with a previous biomarker input (i.e., an enrollment biomarker) to determine whether the current biomarker input matches or closely resembles the previous biomarker input. When the current biomarker input matches or closely resembles the previous biomarker input, the verification process confirms the person's identity taking the current biomarker input into account and allows the person to perform a particular task requiring identity verification.

[0016] A speaker recognition system may perform a verification process to verify whether two or more spoken utterances originate from the same speaker. To perform this verification, a verification system associated with the speaker recognition system compares audio samples (e.g., two audio samples) to determine whether a first audio sample corresponding to a first utterance spoken by a speaker matches or is highly similar to a second audio sample corresponding to another spoken utterance. When the first utterance matches or is highly similar to the other spoken utterance, the verification system determines that both utterances are likely from the same speaker. On the other hand, when the first utterance does not match or is not very similar to the other spoken utterance, the verification system determines that each utterance is likely from a different speaker. In some examples, the speaker recognition system compares text-dependent audio samples to determine a match. In other examples, the speaker recognition system compares two text-independent audio samples to determine whether the two audio samples originate from the same speaker. Often, to perform speaker verification, a user of a speaker recognition system provides one or more spoken utterances to the verification system to register or enroll the user in the speaker enrollment system. A user who enrolls in a speaker enrollment system may be referred to as an "enrolled user," and thus the terms "enrolled user" and "enrolled user" may be used interchangeably. By enrolling in a speaker recognition system, the user's enrollment may authorize the user to perform specific tasks related to the speaker recognition system. Furthermore, the user's enrollment allows the verification system to later verify the user's identity using the enrollment utterances (i.e., the spoken utterances provided to enroll the user).For example, after enrolling in a speaker recognition system as an authorized user of a computing device, when the user submits a spoken utterance to the computing device, the speaker recognition system (e.g., a verification system) compares the submitted spoken utterance with one or more enrollment utterances to determine whether the user is an authorized user.

[0017] To perform verification, a verification system may use a verification model to generate a prediction of whether the speaker of an utterance is an authorized user or an unauthorized user. Unfortunately, automated systems are not without drawbacks, and a verification model may incorrectly identify the speaker of an utterance as an authorized user when the speaker is not an authorized user, or as an unauthorized user when the speaker is an authorized user. When a system identifies a speaker of an utterance as an authorized user when the speaker is not an authorized user, this misidentification is referred to as a false acceptance of the speaker. On the other hand, when a system identifies a speaker of an utterance as an unauthorized user when the speaker is actually an authorized user, this misidentification is referred to as a false rejection of the speaker. Because verification systems may have some performance issues related to false acceptance and / or false rejection, it may be advantageous for verification system providers to collect feedback on or assess the performance of the verification system. However, when a verification system is already implemented (e.g., deployed on a computing device), evaluating the performance of a verification system becomes more complicated.

[0018] Typical approaches to evaluating the performance of verification systems are generally cumbersome and / or involve some degree of manual input for review to ensure the verification system is being properly evaluated. In one such approach, volunteers may make calls from various devices, identify themselves according to some identifier (e.g., a personal identification number (PIN)), and submit recordings that become labeled with the speaker identifier. With a collection of these recordings, the performance of the verification system (e.g., a verification model) may be evaluated using some number of these recordings to determine how well the verification system performs verification against known speaker identities. A drawback of this approach is that volunteers are generally paid for their time, and evaluation set records are usually audited or supervised to ensure accurate evaluation. This can result in a costly and time-consuming process.

[0019] Another approach that can determine the performance of a verification system is to collect user data from devices that use the verification system. In this approach, user data from devices that use the verification system is further assigned a speaker identifier that masks all information about the user. For example, one approach might assume that audio from each device that uses the verification system is associated with a specific speaker and assign a personal identification number (PIN) to the audio as it is collected to remove all user associations. Much like the call-in volunteer approach, audio data collected from devices that use the verification system can then be used as an evaluation data set to evaluate the performance of the verification system. However, of course, even if this process removes all user associations and assumes user consent, verification system providers do not want to assume control of user data and be responsible for all potential security issues that could threaten the security of customer data. Furthermore, any time a user or customer provides their data, even to a trusted source, the customer relinquishes control over their data and runs the risk of being unable to prevent any downstream issues (e.g., security issues) related to this data. This approach is therefore subject to the reality that a single device may contain multiple users or speakers, but may also have implications for privacy and / or security issues.

[0020] To overcome the problems plaguing various techniques for evaluating the performance of a verification system, verification system providers may instead leverage information collected about or during the verification process by the verification system. In other words, when the verification system verifies whether a speaker is an enrolled / authorized user, the verification process generates data (e.g., metadata) regarding the interaction between the speaker and the verification system. For example, the verification process generates information, such as an event log, regarding the interactions during the verification session. To illustrate, an enrolled speaker may speak an utterance to a device to perform an action requiring authorization. When the device receives this spoken utterance requiring authorization, the verification system determines whether the speaker is enrolled in the device, and, when the verification system verifies the speaker, either enables the device to perform the function or generates some type of response indicating that the speaker cannot be verified. With respect to this verification session, the device and / or verification system may generate verification data indicating that the verification process was initiated and that the speaker was either verified and accepted, or not verified and rejected. By collecting verification data related to the verification process, the verification data generated does not include the speaker's identity / identifier or any audio data / voice features associated with the speaker, while still providing important insight into the performance of the verification system. More specifically, the verification data can be used to build metrics that indicate the performance of the verification system, and more specifically, the verification model utilized by the verification system, without sharing any personal or biometric data associated with the speaker. By using verification data that is decoupled from user-specific information and avoiding the collection of additional evaluation audio data (e.g., via volunteer phone calls), this performance evaluation technique overcomes several shortcomings of typical performance evaluation techniques.

[0021] 1A-1D illustrate an example of a speech environment 100 including one or more users 10 communicating spoken utterances 12 to a verification device 110 (also referred to as a device 110 or a user device 110). The user 10 (i.e., the speaker of the utterance 12) may speak a query in the utterance 12 or a command to solicit a response from the device 110. Similarly, the utterance 12, or at least a portion of the utterance 12, may include an invocation phrase (e.g., a hot word, a wake word, a trigger phrase, etc.) that, when detected by the device 110, triggers the device 110 to wake up from a sleep state and begin recording subsequent audio associated with the query or command for processing by an automatic speech recognition (ASR) system running on the device 110 and / or a remote server. The device 110 is configured to capture sounds from one or more users 10 in the speech environment 100. Here, audio sounds may refer to spoken utterances 12 by user 10 that serve as invocation phrases, audible queries, commands for device 110, or audible communications captured by device 110. A voice-enabled system of or associated with device 110 may field queries regarding commands by answering the queries and / or causing the commands to be executed.

[0022] Here, device 110 is configured to detect utterances 12 and invoke a local or remote ASR system. Device 110 may correspond to any computing device associated with user 10 and capable of receiving audio signals corresponding to spoken utterances 12. Some examples of user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-readers, etc.), computers, wearable devices (e.g., smart watches), music players, casting devices, smart appliances (e.g., smart TVs) and Internet of Things (IoT) devices, remote controls, smart speakers, etc. Device 110 includes data processing hardware 112 and memory hardware 114 that communicates with data processing hardware 112 and stores instructions that, when executed by data processing hardware 112, cause data processing hardware 112 to perform one or more operations related to speech detection or some other form of speech / voice processing (e.g., speaker verification).

[0023] In some examples, device 110 includes one or more applications (i.e., software applications), and each application may utilize one or more voice processing systems (e.g., a voice recognition system, a text-to-speech system, a speaker recognition system, verification system 140, etc.) associated with device 110 to perform various functions within the application. In some implementations, device 110 may detect utterances 12 and provide data characterizing the utterances 12 to the one or more voice processing systems. For example, device 110 includes a verification application configured to verify whether a speaker 10 of an utterance 12 is an authorized user. For example, the verification application performs a speaker verification process that includes accepting or rejecting a speaker 10's identity claim based on characteristics of the speaker's voice (i.e., phonetic features) determined by one or more utterances 12 from speaker 10. In some examples, device 110 is configured with the application locally to perform local speaker verification or remotely to perform some portion of speaker verification using remote resources. Verification system 140 may perform text-dependent or text-independent speaker verification. Text-dependent speaker verification may be useful for recognizing a speaker from phonetic features extracted from an invocation phrase spoken by the speaker that is used to trigger the device 110 to wake up from a sleep state.

[0024] Device 110 further includes an audio subsystem having an audio capture device (e.g., a microphone) 116 for capturing and converting spoken utterances 12 in speech environment 100 into electrical signals. While device 110 implements a single audio capture device 116 in the illustrated example, device 110 may implement an array of audio capture devices 116 without departing from the scope of this disclosure, whereby one or more audio capture devices 116 in the array may not be physically present on device 110 but may communicate with the audio subsystem (e.g., peripherals of device 110). For example, device 110 may correspond to a vehicle infotainment system that utilizes an array of microphones located throughout the vehicle. Additionally or alternatively, device 110 also includes an audio output device (e.g., a speaker) 118 for communicating audible audio signals from device 110. For example, device 110 is configured to generate a synthesized playback signal in response to detected speech 12. In other words, utterance 12 may correspond to a query that is generated by device 110 and answered by device 110 with synthesized audio communicated via audio output device 118. By way of example, device 110 may respond to detected utterance 12 with a synthesized playback signal that informs speaker 10 that a verification process has confirmed the speaker's identity as an authorized user of device 110.

[0025] Additionally, device 110 is configured to communicate with remote system 130 via network 120. Remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). Device 110 may utilize remote resources 132 to perform various functions related to voice processing, such as voice recognition and / or speaker identification / verification. For example, device 110 is configured to perform speaker recognition using verification system 140. This system 140 may reside on device 110 (referred to as an on-device system) or may reside remotely (e.g., reside on remote system 130) but communicate with device 110. In some examples, some portions of system 140 reside locally or on-device, while other portions reside remotely. For example, verification model 146, configured to perform voice verification for verification system 140, resides remotely or locally. In some examples, verification system 140 may be combined with other voice processing systems, such as a voice recognition system, a diarization system, a text-to-speech system, etc. In some configurations, the location of verification system 140 is based on processing requirements. For example, if the size or processing requirements of system 140 are significant, system 140 may reside on remote system 130. However, when device 110 can support the size or processing requirements of system 140, one or more systems 140 may reside on device 110 using data processing hardware 112 and / or memory hardware 114.

[0026] Verification system 140 is generally configured to receive verification queries 142 from device 110 on behalf of user 10 and provide responses 144 indicating the results of the verification process performed by verification model 146. In some examples, verification model 146 receives as input the verification query 142 requiring verification and generates as output the response 144 as to whether user 10, who submitted verification query 142 to device 110, is verified (i.e., the identity of user 10 is an identity authorized to use device 110 for the purposes of verification query 142). Here, verification system 140 can perform the verification process with respect to any type of biometric used for verification, including facial features (i.e., facial recognition), vocal features (i.e., voice recognition), handwriting features (i.e., handwriting recognition), etc. 1A-1D , the verification system 140 performs a speech verification process in which the verification query 142 includes speech features characterizing the spoken utterance 12, and the verification system 140 provides a response 144 to the device 110 to indicate the results of the speech verification process performed by the verification model 146. The speech features may include a d-vector or an i-vector. In other words, the verification system 140 is configured to perform a speaker verification process using the verification model 146 to verify the identity of the speaker 10 of the utterance 12. For example, the response 144 may indicate whether the speaker 10 is enrolled in the device 110 (i.e., is an enrolled speaker) based on the utterance 12 spoken by the speaker 10.

[0027] 1A, the device 110 may use a verification system 140 to enroll one or more users 10 into the device 110. By enrolling a user 10 into the device 110, the enrollment is completed by enrolling the user 10, 10 E The device 110 acts as a kind of speaker enrollment process to identify the user as an authorized or enrolled user of the device 110.E The device 110 may be configurable such that an authorized user 10 may have access to or control various functions of the device 110, while an unauthorized user 10 not enrolled in the device 110 may be prohibited from performing one or more functions that the device 110 is capable of performing. Optionally, the device 110 may enroll multiple users 10. If there are multiple users 10, each enrolled user 10 E An adult may configure a user account on device 110 with specific permissions or rights regarding the functions of device 110. For example, a family including a husband, wife, and 9-year-old daughter owns device 110. Here, when each adult enrolls device 110, the adult allows each adult to access or control all functions of device 110, but also allows each adult to access or control all functions of device 110. E Parents may set parental controls that restrict their daughter from having full control of device 110. For example, parents may set up their daughter's account to prevent her from modifying home automation controls, such as the thermostat schedule, that are controlled by device 110. This allows an enrolled user 10 E But another enrolled user 10 E This means that the enrolled user 10 may have certain permissions or rights that overlap or differ from the permissions or rights of the enrolled user 10. Furthermore, instead of generating only the permissions of the enrolled user, the device 110 may generate the permissions of the enrolled user 10. E The device 110 may also be configured to specify permissions for a user 10 of the device 110 who is not an enrolled user 10. For example, E When not in use, the device 110 may be configured to perform limited functionality (e.g., guest mode) or to completely prevent non-enrolled users 10 from using the device 110. E Authorizing the enrolled user EFor example, a user may allow a device 110 to access resources that only that user has permission to access. For example, a user may allow at least two enrolled users 10 to access resources where one of them says the voice command "play my music playlist" that is captured by the device 110. E In a home with a particular enrolled speaker 10, the verification system 140 E The identity of the speaker 10 may be specified, allowing the device 110 to access a particular music playlist associated with the specified speaker 10, but not other enrolled users 10. E access to your music playlist.

[0028] In some configurations, the device 110 uses a verification system 140 to perform an enrollment process to enroll the user 10 as a registered speaker of the device 110. For example, a speaker recognition application associated with the verification system 140 may prompt the user 10 to speak one or more enrollment utterances 12, 12E, from which a speaking signature may be generated for the user 10. In some implementations, the enrollment utterances 12E are short phrases of, for example, one, two, three, four, or more words. The verification system 140 may prompt the user 10 to speak a predefined phrase as the enrollment utterance 12E, or the user 10 may spontaneously speak and provide an enrollment utterance 12E based on a phrase not specifically provided for the user 10. In some examples, the user 10 may speak multiple enrollment utterances 12E, each of which may be the same phrase or a different phrase. The enrollment utterance 12E may include the user 10 speaking a predefined hotword configured to trigger the device 110 to wake up from a sleep state to process spoken audio received after the predefined hotword. While the example shows the user 10 providing the spoken enrollment utterance 12E to the device 110, other examples may include one or more of the users 10 accessing the verification system 140 from another device (e.g., a smartphone) to provide the enrollment utterance 12E.

[0029] In some examples, upon receiving the enrollment utterances 12E, the verification system 140 processes the enrollment utterances 12E to generate a speaker expression for each enrollment utterance 12E. The verification system 140 may generate a speaker signature for the user 10 from all, some, or one of the speaker expressions for the enrollment utterances 12E. In some examples, the speaker signature is an average of the respective speaker expressions for the multiple enrollment utterances 12E. In other examples, the speaker signature corresponds to a particular speaker expression from a particular enrollment utterance 12E selected based on one or more criteria (e.g., based on the audio or voice quality of the audio of the selected enrollment utterance 12E). Once a speaker signature is generated for the speaker 10, the speaker signature may be stored locally on the device 110 or stored in the remote system 130 (e.g., in the remote memory hardware 136).

[0030] After enrollment, when the device 110 detects a query utterance 12, 12Q by a user 10 in the speech environment 100, the verification system 140 determines, based on the query utterance 12Q, that the speaker 10 of the query utterance 12Q is an enrolled user 10 of the device 110. E The query utterance 12Q may refer to a special type of utterance or spoken phrase, such as a text-dependent verification phrase, or more broadly to a text-independent phrase, which may include any utterance 12 spoken by a user 10 after the enrollment process for one or more users 10 is completed. Here, the verification process performed by the verification model 146 determines whether the speaker 10 of the detected query utterance 12Q is an enrolled user 10. E and identify whether the speaker 10 is an enrolled user 10. E In some examples, the verification model 146 generates a response 144 indicating whether the enrolled user 10 EThe system 140 may have access to a speaker signature, such as a d-vector or i-vector generated for the enrolled user 10, and compares the detected query utterance 12Q by the speaker 10 with the speaker signature to determine whether the query utterance 12Q corresponds to a particular speaker signature. In these examples, when the query utterance 12Q corresponds to a particular speaker signature, the system 140 determines whether the query utterance 12Q corresponds to a particular speaker signature. E It is determined that the query utterance 12Q was spoken by the speaker 10 of the enrolled user 10. E The system generates a response 144 indicating that

[0031] 1A shows a user initially enrolling in device 110 by performing an enrollment process. In other words, FIG. 1A shows at least one enrollment utterance 12E being sent by user 10 to device 110 to enroll in device 110. Following the enrollment process, user 10 speaks query utterance 12Q to device 110. When device 110 receives query utterance 12Q, device 110 determines whether confirmation model 146 confirms that speaker 10 is enrolling user 10. E The query utterance 12Q is communicated to the verification system 140 to enable verification that the speaker 10 is indeed the enrolled user 10. E When verifying that the speaker 10 is an enrolled user 10 of the device 110, the verification system 140 E Once verified, the speaker 10 may use the device 110 or may request the enrollment of the user 10. EThe query utterance 12Q may access any aspect of the computing functionality provided by the device 110 that the enrolled user 10 has permission to access, or more specifically, that has been designated for the user 10. In some examples, the query utterance 12Q includes a hotword followed by a query. In these examples, the verification model 146 may verify the speaker 10 based on only the portion of the audio corresponding to the hotword, only the portion of the audio corresponding to the query, or the entire audio including both the hotword and the query. In further examples, the query utterance 12Q may verify the speaker 10 based on only the portion of the audio corresponding to the hotword, only the portion of the audio corresponding to the query, or the entire audio including both the hotword and the query. E Enroll user 10 to verify their identity E may be a particular validation phrase provided to the validation model 146.

[0032] When speaker 10 initiates this verification process performed by model 146 of verification system 140, a verification session has begun, which may include one or more interactions between speaker 10 and verification system 140 (e.g., via device 110). Verification system 140 is configured to record / log verification results 148 that indicate interaction events that occur during the verification process. Some examples of these interaction events that may be captured as verification results 148 include receipt of query 12Q for verification, rejection of query 12Q, acceptance of query 12Q, a decision of verification system 140 (e.g., the probability of the enrolled speaker), feedback from speaker 10 regarding the verification process, or other verification log events. Here, an example of feedback from speaker 10 that may generate verification result 148 is when speaker 10 subsequently interacts with the results of query 12Q. In other words, further interaction with the actual results of query 12Q may indicate that the verification system 140 has correctly verified speaker 10, as speaker 10 has further engaged in responding 144 to query 12Q (e.g., clicking on a search result or using a feature authorized by verification system 140). Due to the nature of these verification results 148, these log events generally do not include any sensitive user information (e.g., a user identifier) and / or include actual audio data corresponding to query 12Q.

[0033] 1B and 1C illustrate some verification results 148 that may occur during the verification process by verification system 140. In FIG. 1B, verification system 140 is configured to verify that device 110 is connected to three enrolled users 10, designated as User X, User Y, and User Z. EIn this example, user X submits query 12Q to device 110 stating, "What's next for me?" Upon receiving this query 12Q, device 110 and / or verification system 140 may first determine whether query utterance 12Q triggers the need for verification. Generally, utterance 12 triggers the need for verification when, for example, the utterance includes a request for personal information stored on or accessible by device 110, a request to perform a restricted function, or a request for other information designated as requiring verification. For example, in the previous example of a family of three, the father may have a music playlist containing explicit lyrics that the father designates as requiring authorization so that his 9-year-old daughter is not exposed to the explicit lyrics. In some examples, device 110 and / or verification system 140 may perform the verification process for query 12Q regardless of whether query 12Q triggers the need for verification. That is, when speaker 10 speaks to device 110, verification system 140 determines whether speaker 10 is an enrolled user 10. E , but only performs authorization when the spoken utterance 12 requires authorization to be performed. In FIG. 1B, the spoken utterance "What's next for me?" is a query requesting personal calendar information. Now, verification system 140 begins the verification process and attempts to verify the speaker, User X. User X is on the list of enrolled users, but in this example, verification system 140 determines whether speaker 10, User X, is an enrolled user 10. E Based on this failure, the verification system 140 communicates a response 144 to the device 110, which in turn communicates the response 144 to the speaker 10, user X, to indicate the verification failure. The response 144 states, "Sorry, we were unable to verify your voice." The speaker 10 then verifies that he is the enrolled user 10 of the verification system 140 of the device 110. EKnowing that speaker 10, user X, was not able to recognize his / her voice, user X may submit a second query 12Q2, e.g., attempting to pronounce the same query 12Q a second time using a louder voice (i.e., attempting to ask the same query 12Q again). As shown in FIG. 1B, this second query 12Q2 also proves to be unsuccessful because verification system 140 fails to verify speaker 10, user X's voice, and again communicates this failure as a second response 144, 144b stating, again, "Sorry, I wasn't able to verify your voice." During this unsuccessful verification session, verification system 140 logs verification results 148 that occurred during the session as verification data. For example, FIG. 1B shows three verification results 148, 148a-c. The first verification result 148a indicates a "punt" event, which refers to an event in which verification system 140 (e.g., verification model 146) did not recognize speaker 10. The second verification result 148b indicates a punt and re-query event in which speaker 10 was not recognized (i.e., punted), but speaker 10 re-asked the same query 12Q2 a second time. The third verification result 148, 148c indicates a double-punt event, which refers to an event in which, after a failed verification (i.e., punt) and subsequent resubmission (i.e., re-query) of the same query 12Q2, the verification model 146 again failed to verify speaker 10 (i.e., a double-punt). In other words, each of these potential verification results 148a-c indicates that speaker 10 is user X, and that user X is in fact an enrolled user 10 in the verification system 140. E Therefore, this indicates the person's refusal.

[0034] 1C illustrates a scenario in which the verification result 148 is false acceptance. One way in which the verification system 140 may determine that an interaction event is false acceptance is by having the verification system 140 verify that multiple enrolled users 10 E When multiple enrolled users E If so, when utterance 12 is spoken by speaker 10, verification system 140 determines whether utterance 12 is EFor example, the verification system 140 may determine how likely it is that the speaker 10 corresponds to the enrolled user 10. E generate an enrollment score for each enrolled user 10E indicating the likelihood that the enrolled speaker 10 E When the enrollment score of a speaker 10 meets the acceptance threshold score, the verification system 140 identifies the speaker 10 as that particular enrolled user 10. E While this is the ideal behavior of the verification system 140, sometimes the execution of the verification model 146 may result in two or more enrolled users 10 being unable to identify the exact utterance 12Q for a particular utterance 12Q. E For example, the validation model 146 may determine that both enrolled users 10 and 11 have scores that meet the acceptance threshold for utterance 12Q. E Two siblings from the same household may have similar voice characteristics that can be difficult to distinguish for speaker verification. When this occurs, the verification results 148 may be different for two or more enrolled users 12. E 10 indicates an impostor acceptance event, which refers to an event that is a determination by the verification system 140 that a speaker 10 has a score that meets the acceptance threshold. 10 E, but the verification system 140 may also transmit the query utterance 12Q to two or more enrolled users 10 E This is called an imposter acceptance event because the identity of the user is accepted as the actual identity. In other words, the accepted identity is an imposter. For example, Figure 1C shows the case where the threshold acceptance value is 0.5 and the query utterance 12Q is accepted by two or more enrolled users 10. E12Q may result in a verification process indicating that speaker 12Q is likely to be speaker 10. Here, speaker 10 is user X, and verification model 146 indicates a score of 0.65 for user X, a score of 0.8 for user Y, and a score of 0.4 for user Z. In this example, because user Y has the highest score, verification system 140 may return a result 144 stating, "User Y, I have an appointment at 5 PM," such that verification system 140 believes that user Y (i.e., an impostor) and not user X spoke utterance 12Q.

[0035] FIG. 1D illustrates how the verification system 140 determines whether the speaker 10 of the utterance 12 is an enrolled user 10 of the device 110. E 1 illustrates that one of several verification models 146 (e.g., shown as three verification models 146a-c) may be used to determine whether a verification process is successful. Different models 146 may be used by the provider of the verification system 140 to test, develop, maintain, or comparatively evaluate the models 146. For example, Group A illustrates several users 10 (e.g., seven users 10) having corresponding devices 110 that use a first verification model 146, 146a, to perform the verification process. Group B illustrates four users 10 having corresponding devices 110 that use a second verification model 146b. Group C illustrates three users 10 having corresponding devices 110 that use a third verification model 146c to perform the verification process. In this depicted example, the first verification model 146a may correspond to a production or primary model deployed to the majority of users 10 of the verification system 140 and should be generally robust because it has stood the test of time. The second verification model 146b may correspond to an experimental or alternative model that the provider of the verification system 140 may use to develop improvements to the verification process (e.g., fix bugs in the verification). For example, the verification model 146 may use a neural network to determine whether the speaker 10 of the utterance 12 is a member of the enrolled user 10. EThe neural networks may have different types of neural network architectures. A production validation model, such as the first validation model 146a, may use one type of neural network architecture, while an experimental validation model, such as the second validation model 146b, uses a different type of neural network architecture or an improvement / update to the neural network architecture of the first validation model 146a. Additionally or alternatively, in some implementations, each model 146 is generally trained with a set of training data to teach the model 146 to verify registered or enrolled identities. Thus, the content of the training data provided to each model 146 may have some impact on its performance. For at least this reason, the production validation model 146a may differ from the experimental validation model 146b in the fact that each model 146 was trained with a different set of training data (e.g., validation training examples). The experimental validation model 146b may then be deployed in parallel with the production validation model 146a to identify whether any changes to the training data improve the validation performance of the validation system 140 using the experimental validation model 146b when compared to the production validation model 146a. According to a comparison-based model approach, two or more models 146 may be deployed to any set of users 10, and data in the form of validation results 148 may be collected for the validation process of each model 146 to allow the provider to compare and / or assess performance similarities and differences between these models 146. For example, FIG. 1D shows that Group A, using a first model 146a, has achieved a first set of validation results 148, 148. S1 and group B using the second model 146b generates a second set of validation results 148, 148 S2 and group C using the third model 146c generates a third set of validation results 148, 148 S3When using a comparison-based model approach, a provider may route a majority of user traffic to the production verification model 146a to avoid deployment complexities or any other issues that may arise with the experimental verification model 146b. Here, the production verification model 146a is deployed to the majority of the verification system users (i.e., deployed to the devices 110 associated with the users 10 running the verification system 140), while the other comparison models 146 are deployed to a minority of the verification system users (i.e., deployed to the devices 110 associated with the users 10 running the verification system 140).

[0036] In some examples, such as FIG. 1D , the validation system 140 uses three or more models 146 for comparative analysis. For example, FIG. 1D shows a dotted line for group C, which uses a third validation model 146c, to indicate that a provider may optionally deploy three or more models for comparative analysis. Here, a third model 146, 146c, is shown to illustrate the use of the control validation model 146. The control validation model 146 may refer to a redundant version of the production validation model 146 (e.g., the first validation model 146a) to confirm the comparison results between the production model 146a and the experimental model 146b. In other words, the comparison results between the production model 146a and the experimental model 146b may indicate some performance differences. By further comparing the performance results of the production model 146a or the experimental model 146b with the control model 146c, the provider may be able to identify whether the performance differences are actually due to validation or some other issue with the model 146. For example, perhaps some number of production models 146a are corrupted or have bugs. Here, the performance of the production models 146a due to the bugs may make the experimental models 146b look better in the process of verifying speaker 10. However, when the experimental models 146b are compared to control models 146c representing the production models 146a in an uncorrupted or bug-free state, the control models 146c perform better than the experimental models 146b. This may indicate to the provider that they should not necessarily replace or mass-deploy the experimental models 146b and / or that some additional issues may exist in the production models 146a.

[0037] Because the verification system 140 may use several different verification models 146, the enrollment process may involve verifying that each model 146 is accurate to the extent that the speaker of the utterance 12 is an enrolled user 10. E To enable users 10 to properly verify whether they are registered users 10 on each model 146, EDepending on the enrollment process, enrolling a user 10 in multiple models may range from being undetectable to the registering user 10 to requiring the enrolling user 10 to provide model-specific enrollment (e.g., a specific enrollment phrase). A user 10 may also have to seemingly re-enroll when the verification system 140 is updated or the verification model 146 undergoes changes that affect the user's enrollment in the model 146. For example, based on a comparative analysis, the provider decides to replace the first verification model 146a with the second verification model 146b. When this occurs, the second verification model 146b may need to be deployed to a majority of users 10 to become the production model 146. In this situation, some number of users 10 may need to re-enroll or enroll for the first time with the second verification model 146b, which is now the production model 146.

[0038] 1A-1D, speech environment 100 also includes a verification analyzer 200 (also referred to as analyzer 200). Analyzer 200 is configured to communicate with verification system 140 and analyze the performance of one or more verification models 146. For example, even though FIGS. 2A and 2B show analyzer 200 analyzing the performance of two and three models 146, respectively, analyzer 200 is scalable to perform analysis for any number of models 146 (e.g., even for one model 146). Analyzer 200 may be located locally (e.g., on user device 110) and configured to communicate its analysis to a provider of verification system 140 (e.g., via network 120), or may be located remotely (e.g., on remote system 130) and in communication with the provider. In some examples, some portions of analyzer 200 may be located locally, while other portions are located remotely. The analyzer 200 is configured to receive the validation results 148 as input and generate a performance comparison 202 as output. In some examples, the performance comparison 202 triggers some action to be taken related to the models 146 that the analyzer 200 compares. For example, the analyzer 200 is configured to coordinate the deployment of different models 146 to one or more groups of users 10. In other words, the analyzer 200 may replace the primary or production validation model 146a with the experimental or alternative validation model 146b when the experimental validation model 146b is found to have better performance validation capabilities than the production validation model 146a.

[0039] Whether analyzing a single model 146 or performing a comparative analysis on two or more models 146, the analyzer 200 generates a set of confirmation results 148 from each model 146 that the analyzer intends to analyze. S2A and 2B, analyzer 200 includes an identifier 210, a scorer 220, and a comparator 230. Identifier 210 is configured to receive verification results 148 and identify each verification result 148 that corresponds to a particular performance metric 212, where performance metric 212 is used to determine whether model 146 correlates the identity of user 10 with one or more registered or enrolled users 10. E The performance metrics 212 may refer to specific interaction events that occurred during the verification session that provide some indication of whether the verification result 148 was successfully or not verified. For example, the performance metrics 212 correspond to false reject or false accept events during the verification session. As previously mentioned, some examples of false reject events include punt, double punt, punt, and re-query, while an example of a false accept event is imposter accept. Note that there may be other similar performance metrics 212 for indicating false reject or false accept events in the verification model 146. As shown in FIG. 2A , the identifier 210 separates the verification results 148 into one or more corresponding performance metrics 212, 212a-n.

[0040] After identifying validation results 148 that correspond to one or more performance metrics 212, the identifier 210 communicates the one or more performance metrics 212a-n to the scorer 220. The scorer 220 is configured to receive the one or more performance metrics 212a-n from the identifier 210 and determine a score 222 based on the number of validation results 148 that include a given performance metric 212. For example, the scorer 220 may determine a score 222 for each different type of performance metric 212. When the scorer 220 determines a score 222, the score 222 is provided to a particular model 146. In other words, for each set of validation results 148 Scorresponds to a given confirmation model 146, the scores 222 form a representation of the performance of the particular model with respect to a given performance metric 212. By way of example, FIG. 2A shows that the grader 220 generates a first score 222, 222a with respect to a first performance metric 212a of the first model 146a, a second score 222, 222b with respect to a second performance metric 212b of the first model 146a, and an nth score 222, 222n with respect to an nth performance metric 212n of the first model 146a. In FIG. 2A, because the analyzer 200 is comparing two models 146a-b (e.g., a primary confirmation model 146a and an alternate confirmation model 146b), the grader 220 generates a set of confirmation results 148 for the same performance metric 212, but for another model (i.e., a set of confirmation results 148 from that particular model 146). S ) and generates a score 222 for the second model 146b. In other words, the second model 146b may receive a score 222 from the scorer 220 with respect to the same performance metrics 212 as the first model 146a. This is because the first model 146a may receive a score 222 for the set of validation results 148 from the first model 146a. S1 while the second model 146b receives a set of confirmation results 148 from the second model 146b. S2 This means receiving a score 222 for the double punt metric 212 based on:

[0041] In some implementations, when a score 222 corresponds to a punt as a performance metric 212, the punt score 222 is calculated based on a particular set of validation results 148 S With respect to the score 222 corresponding to a double punt as a performance metric 212, the double punt score 222 refers to the count of the number of punts in a particular set of confirmation outcomes 148 divided by the total number of confirmation outcomes 148 in the set. S When a score 222 corresponds to punts and re-queries as a performance metric 212, the punt and re-query score 222 may refer to the count of the number of duplicate punts in a particular set of confirmation results 148 divided by the total number of confirmation results 148 in the set. SThe score 222 may refer to the combined count of punts and resubmissions of the same query 12Q that was initially punted divided by the total number of confirmation results 148 in the set. When the score 222 corresponds to imposter acceptance as a performance metric 212, the imposter acceptance score 222 may refer to the number of enrolled users 10 that have an acceptance score threshold at which the user 10 can be confirmed. E This may refer to the proportion of queries 12Q that have a particular set of confirmation results 148 S one or more enrolled users 10 with an acceptance score threshold that allows the users 10 to be confirmed within E is the count of the number of queries with 12Q divided by the total number of queries with 12Q in the set.

[0042] In some examples, the scorer 220 generates a set of validation results 148 SThe scorer 220 generates a score 222 of the verification model 146 based on one or more relationships between types of performance metrics 212 in the verification model 146. The relationships may refer to relationships between metrics 212 representing false acceptance events and metrics 212 representing false rejection events. By way of example, the scorer 220 may generate the score 222 by combining false acceptance events (i.e., false acceptance errors) and false rejection events (i.e., false rejection errors) into a single cost, which the scorer 220 converts to the score 222. In some implementations, the score 222 refers to a cost function that is a weighted combination of false acceptance events and false rejection events. In some configurations, the amount of false acceptance events and / or false rejection events may lead to the scorer 220 being able to identify the probability of a false acceptance event and / or false rejection event occurring in the verification model 146. Here, the score 222 of the model 146 may be equal to a first cost coefficient (also called a weight) multiplied by the probability of a false acceptance event, combined (e.g., added) with a second cost coefficient multiplied by the probability of a false rejection event. In other approaches, the cost that forms the score 222 may be expressed as a combination of a false acceptance cost component and a false rejection cost component, where each cost component is expressed as a cost weight assigned to a false event multiplied by the probability of the same speaker and the probability of the false event. These are some example algorithms by which the scorer 220 generates the score 222, but other algorithms that represent the relationship between the metrics 212 can be used to generate the score 222.

[0043] The scorer 220 communicates the performance metric scores 222 to the comparator 230 so that the comparator 230 may determine whether the validation ability of a validation model 146 (e.g., the production validation model 146a) is better than another validation model 146 (e.g., the experimental validation model 146b). To make this determination, the comparator 230 compares the scores 222 for the same performance metric 212 between the models 146. For example, FIG. 2A shows the comparator 230 comparing the first performance metric score 222a for the first performance metric 212 of the first model 146a with the first performance metric score 222a for the first performance metric 212 of the second model 146b. The comparator 230 may be configured to compare a single performance metric score 222 across two or more models 146 or to compare multiple performance metric scores 222 across two or more models 146 at once. When the analyzer 200 uses the comparator 230 to compare multiple performance metric scores 222 across two or more models 146 at a time, the analyzer 200 may provide some indication of importance for each type of performance metric 222. For example, the analyzer 200 may apply a weight to each type of performance metric score 222 to indicate its overall importance to the validation ability of the validation process.

[0044] 2A and 2B, in some configurations, the comparator 230 is configured to generate a performance comparison 202 indicating whether one model 146 outperforms another model 146. For example, in FIG. 2A, the comparator 230 compares two models 146a-b (e.g., a primary validation model 146a and an alternate validation model 146b), while in FIG. 2B, the comparator 230 compares three models 146a-c (e.g., a primary or production validation model 146a, an alternate or experimental validation model 146b, and a control validation model 146c). In some examples, when the comparison indicates that the performance score 222 of the second model 146b is better than the corresponding performance score 222 of the first model 146a, the comparator 230 recommends replacing the first model 146a with the second model 146b. In some cases, rather than passively recommending this replacement, the analyzer 200 actually replaces the first model 146a with the second model 146b. To replace a model 146 with another model 146 (e.g., to replace the first model 146a with the second model 146b), the analyzer 200 may shift or reallocate the number of users 10 or devices 110 allocated to each of the models 146. For example, when replacing a production-validated model 146 used by a majority of the user devices 110, the analyzer 200 instead deploys the lab-validated model 146b to a majority of the user devices 110, causing the majority to cease using the original production-validated model 146a. 2B , when comparator 230 is comparing performance scores 222 of two or more models 146, analyzer 200 may set conditions for when analyzer 200 will recommend or actively replace a model 146. For example, in a situation with three models 146a-c, production-validated model 146a, lab-validated model 146b, and control-validated model 146c, analyzer 200 may set a condition that lab-validated model 146b must have performance (based on performance metric scores 222) that exceeds each of the other models 146 (e.g., exceeds production-validated model 146a and control-validated model 146c).

[0045] 3 is a flow diagram of an example sequence of operations for a method 300 for evaluating the performance of a validation model 146. In operation 302, the method 300 evaluates a first set of validation results 148, 148 S1 and receiving a first set of verification results 148, 148 S1 Each verification result 148 is generated by a primary verification model 146, 146a running on the respective user device 110 verifying the identity of one or more registered users 10 of the respective user device 110 during a respective interaction between the respective user 10 and the respective user device 110. E In operation 304, the method 300 returns the second set of verification results 148, 148. S2 receiving a second set of verification results 148, 148 S2 Each verification result 148 is generated by the alternative verification model 146, 146b executed on the respective user device 110 during an interaction between the respective user 10 and the respective user device 110, and the alternative verification model 146, 146b .... E At operation 306, the method 300 also includes receiving an indication of whether the primary authentication model 146, 146a or the alternate authentication model 146, 146b identifies one or more registered users 10 during a respective interaction between the respective user 10 and the respective user device 110. E a first and second set of verification results 148 including performance metrics 212 indicating either the success or failure of verifying the identity of each user 10 as one of S1 , 148 S2 In operation 308, the method 300 identifies each validation result 148 in the first set of validation results 148, which includes the performance metrics 212. S1In operation 310, the method 300 determines a first performance score 222, 222a of the primary validation model 146, 146a based on the number of validation results 148 identified within the primary validation model 146, 146a. S2 At operation 312, the method 300 determines whether the verification ability of the alternative verification model 146, 146b is superior to that of the primary verification model 146, 146a based on the first performance score 222, 222a and the second performance score 222, 222a. At operation 314, when the verification ability of the alternative verification model 146, 146b is superior to that of the primary verification model 146, 146a, the method 300 replaces the primary verification model 146, 146a running on at least one respective user device 110 with the alternative verification model 146, 146b.

[0046] 4 is a flow diagram of another example sequence of operations for a method 400 for evaluating the performance of a validation model 146. In operation 402, the method 400 evaluates the performance of a validation model 146 by evaluating a set of validation results 148, 148. S Receive and confirm the result set 148, 148 S Each verification result 148 is generated by verifying that the verification model 146 running on the respective user device 110 verified one or more registered users 10 of the respective user device 110 during a respective interaction between the respective user 10 and the respective user device 110. E At operation 404, method 400 indicates whether the authentication model 146 verifies the identity of each user 10 as one of one or more registered users 10 of each user device 110. E a set of verification results 148 including false rejection metrics 212 indicating that the person falsely rejected identifying themselves as one of the S In operation 406, the method 400 identifies each validation result 148 in the set of validation results 148, which includes the false acceptance metric 212. SThe false acceptance metric 212 determines each verification result 148 in the table. The false acceptance metric 212 determines whether the verification model 146 detects at least two registered users 100 of each user device 110 that meet the verification threshold. E In operation 408, the method 400 determines a set of verification results 148, 148, including a false rejection metric 212 and a false acceptance metric 212. S A performance score 222 is generated for the validation model 146 based on the number of validation results 148 identified within the validation model 146 .

[0047] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems (e.g., verification system 140 and / or analyzer 200) and methods (e.g., methods 300, 400) described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functionality are intended to be merely exemplary and are not intended to limit the implementation of the invention described and / or claimed herein.

[0048] Computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as appropriate. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 or on storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as appropriate. Additionally, multiple computing devices 500 may be connected together (eg, as a server bank, a group of blade servers, or a multi-processor system) with each device providing a portion of the required operations.

[0049] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0050] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or memory on the processor 510.

[0051] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such assignment of roles is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, to the display 580 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 550, which may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and to a low-speed expansion port 590. The low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or may be coupled to a network device, such as a switch or router, for example, via a network adapter.

[0052] Computing device 500, as shown in the figure, may be implemented in many different forms. For example, computing device 500 may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0053] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0054] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0055] The processes and logic flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special purpose microprocessors, as well as any one or more processors of any type of digital computer. A processor generally receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. A computer also generally includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to such mass storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0056] To provide for interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0057] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0058] 10 users 10 E Enrolled Users 12 Spoken Utterances 12E Enrollment Utterances 12Q Query utterance, query 12Q2 Second Query 100 Speech Environment 110 Confirmation Device, Device, User Device 112 Data Processing Hardware 114 Memory Hardware 116 Audio Capture Device 118 Audio Output Devices 120 Network 130 Remote Systems 132 Remote Resources 134 Remote Data Processing Hardware 136 Remote Memory Hardware 140 Verification System 142 Confirmation Query 144 responses 144b Second Response 146 Confirmation Model 146a First confirmation model, production confirmation model 146b Second confirmation model, experimental confirmation model 146c Third Confirmatory Model, Control Model 148 Confirmation Results 148 S Set of confirmation results 148 S1 First set of validation results 148 S2 Second set of validation results 148 S3 Third set of validation results 148a~c Confirmation results 200 Confirmation analyzer, analyzer 202 Performance comparison 210 Specifier 212 Performance Metrics 212a~n Performance Metrics 220 Scorer 222 score 222a First Score, First Performance Metric Score 222b Second Score 222n nth score 230 Comparator 300 ways 400 ways 500 computing devices 500a Server 500b laptop computer 500c Rack Server System 510 processor 520 memory 530 Storage Devices 540 High-Speed Interface / Controller 550 High-Speed Expansion Port 560 Low-Speed Interface / Controller 570 Slow Bus 580 Display 590 Low-Speed Expansion Port

Claims

1. 1. A computer-implemented method executed on data processing hardware, the data processing hardware comprising: receiving a set of verification results, each verification result of the set indicating whether an alternative verification model executed on a respective user device verified the identity of the respective user as one of one or more registered users of the respective user device during an interaction between the respective user and the respective user device; determining a performance score for the alternative validation model based on the number of validation results identified within the set of validation results as including a performance metric; determining whether the verification ability of the alternative verification model is superior to the verification ability of the primary verification model based on the performance score; replacing the primary verification model executed on at least one respective user device with the alternative verification model based on a determination that the verification capability of the alternative verification model is superior to the verification capability of the primary verification model; and A computer-implemented method for performing an operation, comprising:

2. 2. The computer-implemented method of claim 1, wherein the performance metrics indicate that the alternative verification model was one of successful or unsuccessful in verifying the identity of the respective user as one of the one or more registered users during the respective interaction between the respective user and the respective user device.

3. The operation is first allocating the primary verification model to execute on a first plurality of user devices and allocating the alternate verification model to execute on a second plurality of user devices; further comprising 2. The computer-implemented method of claim 1, wherein replacing the primary verification model executing on the at least one respective user device comprises reallocating the alternative verification model to execute on at least one respective user device of the first plurality of user devices in place of the primary verification model.

4. The computer-implemented method of claim 3 , wherein the first plurality of user devices is greater than the second plurality of user devices.

5. The computer-implemented method of claim 1 , wherein none of the verification results received in the set of verification results includes a user identifier identifying the respective user.

6. 2. The method of claim 1, wherein none of the confirmation results received within the set of confirmation results includes audio data associated with the respective interaction between the respective user and the respective user device.

7. the primary validation model is trained on a first set of training data; 10. The computer-implemented method of claim 1, wherein the alternative confirmation model is trained on a second set of training data that is different from the first set of training data.

8. 2. The method of claim 1, wherein the primary validation model includes a first neural network and the alternate validation model includes a second neural network having a different neural network architecture than the first neural network.

9. 2. The computer-implemented method of claim 1, wherein the performance metrics include a false rejection metric indicating that the alternative verification model falsely rejected identifying the respective user as the one of the one or more registered users of the respective user device.

10. 2. The computer-implemented method of claim 1, wherein the performance metrics include a false acceptance metric indicating that the alternative verification model incorrectly accepted the respective user as the one of the one or more registered users of the respective user device.

11. 1. A system comprising: data processing hardware; memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to: an operation of receiving a set of verification results, each verification result of the set indicating whether an alternative verification model executed on a respective user device verified the identity of the respective user as one of one or more registered users of the respective user device during an interaction between the respective user and the respective user device; determining a performance score for the alternative validation model based on the number of validation results identified within the set of validation results as including a performance metric; determining whether the verification capability of the alternative verification model is superior to the verification capability of the primary verification model based on the performance score; and replacing the primary verification model executed on at least one respective user device with the alternative verification model based on determining that the verification capability of the alternative verification model is superior to the verification capability of the primary verification model. memory hardware, which performs operations including A system comprising:

12. 12. The system of claim 11, wherein the performance metrics indicate that the alternative verification model was one of successful or unsuccessful in verifying the identity of the respective user as one of the one or more registered users during the respective interaction between the respective user and the respective user device.

13. The operation is first allocating the primary verification model to execute on a first plurality of user devices and allocating the alternate verification model to execute on a second plurality of user devices; further comprising 12. The system of claim 11, wherein the act of replacing the primary verification model executing on the at least one respective user device includes the act of reallocating the alternative verification model to execute on at least one respective user device of the first plurality of user devices in place of the primary verification model.

14. The system of claim 13 , wherein the first plurality of user devices is greater than the second plurality of user devices.

15. The system of claim 11 , wherein none of the verification results received in the set of verification results includes a user identifier identifying the respective user.

16. 12. The system of claim 11, wherein none of the verification results received within the set of verification results includes audio data associated with the respective interaction between the respective user and the respective user device.

17. the primary validation model is trained on a first set of training data; 12. The system of claim 11, wherein the alternative confirmation model is trained on a second set of training data that is different from the first set of training data.

18. 12. The system of claim 11, wherein the primary validation model includes a first neural network and the alternate validation model includes a second neural network having a different neural network architecture than the first neural network.

19. 12. The system of claim 11, wherein the performance metrics include a false rejection metric indicating that the alternative verification model falsely rejected identifying the respective user as the one of the one or more registered users of the respective user device.

20. 12. The system of claim 11, wherein the performance metrics include a false acceptance metric indicating that the alternative verification model incorrectly accepted the respective user as the one of the one or more registered users of the respective user device.

Images