Evaluate speaker recognition performance

Abstract

A method (300) for evaluating a validation model (146) includes receiving a first set of validation results and a second set of validation results (148), wherein each validation result indicates whether a primary model or an alternative model authenticates a user (10) as a registered user. The method further includes identifying each validation result in the first set and the second set that includes a performance metric (212); determining a first score (222) for the primary model based on the identified validation results in the first set that include the performance metric; and determining a second score for the alternative model based on the identified validation results in the second set that include the performance metric. The method further includes determining whether the validation capability of the alternative model is superior to that of the primary model based on the first score and the second score.

Description

Technical Field

[0001] This disclosure relates to evaluating speaker recognition performance. Background Technology

[0002] Recently, computing devices offering multiple user input modalities have become increasingly common. For example, smartphones and other user devices include voice recognition services that allow users to provide voice input to the device as an alternative to typing or pointing. In some cases, voice-based input may be more convenient as a hands-free way to interact with computing devices. Some devices require user authentication before performing actions based on voice input to prevent privacy and security violations. Summary of the Invention

[0003] One aspect of this disclosure provides a method for evaluating the performance of a verification model. The method includes receiving a first set of verification results at data processing hardware, wherein each verification result in the first set of verification results indicates whether a primary verification model executed on a corresponding user device authenticates a corresponding user as one of one or more registered users of the corresponding user device during a corresponding interaction between the corresponding user and the corresponding user device. The method also includes receiving a second set of verification results at data processing hardware, wherein each verification result in the second set of verification results indicates whether an alternative verification model executed on the corresponding user device authenticates a corresponding user as one of one or more registered users of the corresponding user device during an interaction between the corresponding user and the corresponding user device. The method further includes having the data processing hardware identify each verification result in the first set of verification results and the second set of verification results that includes a performance metric, the performance metric indicating whether a corresponding one of the primary verification model or the alternative verification model is a verification model that is able or unable to authenticate a corresponding user as one of one or more registered users during a corresponding interaction between the corresponding user and the corresponding user device. The method further includes having the data processing hardware determine a first performance score for the primary verification model based on a plurality of verification results including a performance metric identified in the first set of verification results. The method further includes having data processing hardware determine a second performance score for an alternative primary verification model based on multiple verification results, including performance metrics, identified in a second set of verification results. The method further includes having data processing hardware determine, based on a first performance score and a second performance score, whether the verification capability of the alternative verification model is superior to that of the primary verification model. The method also includes having data processing hardware replace the primary verification model executed on at least one corresponding user device with the alternative verification model when the verification capability of the alternative verification model is superior to that of the primary verification model.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the method further includes receiving a third set of verification results at data processing hardware, wherein each verification result in the third set of verification results indicates whether a control verification model executed on a corresponding user device authenticates the corresponding user as one of one or more registered users of the corresponding user device during a corresponding interaction between the corresponding user and the corresponding user device. In these implementations, the method further includes having the data processing hardware identify each verification result in the third set of verification results that includes a performance metric, the performance metric indicating whether the control verification model is a control verification model capable of or unable to verify the identity of the corresponding user during a corresponding interaction between the corresponding user and the corresponding user device. In these implementations, the method further includes having the data processing hardware determine a third performance score of the control verification model based on a plurality of verification results including performance metrics identified in the third set of verification results, and having the data processing hardware determine whether the verification capability of an alternative verification model is superior to the verification capability of the control verification model based on a second performance score and a third performance score. In these implementations, replacing the primary verification model executed on each corresponding user device with an alternative verification model includes replacing the primary verification model executed on each corresponding user device with an alternative verification model when the verification capability of the alternative verification model is superior to the verification capabilities of both the primary verification model and the control verification model.

[0005] In some examples, the method includes initially allocating a primary verification model to be executed on a first plurality of user devices and an alternative verification model to be executed on a second plurality of user devices by data processing hardware. In these examples, replacing the primary verification model executed on at least one corresponding user device includes reassigning the alternative verification model to execute on at least one corresponding user device among the first plurality of user devices instead of the primary verification model. The first plurality of user devices may be larger than the second plurality of user devices.

[0006] Another aspect of this disclosure provides a system for evaluating the performance of a verification model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation. The operation includes receiving a first set of verification results, wherein each verification result in the first set of verification results indicates whether a primary verification model executed on a corresponding user device authenticates the corresponding user as one of one or more registered users of the corresponding user device during a corresponding interaction between the corresponding user and the corresponding user device. The operation also includes receiving a second set of verification results, wherein each verification result in the second set of verification results indicates whether an alternative verification model executed on the corresponding user device authenticates the corresponding user as one of one or more registered users of the corresponding user device during an interaction between the corresponding user and the corresponding user device. The operation further includes identifying each verification result in the first set of verification results and the second set of verification results that includes a performance metric indicating whether a corresponding one of the primary verification model or the alternative verification model is a verification model that is capable or incapable of authenticating the corresponding user as one of one or more registered users during a corresponding interaction between the corresponding user and the corresponding user device. The operation further includes determining a first performance score for the primary verification model based on multiple verification results, including performance metrics, identified in the first set of verification results. The operation also includes determining a second performance score for an alternative primary verification model based on multiple verification results, including performance metrics, identified in the second set of verification results. The operation further includes determining whether the verification capability of the alternative verification model is superior to that of the primary verification model based on the first and second performance scores. The operation also includes replacing the primary verification model executed on at least one corresponding user device with the alternative verification model when the verification capability of the alternative verification model is superior to that of the primary verification model.

[0007] In some embodiments, the operation further includes receiving a third set of verification results, wherein each verification result in the third set of verification results indicates whether a control verification model executed on the corresponding user device authenticates the corresponding user as one of one or more registered users of the corresponding user device during a corresponding interaction between the corresponding user and the corresponding user device. In these embodiments, the operation further includes identifying each verification result in the third set of verification results that includes a performance metric indicating whether the control verification model is capable of or incapable of verifying the identity of the corresponding user during a corresponding interaction between the corresponding user and the corresponding user device. In these embodiments, the operation further includes determining a third performance score of the control verification model based on the plurality of verification results including performance metrics identified in the third set of verification results, and having data processing hardware determine, based on the second and third performance scores, whether the verification capability of an alternative verification model is superior to that of the control verification model. In these embodiments, replacing the primary verification model executed on each corresponding user device with an alternative verification model includes replacing the primary verification model executed on each corresponding user device with the alternative verification model when the verification capability of the alternative verification model is superior to that of both the primary verification model and the control verification model.

[0008] In some examples, the operation includes initially assigning a primary authentication model to be executed on a first plurality of user devices and an alternative authentication model to be executed on a second plurality of user devices. In these examples, replacing the primary authentication model executed on at least one corresponding user device includes reassigning the alternative authentication model to execute on at least one corresponding user device among the first plurality of user devices. The first plurality of user devices may be larger than the second plurality of user devices.

[0009] Implementations of the system or method may include one or more of the following optional features. In some implementations, neither the verification results received in the first set of verification results nor the second set of verification results include a user identifier identifying the corresponding user. In some configurations, neither the verification results received in the first set of verification results nor the second set of verification results include audio data associated with the corresponding interaction between the corresponding user and the corresponding device. Operationally, the primary verification model is trained on the first set of training data, while the alternative verification model is trained on a second set of training data different from the first set of training data. The primary verification model may include a first neural network, and the alternative verification model may include a second neural network having a different neural network architecture than the first neural network.

[0010] In some examples, performance metrics include an error rejection metric that indicates that a corresponding one of the primary or alternative authentication models incorrectly rejects identifying a corresponding user as one of one or more registered users of a corresponding user device. Error metrics may include one of the following: a punt metric that indicates that a corresponding one of the primary or alternative authentication models authorized the corresponding user to guest privileges during a corresponding interaction with the corresponding user device; a double punt metric that indicates that immediately after being authorized to guest privileges by the same corresponding user during a previous corresponding interaction with the corresponding user device, a corresponding one of the primary or alternative authentication models authorized the corresponding user to guest privileges during a corresponding interaction with the corresponding user device; and a punt and re-question metric that immediately after being authorized to guest privileges by the same corresponding user during a previous corresponding interaction with the corresponding user device, a corresponding one of the primary or alternative authentication models authorized the corresponding user to guest privileges during a corresponding interaction with the corresponding user device when the corresponding interaction corresponds to a request for authorization of privileges by the corresponding user.

[0011] In some implementations, the performance metric includes a false acceptance metric, which indicates that a corresponding one in the primary or alternative verification model incorrectly accepts a corresponding user as one of one or more registered users of a corresponding user device. Here, the false acceptance metric may include a proxy imposter acceptance metric, which indicates that a corresponding one in the primary or alternative verification model determines a corresponding verification score associated with at least two registered users of a corresponding user device that meets the verification threshold.

[0012] Details of one or more embodiments of this disclosure are set forth below in the accompanying drawings and detailed description. Other aspects, features, and advantages will become apparent from the detailed description, the drawings, and the claims. Attached Figure Description

[0013] Figures 1A to 1D This is a schematic diagram of a sample speech environment used to validate the analyzer.

[0014] Figure 2A and 2B This is a schematic diagram of an example validation analyzer used to evaluate the performance of a validation model.

[0015] Figure 3 This is a flowchart illustrating an example of the operational setup for methods used to evaluate the performance of a validation model.

[0016] Figure 4 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0017] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0018] Typically, an authentication process refers to the process of identifying whether an entity is authorized to perform certain tasks or actions based on one or more characteristics associated with it. When the entity is a person, the characteristics involved in the authentication process are typically that person's biometrics. For example, an authentication process uses audio features extracted from a person's spoken voice or audio to verify whether he or she is authorized to perform a specific task. Other examples of biometrics that an authentication process can use besides voice include fingerprints, retinal features, facial features (e.g., in facial recognition), and / or handwriting (e.g., signatures). For any of these biometrics, the authentication process typically compares the current biometric input with a previous biometric input (i.e., registered biometrics) to determine whether the current biometric input matches or is very similar to the previous biometric input. When the current biometric input matches or is very similar to the previous biometric input, the authentication process considers the current biometric input to verify the person's identity; allowing the person to perform the specific task that requires authentication.

[0019] Speaker recognition systems can perform a verification process to determine whether two or more spoken utterances originate from the same speaker. To perform this verification, a verification system associated with the speaker recognition system compares audio samples (e.g., two audio samples) and determines whether a first audio sample corresponding to a first utterance spoken by the speaker matches or is highly similar to a second audio sample corresponding to another spoken utterance. When the first utterance matches or is highly similar to the other spoken utterance, the verification system identifies that the two utterances may come from the same speaker. Conversely, when the first utterance does not match or is highly similar to the other spoken utterance, the verification system identifies that each utterance may come from a different speaker. In some examples, the speaker recognition system compares text-related audio samples to determine a match. In other examples, the speaker recognition system compares two text-independent audio samples to determine whether the two audio samples originate from the same speaker. Typically, to perform speaker verification, a user of the speaker recognition system provides one or more spoken utterances to the verification system in order to register or enroll as a user with a speaker registration system. A user who registers with a speaker registration system can be called a "registered user," and therefore the terms 'registered user' and 'enrolled user' are used interchangeably. By registering with a speaker recognition system, a user's registration authorizes them to perform certain tasks associated with the system. Furthermore, user registration enables a verification system to use the registered utterance (i.e., the spoken utterance provided to the registered user) to later verify the user's identity. For example, after registering as an authorized user of a computing device with the speaker recognition system, when a user submits spoken utterance to the computing device, the speaker recognition system (e.g., the verification system) compares the submitted utterance with one or more registered utterances to determine if the user is an authorized user.

[0020] To perform verification, a verification system can use a verification model to generate predictions about whether the speaker of a utterance is an authorized or unauthorized user. Unfortunately, automated systems are not without flaws, and the verification model may sometimes incorrectly identify the speaker as authorized when the speaker is not, or incorrectly identify the speaker as unauthorized when the speaker is authorized. This incorrect identification is called a false acceptance of the speaker when the system identifies the speaker as authorized when the speaker is not. Conversely, this incorrect identification is called a false rejection of the speaker when the system identifies the speaker as unauthorized when the speaker is actually authorized. Because verification systems can have performance issues related to false acceptance and / or false rejection, it can be beneficial for the provider of the verification system to collect feedback on or evaluate the system's performance. However, evaluating the performance of a verification system becomes more complex when it is already in implementation (e.g., deployed on a computing device).

[0021] Traditional methods for evaluating the performance of verification systems are often cumbersome and / or involve a degree of manual input for review to ensure the system is properly evaluated. In one such method, volunteers call from various devices, identify themselves using identifiers such as personal identification numbers (PINs), and submit recordings tagged with the speaker's identifier. A set of these recordings can then be used to evaluate the performance of a verification system (e.g., a verification model) to determine how well the system performs verification on known speaker identities. The drawbacks of this approach are that volunteers are typically paid by their time, and the evaluation set of recordings is usually audited or retouched to ensure accurate evaluation. This can result in a costly and time-consuming process.

[0022] Another method to determine the performance of an authentication system is the collection of user data from devices using the system. In this method, a speaker identifier is assigned to user data from devices using the system, masking any information about the user. For example, this method assumes that audio from each device using the system is associated with a specific speaker, and assigns a Personal Identifier (PIN) to the audio as it is collected to eliminate any user association. Much like the call-to-volunteer approach, the audio data collected from devices using the system can then be used as an evaluation dataset to assess the system's performance. However, it is true that even when this process eliminates any user association and is based on user consent, the provider of the authentication system does not want to assume control over the user data or be responsible for any potential security issues that might jeopardize client data security. Furthermore, whenever a user or client provides their own data, even to a reliable source, the client relinquishes control over their data and faces the risk of being unable to prevent any downstream problems (e.g., security issues) arising from this data. Therefore, this approach faces the reality that a single device may include multiple users or speakers, but also raises privacy and / or security concerns.

[0023] To overcome the problems associated with various technologies that hinder the evaluation of authentication system performance, authentication system providers can alternatively utilize information collected by the authentication system about the authentication process or during the authentication process. In other words, when the authentication system verifies whether a speaker is a registered / authorized user, the authentication process generates data (e.g., metadata) about the interaction between the speaker and the authentication system. For example, the authentication process generates information similar to an event log for interactions during an authentication session. To illustrate, a registered speaker can speak uttered words to the device to perform an action requiring authorization. Once the device receives such a spoken utterance requiring authorization, the authentication system determines whether the speaker is registered on the device and, when the authentication system verifies the speaker, allows the device to perform a function, or generates a certain type of response indicating that the speaker cannot be verified. For this authentication session, the device and / or the authentication system can generate authentication data indicating that the authentication process should be initiated and that the speaker should be verified and accepted, or not verified and rejected. By collecting authentication data about the authentication process, the generated authentication data does not include the speaker's identity / identifier or any audio data / features associated with the speaker, while still providing key insights into the performance of the authentication system. More specifically, validation data can be used to construct metrics indicative of the performance of a validation system, and even more specifically, validation data can be used to construct validation models that the validation system can utilize, without sharing any personal or biometric data associated with the speaker. By using validation data that is detached from user-specific information and avoiding the collection of additional evaluation audio data (e.g., through volunteer calls), this performance evaluation technique overcomes several drawbacks of traditional performance evaluation techniques.

[0024] Figures 1A to 1D This is an example of a voice environment 100 including one or more users 10, who transmit spoken utterance 12 to an authentication device 110 (also referred to as device 110 or user device 110). User 10 (i.e., the speaker of utterance 12) can speak utterance 12 as a query or command requesting a response from device 110. Similarly, utterance 12, or at least a portion thereof, may include a calling phrase (e.g., a hot word, wake word, trigger phrase, etc.) that, when detected by device 110, triggers device 110 to wake from a sleep state and begin recording subsequent audio associated with the query or command to be processed by an automatic speech recognition (ASR) system running on device 110 and / or a remote server. Device 110 is configured to capture sound from one or more users 10 within the voice environment 100. Here, the audio sound may refer to uttered utterance 12 by user 10, which serves as a calling phrase, an audible query, a command from device 110, or audible communication captured by device 110. The voice-enabled system of device 110 or associated with device 110 can field the query of a command by answering the query and / or causing the execution of a command.

[0025] Here, device 110 is configured to detect utterance 12 and invoke a local or remote ASR system. Device 110 may correspond to any computing device associated with user 10 and is capable of receiving audio signal 12 corresponding to spoken utterance 12. Some examples of user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, platform computers, laptop computers, e-book readers, etc.), computers, wearable devices (e.g., smartwatches), music players, playback devices, smart appliances (e.g., smart TVs), and Internet of Things (IoT) devices, remote controls, smart speakers, etc. Device 110 includes data processing hardware 112 and memory hardware 114, which communicates with data processing hardware 112 and stores instructions that, when executed by data processing hardware 112, cause data processing hardware 112 to perform one or more operations related to utterance detection or some other form of utterance / speech processing (e.g., speaker verification).

[0026] In some examples, device 110 includes one or more applications (i.e., software applications), each of which can utilize one or more speech processing systems (e.g., speech recognition systems, text-to-speech systems, speaker recognition systems, verification system 140, etc.) associated with device 110 to perform various functions within the application. In some implementations, device 110 can detect utterance 12 and provide data characterizing utterance 12 to one or more speech processing systems. For example, device 110 includes a verification application configured to verify whether the speaker 10 of utterance 12 is an authorized user. For example, the verification application performs a speaker verification process involving accepting or rejecting a speaker 10's identity claim based on characteristics of the speaker's speech (i.e., audio features) determined from one or more utterances 12 from speaker 10. In some examples, device 110 has an application configured locally to perform local speaker verification or an application configured remotely to utilize remote resources to perform certain parts of speaker verification. Verification system 140 can perform text-related or text-independent speaker verification. Text-related speaker verification can be useful for identifying the speaker from audio features extracted from the call phrase spoken by the speaker, which is used to trigger device 110 to wake up from a sleep state.

[0027] Device 110 further includes an audio subsystem having an audio capture device (e.g., a microphone) 116 for capturing spoken utterance 12 within the speech environment 100 and converting the spoken utterance into an electrical signal. While device 110 implements a single audio capture device 116 in the illustrated example, device 110 may implement an array of audio capture devices 116 without departing from the scope of this disclosure, whereby one or more audio capture devices 116 in the array may not physically reside on device 110 but communicate with the audio subsystem (e.g., peripherals of device 110). For example, device 110 may correspond to a vehicle infotainment system utilizing a microphone array located throughout the vehicle. Additionally or alternatively, device 110 also includes a voice output device (e.g., a speaker) 118 for transmitting audible audio signals from device 110. For example, device 110 is configured to generate a synthesized playback signal in response to detected utterance 12. In other words, utterance 12 may correspond to a query responded to by device 110 using synthesized audio generated by device 110 and transmitted via voice output device 118. For illustration, device 110 can respond to detected speech 12 using a synthesized playback signal that informs speaker 10 that the verification process has verified his or her identity as an authorized user of device 110.

[0028] Furthermore, device 110 is configured to communicate with remote system 130 via network 120. Remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). Device 110 may utilize remote resources 132 to perform various functions related to speech processing, such as speech recognition and / or speaker identification / verification. For example, device 110 is configured to perform speaker recognition using verification system 140. System 140 may reside on device 110 (referred to as an on-device system) or reside remotely (e.g., reside on remote system 130) but communicate with device 110. In some examples, some portions of system 140 reside locally or on the device, while other portions reside remotely. For example, verification model 146 configured to perform speech verification for verification system 140 resides either remotely or locally. In some examples, the verification system 140 may be combined with other speech processing systems such as speech recognition systems, binarization systems, text-to-speech systems, etc. In some configurations, the location where the verification system 140 resides is based on processing requirements. For example, when the size or processing requirements of system 140 are considerable, system 140 may reside in a remote system 130. However, when device 110 can support the size or processing requirements of system 140, one or more systems 140 may reside on device 110 using data processing hardware 112 and / or memory hardware 114.

[0029] Verification system 140 is typically configured to receive verification query 142 from device 110 on behalf of user 10 and provide a response 144 indicating the result of a verification process performed by verification model 146. In some examples, verification model 146 receives the verification query 142 requiring verification as input and generates a response 144 as output regarding whether to verify the user 10 who submitted the verification query 142 to device 110 (i.e., the identity of user 10 authorized to use device 110 for the purpose of verifying query 142). Here, verification system 140 is capable of performing verification processes on any type of biometrics used for verification, including facial features (i.e., face recognition), voice features (i.e., speech recognition), handwriting features (i.e., handwriting recognition), etc. Figures 1A to 1D In some examples, verification system 140 performs a voice verification process, where verification query 142 includes audio features characterizing spoken utterance 12, and verification system 140 provides response 144 to device 110 to indicate the result of the voice verification process performed by verification model 146. Audio features may include d-vectors or i-vectors. In other words, system 140 is configured to perform a speaker verification process using verification model 146 to verify the identity of speaker 10 of utterance 12. For example, response 144 may indicate whether speaker 10 has registered with device 110 based on spoken utterance 12 (i.e., a registered speaker).

[0030] Still referencing Figure 1A Device 110 can use authentication system 140 to register one or more users 10 to device 110. By registering user 10 to device 110, this registration serves as a type of speaker registration process to register the user 10, 10 E The identifier is the authorized or registered user of device 110. Device 110 can be configurable, allowing registered users 10 of device 110 to... E Users 10 may have access to various functions of device 110 or permission to control various functions of device 110, while unauthorized users 10 not registered with device 110 are prohibited from performing one or more functions that device 110 is capable of performing. Optionally, device 110 may register multiple users 10. For multiple users 10, each registered user 10... E User accounts with specific permissions or rights regarding the functions of device 110 can be configured on device 110. For example, a family including a husband, wife, and nine-year-old daughter owns device 110. Here, when each adult registers with device 110, the adult can set parental controls that allow each adult access to or control all functions of device 110, but restrict their daughter (who is also a registered user 10) from accessing or controlling all functions of device 110. EThis allows for complete control over device 110. For example, parents could set up an account for their daughter to prevent her from modifying home automation controls, such as the thermostat schedule controlled by device 110. This means that a registered user 10... E It can have 10 with another registered user E The licenses or rights may overlap or differ from specific permissions or rights. Furthermore, instead of only generating permissions for the registrant, device 110 can also be configured to specify user 10 of device 110 (not the registered user 10). E () permission. For example, when user 10 of device 110 is not registered user 10 E Device 110 can be configured to perform limited functions (e.g., guest mode) or completely prevent unregistered users 10 from using device 110. Without restrictions, authorized registered users 10... E Device 110 can be authorized to access only registered users 10 E Resources with access permissions. For example, in a scenario with at least two registered users (e.g., 10). E In a household, when one of the registered users utters the voice command "Play my music playlist" captured by device 110, the verification system 140 is able to identify the specific registered speaker 10. E The identity of the speaker 10 is confirmed, and the device 110 is authorized to access a specific music playlist associated with the identified speaker 10, rather than other registered users 10. E Music playlist.

[0031] In some configurations, device 110 uses verification system 140 to perform a registration process that registers user 10 as a registered speaker for device 110. For example, a speaker recognition application associated with verification system 140 prompts user 10 to say one or more registration utterances 12, 12E, from which a speech signature can be generated for user 10. In some implementations, registration utterance 12E is, for example, a short phrase of one, two, three, four, or more words. Verification system 140 may prompt user 10 to say a predefined phrase as registration utterance 12E, or user 10 may spontaneously say and provide registration utterance utterance 12E based on a phrase not specifically provided for user 10. In some examples, user 10 may say multiple registration utterances 12E, where each registration utterance is the same phrase or a different phrase. Registration utterance 12E may include user 10 saying a predefined hot word, which is configured to trigger device 110 to wake from sleep to process spoken audio received after the predefined hot word. While the example shows user 10 providing multiple verbal registration statements 12E to device 110, other examples may include one or more users 10 accessing authentication system 140 from another device (e.g., a smartphone) to provide multiple registration statements 12E.

[0032] In some examples, after receiving a registered utterance 12E, the verification system 140 processes the registered utterance 12E to generate a speaker representation for each registered utterance 12E. The verification system 140 can generate a speaker signature for user 10 from all, some, or one speaker representation of the registered utterance 12E. In some examples, the speaker signature is the average of the corresponding speaker representations of multiple registered utterances 12E. In other examples, the speaker signature corresponds to a specific speaker representation from a particular registered utterance 12E, which is selected based on one or more criteria (e.g., based on the audio or speech quality of the audio of the selected registered utterance 12E). Once a speaker signature is generated for speaker 10, the speaker signature can be stored locally on device 110 or stored in a remote system 130 (e.g., stored in remote memory hardware 136).

[0033] After registration, when device 110 detects user 10's query utterances 12 and 12Q within voice environment 100, verification system 140 is configured to identify whether the speaker 10 of query utterance 12Q is the registered user 10 of device 110 based on query utterance 12Q. EThe query utterance 12Q can refer to a specific type of utterance or spoken phrase, such as a text-related verification phrase, or more generally, a text-independent phrase, which can include any utterance 12 spoken by user 10 after completing the registration process of one or more users 10. Here, the verification process performed by verification model 146 identifies whether the speaker 10 of the detected query utterance 12Q is a registered user 10. E And generate response 144 to indicate whether the speaker 10 is a registered user 10. E In some examples, verification model 146 can access information already provided to registered user 10. E The system generates a speaker signature, such as a d-vector or an i-vector, and compares the query utterance 12Q detected by speaker 10 with the speaker signature to determine whether the query utterance 12Q corresponds to a specific speaker signature. In these examples, when the query utterance 12Q corresponds to a specific speaker signature, the verification system 140 determines that the query utterance 12Q was generated by registered user 10. E The speaker 10, who utters and generates the instruction query 12Q, is a registered user 10. E The response number is 144.

[0034] Figure 1A The diagram illustrates a user who first registers with device 110 through a registration process. In other words, Figure 1A At least one registration utterance 12E is sent by user 10 to device 110 to register with device 110. After the registration process, user 10 utters a query utterance 12Q to device 110. When device 110 receives query utterance 12Q, device 110 transmits query utterance 12Q to verification system 140 so that verification model 146 can verify that speaker 10 is registered user 10. E Here, when verification model 146 verifies that speaker 10 is indeed registered user 10... E At that time, the verification system 140 will send a response 144 to the device 110 to indicate that the speaker 10 is a registered user 10 of the device 110. E Once verified, speaker 10 can register as user 10 by using device 110 or accessing some aspect of the computing capabilities provided by device 110. E Access to this computational function is granted, or more specifically, the computational function is assigned to user 10. In some examples, the query utterance 12Q includes hot words that follow the query. In these examples, the verification model 146 may verify speaker 10 based on only a portion of the audio corresponding to the hot words, only a portion of the audio corresponding to the query, or the entire audio including both the hot words and the query. In an additional example, the query utterance 12Q may be registered user 10. E Provided to verification model 146 to verify registered user 10E A specific verification phrase for the identity.

[0035] When speaker 10 initiates the verification process executed by model 146 of verification system 140, a verification session begins. This session may include one or more interactions between speaker 10 and verification system 140 (e.g., via device 110). Verification system 140 is configured to record / log a verification result 148, which indicates the interaction events that occurred during the verification process. Some examples of these interaction events that may be captured as verification result 148 include the receipt of query 12Q for verification, the rejection of query 12Q, the acceptance of query 12Q, the determination by verification system 140 (e.g., the probability of a registered speaker), feedback from speaker 10 regarding the verification process, or other verification log events. Here, an example of feedback from speaker 10 that may generate verification result 148 is when speaker 10 subsequently interacts with the result of query 12Q. In other words, further interaction with the actual results of query 12Q can instruct the verification system 140 to correctly verify speaker 10, as speaker 10 is further engaging in a response 144 to query 12Q (e.g., clicking on a search result or using a function authorized by the verification system 140). Due to the nature of these verification results 148, these log events typically do not include any sensitive user information (e.g., user identifiers) and / or do not include the actual audio data corresponding to query 12Q.

[0036] Figure 1B and 1C This describes some possible verification results 148 that may occur during the verification process of verification system 140. Figure 1B In the diagram, the verification system 140 includes three registered users 10, shown as user X, user Y, and user Z. EIn this example, user X submits a query 12Q to device 110 stating "What is next on my calendar?". Upon receiving this query 12Q, device 110 and / or authentication system 140 may first determine whether the query utterance 12Q invokes a need for authentication. The utterance 12Q typically invokes a need for authentication when it includes, for example, a request for personal information stored on or accessible by device 110, a request to perform restricted functions, or a request for other information designated as requiring authentication. For example, in the previous example of a family of three, the father might have a music playlist containing explicit lyrics that he has designated require authorization for so that his nine-year-old daughter would not be exposed to them. In some examples, device 110 and / or authentication system 140 may perform an authentication process on query 12Q regardless of whether query 12Q invokes a need for authentication. In other words, when speaker 10 speaks to device 110, verification system 140 can initially determine whether speaker 10 is registered user 10. E However, authorization is only enforced when verbal statement 12 requires authorization to execute. Figure 1B In this example, the spoken question, "What is next on my calendar?" is a query requesting personal calendar information. Here, verification system 140 initiates the verification process and attempts to verify the speaker, i.e., user X. Although user X is on the list of registered users, in this example, verification system 140 fails to verify the speaker, i.e., user X is registered user 10. E Based on this failure, verification system 140 sends response 144 to device 110, which in turn sends it to speaker 10, i.e., user X, to indicate verification failure. Response 144 states, "Sorry. I could not verify your voice." Since speaker 10 knows he or she is a registered user of device 110's verification system 140... E Therefore, speaker 10 can submit a second query 12Q2, for example, that second query attempts to issue the same query 12Q a second time using added wording (i.e., re-ask the same query 12Q). Figure 1B As shown, the second query 12Q2 is also proven unsuccessful because the verification system 140 failed to verify the voice of speaker 10, i.e., user X, and transmits this failure again as a second response 144, 144b to the statement "Sorry. I could not verify your voice." During this unsuccessful verification session, the verification system 140 records the verification result 148 that occurs during the session as verification data. For example, Figure 1B The diagram illustrates three validation results 148, 148a-c. The first validation result 148a indicates a punt event, which means that validation system 140 (e.g., validation model 146) failed to recognize speaker 10. The second validation result 148b indicates a punt and resubmission event, where speaker 10 was not recognized (i.e., punt), but speaker 10 resubmitted the same query 12Q2. The third validation results 148, 148c indicate a double punt event, which means that after the validation failure (i.e., punt) of the same query 12Q2 (i.e., resubmission) and subsequent resubmission, validation model 146 again failed to validate speaker 10 (i.e., double punt). In other words, each of these potential validation results 148a–148c indicates an incorrect rejection because speaker 10 is user X, and user X is actually registered user 10 of validation system 140. E .

[0037] As another example, Figure 1C The illustration depicts a scenario where verification result 148 is an error acceptance. One way to determine if verification system 140 has an interaction event that constitutes an error acceptance is when verification system 140 has multiple registered users 10. E At that time. For multiple registered users 10 E When speaker 10 utters utterance 12, verification system 140 can determine that utterance 12 corresponds to each registered user 10. E The possibility of this. For example, the verification system 140 generates a registration score for each registered user 10E, which indicates whether the speaker 10 is the corresponding registered user 10. E The possibility of 10 registered speakers. E When the registration score meets the acceptance threshold score, the verification system 140 is configured to designate the speaker 10 as a specific registered user 10. E Although this is the ideal behavior of the verification system 140, sometimes the performance of the verification model 146 indicates that for a specific utterance 12Q, the verification model 146 identifies more than one registered user 10. E A score that satisfies the acceptance threshold of discourse 12Q. For example, in a household, all registered users are 10. E The two brothers may have similar speech features, which can sometimes be difficult to distinguish for speaker verification. When this happens, verification result 148 indicates that the imposter has accepted an event, signifying that verification system 140 has identified more than one registered user 12. E An event that scores a point that meets the acceptance threshold. It is called an imposter acceptance event because speaker 10 is only a single registered user 10. EHowever, the verification system 140 accepts query 12Q as more than one registered user 10. E The identity. In other words, the accepted identity that is not actually spoken is that of an imposter. For example, Figure 1C The threshold acceptance value is 0.5, and the query utterance 12Q triggers a verification process that indicates more than one level of user 10. E Speaker 10 could be user X. Here, speaker 10 is user X and verification model 146 indicates that user X scores 0.65, user Y scores 0.8, and user Z scores 0.4. In this example, because user Y has the highest score, verification system 140 can return result 144, saying "User Y, you have an appointment at 5 pm," causing verification system 140 to conclude that user Y (i.e., the imposter) spoke 12Q instead of user X.

[0038] Figure 1D The illustration shows that the verification system 140 can use one of several verification models 146 (e.g., three verification models 142a–142c shown) to determine whether the speaker 10 of the utterance 12 is a registered user 10 of the device 10. E The provider of verification system 140 can use different models 146 to test, develop, maintain, or comparatively evaluate model 146. For example, group A depicts several users 10 (e.g., seven users 10) with corresponding devices 110 using first verification models 142, 142a to perform the verification process. Group B depicts four users 10 with corresponding devices 110 using a second verification model 142b. Group C depicts three users 10 with corresponding devices 110 using a third verification model 142c to perform the verification process. In this depicted example, the first verification model 146a may correspond to a production model or master model that has been deployed to most users 10 of verification system 140 and should generally be robust as it has stood the test of time. The second verification model 146b may correspond to an experimental or alternative model that the provider of verification system 140 can use to develop improvements to the verification process (e.g., fixing verification errors). For example, verification model 146 uses a neural network to determine whether the speaker 10 of utterance 12 is a registered user 10. EThe neural network can have different types of neural network architectures. Production validation models, such as the first validation model 146a, can use one type of neural network architecture, while experimental validation models, such as the second validation model 146b, use different types of neural network architectures or improvements / updates to the neural network architecture of the first validation model 146a. Alternatively, in some implementations, each model 146 is typically trained on a set of training data to teach the model 146 to verify the identity of a registration or sign-in. Therefore, the actual content of the training data provided to each model 146 can have some impact on its performance. At least for this reason, the difference between the production validation model 146a and the experimental validation model 146b can be that each model 146 has been trained on a different training dataset (e.g., validation training examples). The experimental validation model 146b can then be deployed in parallel with the production validation model 146a to identify whether any changes to the training data improve the validation performance of the validation system 140 using the experimental validation module 146b compared to the production validation module 146a. By using a comparative model approach, two or more models 146 can be deployed to a group of users 10, and data in the form of validation results 148 regarding the validation process of each corresponding model 146 can be collected to allow providers to compare and / or evaluate the performance similarities and differences between these models 146. For example, Figure 1D The illustration shows the generation of the first set of validation results 148 and 148 using group A of the first model 146a. S1 The second set of validation results 148 and 148s2 are generated using group B of the second model 146b, and the third set of validation results 148 and 148s2 are generated using group C of the third model 146c. s3 When using the comparison model approach, the provider can route the majority of user traffic to the production validation model 146a to avoid the deployment complexity of the experimental validation model 146b or any other potential problems. Here, the production validation model 146a will be deployed to the majority of validation system users (i.e., to the device 110 associated with user 10 performing the validation system 140), while the other models 146 used for comparison are deployed to a minority of validation system users (i.e., to the device 110 associated with user 10 performing the validation system 140).

[0039] In such Figure 1D In some examples, the validation system 140 uses more than two models 146 for comparative analysis. For example, Figure 1DGroup C, using the third verification model 146c, is depicted with a dashed line to indicate that the provider may optionally deploy more than two models for comparative analysis. Here, the third models 146 and 146c are shown to illustrate the use of the control verification model 146. The control verification model 146 may reference a redundant version of the production verification model 146 (e.g., the first verification model 146a) to confirm the comparison results between the production model 146a and the experimental model 146b. In other words, the comparison results between the production model 146a and the experimental model 146b can indicate some performance differences. By also comparing the performance results of the production model 146a or the experimental model 146b with the control model 146c, the provider can be able to identify whether the performance differences are actually due to the verification of model 146 or some other problem. For example, a certain number of production models 146a may be damaged or defective. Here, the performance of the production model 146a due to defects may cause the experimental model 146b to perform better in verifying speaker 10. However, when the experimental model 146b is compared with the control model 146c, which represents the production model 146a in an undamaged or defect-free state, the control model 146c indicates better performance than the experimental model 146b. This can indicate to the provider that the provider does not need to replace or deploy the experimental model 146b on a large scale and / or that there may be some additional problems in the production model 146a.

[0040] Because the verification system 140 can use several different verification models 146, the registration process can make user 10 a registered user 10 on each model 146. E This is to enable each model 146 to correctly verify whether the speaker of discourse 12 is a registered user 10. E Depending on the registration process, registering user 10 on multiple models may range from not being able to detect user 10 to requiring user 10 to provide model-specific registration (e.g., a specific registration phrase). User 10 may also need to appear to re-register when verification system 140 is updated or verification model 146 undergoes changes that will affect user registration to model 146. For example, based on comparative analysis, the provider decides to replace the first verification model 146a with a second verification model 146b. When this happens, the second verification model 146b may need to be deployed to a majority of users 10 to become the production model 146. In this case, a certain number of users 10 may need to re-register or register for the first time on the second verification model 146b, which is now the production model 146.

[0041] like Figures 1A to 1DAs shown, the voice environment 100 also includes a verification analyzer 200 (also referred to as analyzer 200). Analyzer 200 communicates with the verification system 140 and is configured to analyze the performance of one or more verification models 146. For example, although... Figure 2A and 2B The illustration shows analyzer 200 analyzing the performance of two and three models 146 separately, but analyzer 200 is scalable to perform analysis on any number of models 146 (e.g., even a single model 146). Analyzer 200 can be located locally (e.g., on user equipment 110) and configured to transmit its analysis to a provider of verification system 140 (e.g., via network 120), or remotely (e.g., in remote system 130) and communicate with the provider. In some examples, some parts of analyzer 200 can be located locally, while others are located remotely. Analyzer 200 is configured to receive verification results 148 as input and generate a performance comparison 202 as output. In some examples, performance comparison 202 triggers actions to be taken regarding the models 146 compared by analyzer 200. For example, analyzer 200 is configured to adjust the deployment of different models 146 to one or more groups of users 10. In other words, when experimental validation model 146b is proven to have better performance validation capabilities than production validation model 146a, analyzer 200 can replace the primary validation model or production validation model 146a with experimental or alternative validation model 146b.

[0042] Whether analyzing a single model 146 or performing a comparative analysis on two or more models 146, the analyzer 200 receives a set of validation results 148 from each model 146 it intends to analyze. S . refer to Figure 2A and 2B The analyzer 200 includes an identifier 210, a scorer 220, and a comparator 230. The identifier 210 is configured to receive verification results 148 and identify each verification result 148 corresponding to a specific performance metric 212. Here, the performance metric 212 may refer to a specific interaction event that occurs during a verification session, providing the model 146 with an indication of whether or not it can authenticate user 10 as one or more registered or audited users 10. E This is a specific indication. For example, performance metric 212 corresponds to an erroneous rejection event or an erroneous acceptance event during a verification session. As previously mentioned, some examples of erroneous rejection events include absconding, double absconding, absconding, and re-interrogation, while an example of an erroneous acceptance event is imposter acceptance. Note that other similar performance metrics 212 may indicate erroneous rejection events or erroneous acceptance events at verification model 146. Figure 2A As shown, the identifier 210 classifies the verification result 148 into one or more corresponding performance metrics 212, 212a–212n.

[0043] After identifying the validation results 148 corresponding to one or more performance metrics 212, the identifier 210 transmits one or more performance metrics 212a–212n to the scorer 220. The scorer 220 receives one or more performance metrics 212a–212n from the identifier 210 and is configured to determine a score 222 based on multiple validation results 148 including a given performance metric 212. For example, the scorer 220 is capable of determining a score 222 for each different type of performance metric 212. When the scorer 220 determines the score 222, the scorer 222 caters to a specific model 146. In other words, the score 222 forms a representation of the performance of a specific model for a given performance metric 212, because each set of validation results 148… S This corresponds to the given validation model 146. For illustration, Figure 2A The scorer 220 generates first scores 222, 222a of the first performance metric 212a of the first model 146a, second scores 222, 222b of the second performance metric 212b of the first model 146a, and nth scores 222, 222n of the nth performance metric 212n of the first model 146a. Because in Figure 2A In the process, analyzer 200 is comparing two models 146a–146b (e.g., primary validation model 146a and alternative validation model 146b), so scorer 220 scores the validation results 148 (i.e., a set of validation results 148 from a specific model 146) of the same performance metric 212 but relative to another model. S A score 222 is generated. In other words, the second model 146b can receive a score 222 from the scorer 220 regarding the same performance metric 212 as the first model 146a. This means that the first model 146a is based on a set of validation results 146 from the first model 146a. S1 The score 222 of the double abandonment metric 212 is received, while the second model 146b is based on a set of validation results 146 from the second model 146. S2 The score is 222 for the double abandonment measure of 212.

[0044] In some implementations, when score 222 corresponds to abandonment as performance metric 212, abandonment score 222 refers to the number of abandonments within a specific set of verification results 148S divided by the total number of verification results 148 within that set. For score 222 corresponding to double abandonment as performance metric 212, double abandonment score 222 can refer to a specific set of verification results 148. S The number of double abandonments within a set is divided by the total number of validation results 148 within that set. When score 222 corresponds to abandonment and re-entry as a performance metric 212, the abandonment and re-entry score 222 can refer to the initial set of validation results 148 within that set.S The combined count of abandoned and resubmitted instances of the same query 12Q within the group is divided by the total number of verification results 148 within that group. When score 222 corresponds to imposter acceptance as performance metric 212, the imposter acceptance score 222 can refer to one or more registered users 10 who possess an acceptance score threshold capable of verifying user 10. E The percentage of queries 12Q. Here, the percentage of queries 12Q is the percentage that has the ability to validate a specific set of validation results. S One or more registered users within the user 10 who meet the acceptance score threshold E The query count of 12Q is divided by the total number of queries of 12Q in the group.

[0045] In some examples, the scorer 220 is based on a set of validation results 148 S A score 222 for validating model 146 is generated by one or more relationships between the types of performance metrics 212 within the model. A relationship may refer to the relationship between metric(s)212 representing false acceptance events and metric(s)212 representing false rejection events. For illustration, scorer 220 can generate score 222 by combining false acceptance events (i.e., false acceptance errors) and false rejection events (i.e., false rejection errors) into a single cost, which scorer 220 then transforms into score 222. In some implementations, score 222 refers to a cost function that is a weighted combination of false acceptance events and false rejection events. In some configurations, the number of false acceptance events and / or false rejection events can cause scorer 220 to be able to identify the probability of a false acceptance event and / or false rejection event occurring at validating model 146. Here, score 222 for model 146 may be equal to a first cost coefficient (also called a weight) multiplied by the probability combination of false acceptance events (e.g., plus) a second cost coefficient multiplied by the probability of false rejection events. In other methods, the cost of forming score 222 can be represented as a combination of error rejection cost components and error acceptance cost components, where each cost component is represented as a cost weight assigned to the error event multiplied by the probability of the same speaker and the probability of the error event. Although these are some examples of algorithms by which scorer 220 generates score 222, other algorithms representing the relationship between metrics 212 can be used to generate score 222.

[0046] Scorer 220 transmits performance metric scores 222 to comparator 230, enabling comparator 230 to determine whether the validation capability of validation model 146 (e.g., production validation model 146a) is superior to that of another validation model 146 (e.g., experimental validation model 146b). To perform this determination, comparator 230 compares the scores 222 of the same performance metric 212 among models 146. For example, Figure 2AA comparator 230 is described that compares the first performance metric score 222a of the first performance metric 212 of the first model 146a with the first performance metric score 222a of the first performance metric 212 of the second model 146b. The comparator 230 can be configured to compare individual performance metric scores 222 on two or more models 146, or to compare multiple performance metric scores 222 on two or more models 146 simultaneously. When the analyzer 200 uses the comparator 230 to simultaneously compare multiple performance metric scores 222 on two or more models 146, the analyzer 200 can provide an indication of the importance of each type of performance metric 212. For example, the analyzer 200 applies weights to each type of performance metric score 222 to indicate its overall importance to the validation capability of the validation process.

[0047] Continue to refer to Figure 2A and 2B In some configurations, comparator 230 is configured to generate a performance comparison 202, which indicates whether one model 146 is superior to another model 146. For example, in Figure 2A In the middle, comparator 230 compares two models 146a-146b (e.g., primary validation model 146a and alternative validation model 146b), while... Figure 2B In this configuration, comparator 230 compares three models 146a-c (e.g., primary or production validation model 146a, alternative or experimental validation model 146b, and control validation model 146c). In some examples, when the comparison indicates that the performance score(s) 222 of the second model(s) 146b is superior to the corresponding performance score(s) 222 of the first model(s) 146a, comparator 230 suggests replacing the first model(s) 146a with the second model(s). In some cases, analyzer 200 actually replaces the first model(s) 146a with the second model(s) rather than passively recommending such a replacement. To replace model(s) 146 with another model(s) (e.g., to replace the first model(s) 146a with the second model(s) 146b), analyzer 200 may transfer or reallocate the number of users(s) 10 or devices(s) 110 assigned to each model(s). For example, when replacing the production validation model 146 used by the majority of user equipment 110, the analyzer 200 instead deploys the experimental validation model 146b to the majority of user equipment 110 and stops the majority of user equipment from using the original production validation model 146a. In situations such as... Figure 2BIn some configurations, when comparator 230 compares the performance metrics 222 of more than two models 146, analyzer 200 can set conditions regarding when analyzer 200 will recommend or actively replace model 146. For example, in the case of three models 146a-c with production validation model 146a, experimental validation model 146b, and control validation model 146c, analyzer 200 sets the condition that experimental validation model 146b needs to have performance exceeding that of each of the other models 146 (e.g., exceeding that of production validation model 146a and control validation model 146c) (based on performance metric score 222).

[0048] Figure 3 This is a flowchart illustrating an example layout of the operation of method 300 for evaluating the performance of verification model 146. At operation 302, method 300 receives the first set of verification results 148, 148... S1 The first set of verification results is 148, 148. S1 Each verification result 148 indicates whether the main verification model 146, 146a executed on the corresponding user equipment 110 authenticated the corresponding user 10 as one or more registered users 10 of the corresponding user equipment 110 during the corresponding interaction between the corresponding user 10 and the corresponding user equipment 110. E One of them. At operation 304, method 300 also includes receiving a second set of verification results 148, 148. S2 The second set of verification results is 148, 148. S2 Each verification result 148 indicates whether the alternative verification models 146, 146b executed on the corresponding user equipment 110 authenticated the corresponding user 10 as one or more registered users 10 of the corresponding user equipment 110 during the interaction between the corresponding user 10 and the corresponding user equipment 110. E One of them. At operation 306, method 300 identifies the first set of verification results 148. S1 The second set of verification results 148 S2 Each verification result 148, including performance metric 212, indicates whether a corresponding one of the primary verification models 146, 146a or alternative verification models 146, 146b is able or unable to authenticate the corresponding user 10 as one or more registered users 10 during a corresponding interaction between the corresponding user 10 and the corresponding user device 110. E One of the validation models. At operation 308, method 300 is based on the first set of validation results 148. S1 The multiple validation results 148, including performance metric 212, are used to determine the first performance scores 222, 222a of the main validation models 146, 146a. At operation 310, method 300 bases its calculations on the second set of validation results 148.S2 The method 300 determines second performance scores 222, 222a for alternative primary verification models 146, 146b based on multiple verification results 148, including performance metric 212. At operation 312, the method 300 determines whether the verification capability of the alternative verification models 146, 146b is superior to that of the primary verification models 146, 146a, based on the first and second performance scores 222, 222a. At operation 314, when the verification capability of the alternative verification models 146, 146b is superior to that of the primary verification models 146, 146a, the method 300 replaces the primary verification models 146, 146a executed on at least one corresponding user equipment 110 with the alternative verification models 146, 146b.

[0049] Figure 4 This is a flowchart of another example arrangement of the operation of method 400 for evaluating the performance of verification model 146. At operation 402, method 400 receives a set of verification results 148, 148 S1 Among them, the verification results of this group are 148 and 148. S Each verification result 148 indicates whether the verification model 146 executed on the corresponding user equipment 110 authenticated the corresponding user 10 as one or more registered users 10 of the corresponding user equipment 110 during the corresponding interaction between the corresponding user 10 and the corresponding user equipment 110. E One of them. At operation 404, method 400 identifies the set of validation results 148. S Each verification result 148 includes an error rejection metric 212, which indicates that the verification model 146 incorrectly rejects one or more registered users 10 that identify the corresponding user 10 as the corresponding user device 110. E One of them. At operation 406, method 400 determines the validation result 148 for this group. S This includes each verification result 148 of the error acceptance metric 212. The error acceptance metric 212 indicates that the verification model 146 identifies at least two registered users 10 of the corresponding user equipment 110 that meet the verification threshold. E The associated corresponding validation scores. At operation 408, method 400 bases its calculations on validation results 148, 148 in that set. S Multiple validation results 148, including the false rejection metric 212 and the false acceptance metric 212, are used to generate a performance score 222 for the validation model 146.

[0050] Figure 5This is a schematic diagram of an example computing device 500 that can be used to implement the systems (e.g., verification system 140 and / or analyzer 200) and methods (e.g., methods 300, 400) described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, connections, and relationships between components shown herein, as well as the functionality of the components, are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0051] Computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of components 510, 520, 530, 540, 550, and 560 is interconnected using various buses, and the components may be mounted on a general-purpose motherboard or otherwise. The processor 510 is capable of processing instructions for execution within computing device 500, including instructions stored in the memory 520 or on the storage device 530 for displaying graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to the high-speed interface 540. In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and various types of memory, depending on the situation. In addition, multiple computing devices 500 can be connected, each of which provides a portion of the necessary operation (e.g., as a server group, blade server cluster, or multiprocessor system).

[0052] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, volatile memory cells(s), or non-volatile memory cells(s). Non-temporarily stored memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0053] Storage device 530 provides mass storage for computing device 500. In some embodiments, storage device 530 is a computer-readable medium. In various embodiments, storage device 530 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or an array of devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0054] High-speed controller 540 manages ultra-wideband operation of computing device 500, while low-speed controller 560 manages lower ultra-wideband operation. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerometer), and high-speed expansion port 550 which can accept various expansion cards (not shown). In some embodiments, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, for example, via a network adapter.

[0055] As shown in the figure, the computing device 500 can be implemented in a variety of different forms. For example, the computing device can be implemented as a standard server 500a, or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0056] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can be included in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be used for dedicated or general purposes and coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0057] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0058] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to said mass storage device, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0059] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer with a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or touchscreen) to display information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input generated from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0060] Several embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method (300) for evaluating and validating a model, comprising: A first set of verification results (148) is received at the data processing hardware (510), each of the first set of verification results (148) indicating whether the main verification model (146a) executed on the corresponding user equipment (110) authenticates the corresponding user (10) as one of one or more registered users (10) of the corresponding user equipment (110) during a corresponding interaction between the corresponding user (10) and the corresponding user equipment (110); A second set of verification results (148) is received at the data processing hardware (510), each of the second set of verification results (148) indicating whether an alternative verification model (146b) executed on the corresponding user equipment (110) authenticates the corresponding user (10) as one of one or more registered users (10) of the corresponding user equipment (110) during the interaction between the corresponding user (10) and the corresponding user equipment (110); The data processing hardware (510) identifies each verification result (148) in the first set of verification results and the second set of verification results (148) that includes a performance metric (212), the performance metric indicating whether a corresponding one of the main verification model (146a) or the alternative verification model (146b) is a verification model that is capable or incapable of authenticating the corresponding user (10) as one of the one or more registered users (10) during the corresponding interaction between the corresponding user (10) and the corresponding user device (110); The data processing hardware (510) determines a first performance score (222) of the master verification model (146a) based on a plurality of verification results (148) including the performance metric (212) identified in the first set of verification results (148). The data processing hardware (510) determines a second performance score (222) of the alternative verification model (146b) based on a plurality of verification results (148) including the performance metric (212) identified in the second set of verification results (148). The data processing hardware (510) determines whether the verification capability of the alternative verification model (146b) is superior to that of the main verification model (146a) based on the first performance score (222) and the second performance score (222). as well as When the verification capability of the alternative verification model (146b) is superior to that of the primary verification model (146a), the data processing hardware (510) uses the alternative verification model (146b) to replace the primary verification model (146a) executed on at least one corresponding user equipment (110).

2. The method (300) according to claim 1, further comprising: A third set of verification results (148) is received at the data processing hardware (510), each of the third set of verification results (148) indicating whether the control verification model (146) executed on the corresponding user equipment (110) authenticates the corresponding user (10) as one of one or more registered users (10) of the corresponding user equipment (110) during a corresponding interaction between the corresponding user (10) and the corresponding user equipment (110); Each verification result (148) in the third set of verification results (148) is identified by the data processing hardware (510) and includes the performance metric (212), the performance metric indicating whether the control verification model (146) is a control verification model that is capable of or incapable of verifying the identity of the corresponding user (10) during the corresponding interaction between the corresponding user (10) and the corresponding user device (110); The data processing hardware (510) determines a third performance score (222) of the control verification model (146) based on a plurality of verification results (148) including the performance metric (212) identified in the third set of verification results (148). as well as The data processing hardware (510) determines whether the verification capability of the alternative verification model (146b) is superior to the verification capability of the control verification model (146) based on the second performance score (222) and the third performance score (222); The substitution of the primary verification model (146a) executed on each corresponding user equipment (110) with the alternative verification model (146b) includes replacing the primary verification model (146a) executed on each corresponding user equipment (110) with the alternative verification model (146b) when the verification capability of the alternative verification model (146b) is superior to the verification capability of both the primary verification model (146a) and the control verification model (146).

3. The method (300) according to claim 1, further comprising: The primary verification model (146a) is initially allocated by the data processing hardware (510) to be executed on a first plurality of user devices (110) and the alternative verification model (146b) to be executed on a second plurality of user devices (110); The replacement of the primary verification model (146a) executed on at least one corresponding user equipment (110) includes the reallocation of the alternative verification model (146b) to replace the primary verification model (146a) executed on at least one corresponding user equipment (110) among the first plurality of user equipments (110).

4. The method (300) of claim 3, wherein, The first plurality of user equipment (110) is greater than the second plurality of user equipment (110).

5. The method (300) according to claim 1, wherein, The verification results (148) received in the first set of verification results and the second set of verification results (148) do not include the user identifier (210) that identifies the corresponding user (10).

6. The method (300) according to claim 1, wherein, The verification results (148) received in the first set of verification results and the second set of verification results (148) do not include audio data associated with the corresponding interaction between the corresponding user (10) and the corresponding user device (110).

7. The method (300) according to claim 1, wherein: The master validation model (146a) is trained on the first set of training data; and The alternative verification model (146b) is trained on a second set of training data that is different from the first set of training data.

8. The method (300) according to claim 1, wherein, The primary verification model (146a) includes a first neural network, and the alternative verification model (146b) includes a second neural network having a different neural network architecture than the first neural network.

9. The method (300) according to claim 1, wherein, The performance metric (212) includes an error rejection metric (212) that indicates that the corresponding verification model in the primary verification model (146a) or the alternative verification model (146b) incorrectly rejects identifying the corresponding user (10) as one of the one or more registered users (10) of the corresponding user device (110).

10. The method (300) according to claim 9, wherein, The false rejection metric (212) includes one of the following: Abandon metric (212), which indicates that the corresponding one of the main verification model (146a) or the alternative verification model (146b) authorizes the corresponding user (10) to obtain visitor privileges during the corresponding interaction with the corresponding user device (110); A double-abandonment metric (212), the double-abandonment metric indicating that immediately following a previous corresponding interaction with the corresponding user device (110) where the same corresponding user (10) was authorized to obtain visitor privileges, one of the corresponding authentication models (146a) or the alternative authentication model (146b) authorized the corresponding user (10) to obtain visitor privileges during the corresponding interaction with the corresponding user device (110); and A waiver and re-question metric (212) indicates that immediately after the corresponding one in the main authentication model (146a) or the alternative authentication model (146b) has authorized the same corresponding user (10) to obtain visitor privileges during a previous corresponding interaction with the corresponding user device (110), the corresponding one in the main authentication model (146a) or the alternative authentication model (146b) has authorized the corresponding user (10) to obtain visitor privileges during the corresponding interaction with the corresponding user device (110) when the corresponding interaction corresponds to the corresponding user (10) requesting authorization of privileges.

11. The method (300) according to any one of claims 1 to 10, wherein, The performance metric (212) includes an error acceptance metric (212) indicating that the corresponding one of the primary verification model (146a) or the alternative verification model (146b) incorrectly accepts the corresponding user (10) as one of the one or more registered users (10) of the corresponding user device (110).

12. The method (300) according to claim 11, wherein, The error acceptance metric (212) includes a proxy imposter acceptance metric (212), which indicates the corresponding verification score in the primary verification model (146a) or the alternative verification model (146b) that is associated with at least two registered users (10) of the corresponding user device (110) that meet the verification threshold.

13. A system (500) for evaluating and validating a model, comprising: Data processing hardware (510); as well as A memory hardware (520) communicates with the data processing hardware (510), the memory hardware (520) stores instructions, and when executed on the data processing hardware (510), the instructions cause the data processing hardware (510) to perform operations, the operations including: Receive a first set of verification results (148), each of the first set of verification results (148) indicating whether the main verification model (146a) executed on the corresponding user equipment (110) authenticates the corresponding user (10) as one of one or more registered users (10) of the corresponding user equipment (110) during a corresponding interaction between the corresponding user (10) and the corresponding user equipment (110); Receive a second set of verification results (148), each of the second set of verification results (148) indicating whether the alternative verification model (146b) executed on the corresponding user equipment (110) authenticates the corresponding user (10) as one of one or more registered users (10) of the corresponding user equipment (110) during the interaction between the corresponding user (10) and the corresponding user equipment (110); Each verification result (148) in the first set of verification results and the second set of verification results (148) includes a performance metric (212), the performance metric indicating whether a corresponding one of the main verification model (146a) or the alternative verification model (146b) is a verification model that is able or unable to authenticate the corresponding user (10) as one of the one or more registered users (10) during the corresponding interaction between the corresponding user (10) and the corresponding user device (110); A first performance score (222) of the master verification model (146a) is determined based on a plurality of the verification results (148) including the performance metric (212) identified in the first set of verification results (148). A second performance score (222) of the alternative verification model (146b) is determined based on a plurality of the verification results (148) including the performance metric (212) identified in the second set of verification results (148). Based on the first performance score (222) and the second performance score (222), determine whether the verification capability of the alternative verification model (146b) is superior to that of the main verification model (146a); and When the verification capability of the alternative verification model (146b) is superior to that of the primary verification model (146a), the primary verification model (146a) executed on at least one corresponding user equipment (110) is replaced by the alternative verification model (146b).

14. The system (500) according to claim 13, wherein, The operation further includes: Receive a third set of verification results (148), each of the third set of verification results (148) indicating whether the control verification model (146) executed on the corresponding user equipment (110) authenticates the corresponding user (10) as one of one or more registered users (10) of the corresponding user equipment (110) during a corresponding interaction between the corresponding user (10) and the corresponding user equipment (110); Each verification result (148) in the third set of verification results (148) includes the performance metric (212), the performance metric indicating whether the control verification model (146) is a control verification model that is able or unable to verify the identity of the corresponding user (10) during the corresponding interaction between the corresponding user (10) and the corresponding user device (110); A third performance score (222) of the control verification model (146) is determined based on multiple verification results (148) including the performance metric (212) identified in the third set of verification results (148); and Based on the second performance score (222) and the third performance score (222), it is determined whether the verification capability of the alternative verification model (146b) is superior to the verification capability of the control verification model (146); The substitution of the primary verification model (146a) executed on each corresponding user equipment (110) with the alternative verification model (146b) includes replacing the primary verification model (146a) executed on each corresponding user equipment (110) with the alternative verification model (146b) when the verification capability of the alternative verification model (146b) is superior to the verification capability of both the primary verification model (146a) and the control verification model (146).

15. The system (500) according to claim 13, wherein, The operation further includes: The primary verification model (146a) is initially assigned to execute on a first plurality of user devices (110) and the alternative verification model (146b) is assigned to execute on a second plurality of user devices (110); The replacement of the primary verification model (146a) executed on at least one corresponding user equipment (110) includes the reallocation of the alternative verification model (146b) to replace the primary verification model (146a) executed on at least one corresponding user equipment (110) among the first plurality of user equipments (110).

16. The system (500) according to claim 15, wherein, The first plurality of user equipment (110) is greater than the second plurality of user equipment (110).

17. The system (500) according to claim 13, wherein, The verification results (148) received in the first set of verification results and the second set of verification results (148) do not include the user identifier (210) that identifies the corresponding user (10).

18. The system (500) according to claim 13, wherein, The verification results (148) received in the first set of verification results and the second set of verification results (148) do not include audio data associated with the corresponding interaction between the corresponding user (10) and the corresponding user device (110).

19. The system (500) according to claim 13, wherein: The master validation model (146a) is trained on the first set of training data; and The alternative verification model (146b) is trained on a second set of training data that is different from the first set of training data.

20. The system (500) according to claim 13, wherein, The primary verification model (146a) includes a first neural network, and the alternative verification model (146b) includes a second neural network having a different neural network architecture than the first neural network.

21. The system (500) according to claim 13, wherein, The performance metric (212) includes an error rejection metric (212) that indicates that the corresponding verification model in the primary verification model (146a) or the alternative verification model (146b) incorrectly rejects identifying the corresponding user (10) as one of the one or more registered users (10) of the corresponding user device (110).

22. The system (500) according to claim 21, wherein, The false rejection metric (212) includes one of the following: Abandon metric (212), which indicates that the corresponding one of the main verification model (146a) or the alternative verification model (146b) authorizes the corresponding user (10) to obtain visitor privileges during the corresponding interaction with the corresponding user device (110); A double-abandonment metric (212), the double-abandonment metric indicating that immediately following a previous corresponding interaction with the corresponding user device (110) where the same corresponding user (10) was authorized to obtain visitor privileges, one of the corresponding authentication models (146a) or the alternative authentication model (146b) authorized the corresponding user (10) to obtain visitor privileges during the corresponding interaction with the corresponding user device (110); and A waiver and re-question metric (212) indicates that immediately after the corresponding one in the main authentication model (146a) or the alternative authentication model (146b) has authorized the same corresponding user (10) to obtain visitor privileges during a previous corresponding interaction with the corresponding user device (110), the corresponding one in the main authentication model (146a) or the alternative authentication model (146b) has authorized the corresponding user (10) to obtain visitor privileges during the corresponding interaction with the corresponding user device (110) when the corresponding interaction corresponds to the corresponding user (10) requesting authorization of privileges.

23. The system (500) according to any one of claims 13 to 22, wherein, The performance metric (212) includes an error acceptance metric (212) indicating that the corresponding one of the primary verification model (146a) or the alternative verification model (146b) incorrectly accepts the corresponding user (10) as one of the one or more registered users (10) of the corresponding user device (110).

24. The system (500) according to claim 23, wherein, The error acceptance metric (212) includes a proxy imposter acceptance metric (212), which indicates the corresponding verification score in the primary verification model (146a) or the alternative verification model (146b) that is associated with at least two registered users (10) of the corresponding user device (110) that meet the verification threshold.

Images