Method for Obtaining Enrolment Data for a Speaker Verification Model

Abstract

A method for obtaining enrolment data for a speaker verification mode, comprises capturing audio samples of utterances spoken by a speaker, generating an embedding vector for some samples, identifying

Description

Field

[001] The present application generally relates to a method for obtaining enrolment data for a speaker verification model. In particular, the present application provides a computer-implemented method, performed on a user device, for obtaining enrolment data for use by a speaker verification model, without requiring a user to actively record audio samples. Background

[002] Voice authentication systems are increasingly used in banking, healthcare and insurance. Furthermore, device unlock, payment authorisation, and other activities performed using user devices (such as smartphones) require high security and therefore need the accuracy of voice biometrics.

[003] Current technologies enable speaker verification models to more accurately recognise individual speakers by personalizing the models to individual speakers / users. However, to use speaker verification models, a user currently needs to record several audio samples in which they speak. These audio samples are recorded in quiet environments so that the user can be clearly heard, and the user needs to say a specific phrase, such as “Hi Bixby”. However, this places a burden on a user to take action before the speaker verification model can be used, which may prevent users from making use of the model.

[004] The present applicant has therefore identified the need for an improved technique for gathering the data needed to enable speaker verification models to be used. Summary

[005] In a first approach of the present techniques, there is provided a computer-implemented method, performed on a user device, for obtaining enrolment data for use by a speaker verification model, the method comprising: capturing a plurality of audio samples of utterances spoken by at least one speaker; generating an embedding vector for at least some of the plurality of audio samples; identifying a subset of the embedding vectors which correspond to a specific speaker of the at least one speaker; and calculating an embedding vector for the specific speaker using the identified subset of embedding vectors, the embedding vector representing a voice of the specific speaker and forming enrolment data for the speaker verification model. The speaker verification model may be, but is not necessarily limited to, a machine learning, ML, model.

[006] As explained above, known techniques record a number of enrolments, for example five positive samples, from a device user and store the enrolments on memory during speaker verification setup. This involves the device user pre-recording a set of “clean” utterances, that are used for training personalisation. The utterances are “clean” in that they are recorded in a non-noisy environment, by one speaker. After setup, speaker verification uses the prerecorded enrolments to verify a user from real-time recordings.

[007] In contrast, the present techniques provide automatic enrolment for speaker verification, where enrolment samples (i.e. audio samples, also referred to interchangeably herein as “enrols” or “enrolments”) for speaker verification are found automatically. The enrolment samples may be calculated automatically while the user interacts with a user device. For example, the user may interact with the user device by speaking a predefined wake-word and then instructing the user device to perform operations, such as playing music or turning on lights. The device user does not need to record enrolment samples actively or before starting to use a speaker verification pipeline. In other words, no initial speaker verification setup stage is needed. Additionally, a user may not need to record “clean” utterances as references. Multiple enrolments may not need to be stored on a user device’s memory. The enrolments may be used more accurately, owing to positive data filtering. Intelligent decisions may be made about which training samples are important (and, hence, should be saved on-device) and which are not (and, hence, can be deleted). This reduces the amount of data to be stored on-device. This can be particularly effective for devices with limited internal storage. The enrolment sample may also be used to personalise a speaker verification model as explained in more detail below.

[008] Experimental results indicate that such automatic enrolment may significantly improve a false rejection (FR) rate compared to speaker verification employing five pre-recorded enrols. A false acceptance (FA) may increase slightly when automatic enrolment is used. However, the improved FR rate may be an acceptable compromise in terms of useability and given a significant reduction of memory used for operations such as speaker verification personalisation. Furthermore, the usability of speaker verification improves as a user is not required to record the enrolment samples. Experimental results indicate that ten or fifteen positive utterances may be an optimal number of utterances for a good balance between FA and FR results.

[009] Generating an embedding vector may also be referred to herein as generating embeddings or as extracting embeddings. Generating embeddings of “at least some of” the utterance recordings encompasses both generating embeddings of some but not all of the utterance recordings and generating embeddings all of the utterance recordings. Embeddings may be not generated for some of the utterance recordings, for example if they are discarded by a keyword detector. The term “speaker” is used herein to mean an entity (such as a person) that speaks, and not a loudspeaker (which may sometimes also be referred to as a “speaker”).

[010] The identifying may comprise: clustering, in embedding space, the generated embedding vectors into at least one cluster; and selecting the embedding vectors in one cluster of the at least one cluster as the subset of embedding vectors which correspond to a specific speaker.

[011] The clustering may comprise using a clustering algorithm. The clustering algorithm may be an unsupervised algorithm. Using an unsupervised clustering algorithm facilitates automatic enrolment. The embeddings in the cluster of embeddings are likely to be highly representative of the specific speaker. The resulting calculated embedding vector may therefore also be highly representative of the specific speaker. This may result in more accurate speaker verification which can, for example, improve security. The clustering algorithm may be any one of: a density-based spatial clustering of applications with noise, DBSCAN; k-means clustering; and / or an Infomap algorithm.

[012] Experimental results indicate that DBSCAN may work well even in extreme cases, where 70% of a training set includes outliers, in other words audio samples of utterances that are not from the specific speaker.

[013] The method may comprise: discarding audio samples corresponding to embedding vectors which are not within a cluster. This can improve accuracy in respect of the calculated embedding vector for the specific speaker, since the calculated embedding vector will be more representative of the specific speaker. Discarding non-ideal positive samples may also improve training personalisation. Discarding such utterance recordings also frees up on-device storage.

[014] The calculating may comprise: calculating a mean embedding vector using the embedding vectors in the selected cluster. Speaker verification performance may be enhanced by using a “centroid” of the embeddings within a cluster, instead of a centroid of the embeddings of all audio samples.

[015] The identifying may comprise: classifying, using an out-of-vocabulary, OOV, classifier, the plurality of audio samples into audio samples that include a predefined keyword, and audio samples that do not include the predefined keyword; wherein the clustering comprises clustering the embedding vectors for audio samples that include the predefined keyword.

[016] OOV may be used to provide enhanced filtering of utterances that wrongly pass a keyword-spotting block. Higher speaker verification performance may be achieved by preventing OOV samples from being used.

[017] Experimental results indicate that using the OOV classifier improves the FR rate, potentially significantly, compared to the OOV classifier not being used, in the presence of OOV data.

[018] Embeddings that are classified using the OOV classifier may be generated using a pretrained, text-dependent speaker verification model, or any ML model that is able to extract meaningful embeddings.

[019] An OOV detector may provide the OOV classifier functionality.

[020] In some cases, the classifying may comprise: using a supervised OOV classification ML model, the supervised OOV classification ML model having been trained, using a plurality of training audio samples which are labelled as containing a predefined keyword, to separate the plurality of audio samples into audio samples that include the predefined keyword, and audio samples that do not include the predefined keyword.

[021] Using supervised methods, audio samples containing the predefined keyword can be separated from OOV audio samples. This may be particularly effective where a supervised model can be pre-trained and used on-device.

[022] Using the supervised OOV classification method may comprise: using a linear support vector machine, SVM, by applying it, in embedding space, to the embeddings generated for audio samples. That is, the classification may happen after the embeddings have been generated for the audio samples, in embedding space, rather than by analysing the audio samples themselves.

[023] Experimental results indicate that a linear SVM works well in identifying and removing OOV utterances. A linear SVM may be able to remove most OOV samples, leaving DBSCAN to find a centroid with a small number of positive samples.

[024] Other examples of supervised OOV classification methods include, but are not limited, decision trees and random forests.

[025] In other cases, the classifying may comprise: using an unsupervised OOV classification algorithm, in embedding space, to identify a cluster of embeddings of audio samples that do not include the predefined keyword. That is, the classification may happen after the embeddings have been generated for the audio samples, in embedding space, rather than by analysing the audio samples themselves.

[026] Using unsupervised methods enables a cluster of OOV utterances to be found and labelled. This may be particularly effective when running OOV classification in real-time, while a user device is recording new data.

[027] Examples of unsupervised methods include, but are not limited to, DBSCAN, Gaussian mixture model (GMM), and k-means clustering.

[028] Experimental results indicate that DBSCAN, for example, is effective for up to 50% of OOV outliers in a training set. For example, the FR rate may improve almost seven-fold.

[029] The method may comprise: discarding any audio sample that does not include the predefined keyword. This helps automatic enrolment sampling find an accurate enrolment, even when false positives are passed through a keyword-spotting model in a speaker verification pipeline.

[030] The method may comprise, before the classifying: detecting, using a keyword detector, the predefined keyword in captured audio samples; and storing the audio samples in which the predefined keyword is detected for use in the classifying. The keyword detector provides initial keyword filtering. However, the keyword detector may not be as accurate as the OOV classifier. For example, the keyword detector may be used for other tasks that do not need as a high a level of keyword detection accuracy as speaker verification. Such other tasks may continue to use the keyword detector accordingly. However, accuracy may be enhanced for speaker verification by additionally using the OOV classifier.

[031] Capturing a plurality of audio samples may comprise: capturing audio samples spoken by at least two speakers.

[032] Unlike known techniques in which enrols are “clean”, with one speaker and reduced noise, the present techniques can adapt to multiple utterances of multiple, different speakers. This can result in more natural and passive enrolment compared to known techniques.

[033] When audio samples represent multiple speakers, the identifying may comprise: clustering, in embedding space, the generated embedding vectors into at least two clusters; and selecting the embedding vectors in a first cluster of the at least two clusters as the subset of embedding vectors which correspond to a first speaker of the at least two speakers, and selecting the embedding vectors in a second cluster of the at least two clusters as the subset of embedding vectors which correspond to a second speaker of the at least two speakers. As such, multiple speakers can be verified. This is especially effective where a user device is used in a home setting with multiple users, such as a family. Different members of the family be able to control the user device differently.

[034] The method may comprise buffering the audio samples until a predefined number of audio samples has been captured for the identifying step.

[035] The buffered utterance recordings may be saved temporarily in the buffer and readily discarded once the enrolment sample has been calculated.

[036] The predefined number of buffered utterance recordings may be selected in various different ways. For example, the predefined number of buffered utterance recordings may be selected to provide a target balance between FA and FR results.

[037] The method may comprise discarding the audio samples in response to calculating the embedding vector. This frees up on-device storage.

[038] The method may comprise capturing further audio samples and using these to update or fine-tune the calculated embedding vector. Thus, the calculated embedding vector for a specific speaker may thereby be updated over time as more audio samples of the specific speaker become available. This can provide more accurate speaker verification.

[039] In a second approach of the present techniques, there is provided a computer-implemented method for using, on a user device, a speaker verification model, the method comprising: disabling speaker verification before the enrolment data has been calculated using the method for obtaining enrolment data described herein; and enabling speaker verification after the enrolment data has been calculated using the method for obtaining enrolment data described herein. Thus, speaker verification may be enabled once there is sufficient confidence that the obtained enrolment data is highly representative of the specific speaker.

[040] Enabling speaker verification may comprise: enabling speaker verification in response to input from the specific speaker indicating that speaker verification is to be enabled. This can enhance useability in that the specific speaker can be notified that speaker verification enrolment has been completed and that speaker verification can be enabled. The specific speaker can authorise such enabling when they are ready do so.

[041] The method may comprise: capturing an audio sample of an utterance spoken by a speaker; and using a speaker verification model to obtain a speaker verification result, by: generating an embedding vector for the audio sample; determining a distance, in embedding space, between the calculated embedding vector of the enrolment data for the specific speaker and the generated embedding vector for the audio sample; and determining, using the distance, whether the audio sample corresponds to the specific speaker. Preferably, the speaker verification model may determine that the audio sample corresponds to the specific speaker when the distance is less than a threshold distance. As such, speaker verification may be performed reliably. By using a threshold, new audio samples of utterances can readily be classified as being from to a device user, or not being from the device user.

[042] When enrolment data exists for a plurality of specific speakers: the step of determining a distance may comprise determining a distance between the calculated embedding vector of the enrolment data for the plurality of specific speakers and the generated embedding vector for the audio sample; and the step of determining, using the distance, whether the audio sample corresponds to the specific speaker may comprise determining which one specific speaker of the plurality of specific speakers the audio sample corresponds to. Thus, where enrolment data exists of multiple individual speakers, as may be the case where there are multiple users of the same user device, the method involves determining which of the calculated embedding vectors for the multiple speakers the generated embedding vector is closest to.

[043] The method may comprise: controlling operation of the user device based on the speaker verification result. This can reduce physical interaction between the specific speaker and the user device.

[044] The controlling may comprise: unlocking the user device in response to the speaker verification result classifying the captured audio sample as being from the specific speaker. As such, physical interaction of a user with the user device to unlock the user device may be reduced.

[045] By using the same speaker verification model to generate embeddings of the audio samples during the ‘enrolment’ phase and the embedding of each new audio sample during the ‘use’ phase, the embeddings may be readily compared to each other, for example in terms of their distance apart in an embedding space, to identify similarity.

[046] The method may comprise: using a triplet loss to personalise the speaker verification model to: minimise distances between the calculated embedding vector and embeddings of audio samples of utterances from the specific speaker; and maximise distances between the enrolment sample and embeddings of utterance recordings of utterances that are not from the specific speaker. The triplet loss may be employed to force pseudo-positive utterances to be as close as possible to a cluster centroid, and negative samples as far as possible from the cluster centroid.

[047] On-device personalisation of the speaker verification model may thereby be provided.

[048] In a third approach of the present techniques, there is provided a user device for obtaining enrolment data for use by a speaker verification model, the user device comprising: a microphone arranged to capture a plurality of audio samples of utterances spoken by at least one speaker; and at least one processor coupled to a memory, arranged for: generating an embedding vector for at least some of the plurality of audio samples; identifying a subset of the embedding vectors which correspond to a specific speaker of the at least one speaker; and calculating an embedding vector for the specific speaker using the identified subset of embedding vectors, the embedding vector representing a voice of the specific speaker and forming enrolment data for the speaker verification model.

[049] The features described above with respect to the first approach apply equally to the third approach and therefore, for the sake of conciseness, are not repeated.

[050] In a fourth approach of the present techniques, there is provided a user device for using, on the user device, a speaker verification model, the user device comprising: a microphone arranged to capture audio samples of utterances spoken by at least one speaker; and at least one processor coupled to a memory, arranged for: disabling speaker verification before the enrolment data has been calculated using the method for obtaining enrolment data described herein; and enabling speaker verification after the enrolment data has been calculated using the method for obtaining enrolment data described herein. Thus, speaker verification may be enabled once there is sufficient confidence that the obtained enrolment data is highly representative of the specific speaker.

[051] The features described above with respect to the second approach apply equally to the fourth approach and therefore, for the sake of conciseness, are not repeated.

[052] It will be understood that the same user device is used for the third and fourth approaches.

[053] The user device may be a constrained-resource device, but which has the minimum hardware capabilities to personalise and use a trained neural network / ML model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge or vacuum cleaner). It will be understood that this is a non-exhaustive and non-limiting list of example user devices.

[054] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[055] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[056] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[057] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[058] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[059] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[060] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[061] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[062] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[063] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[064] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[065] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[066] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[067] Figure 1 is a schematic diagram showing an existing technique for gathering data needed to enable a speaker verification model to be used on a user device;

[068] Figure 2 is a schematic diagram showing the present techniques for gathering data needed to enable a speaker verification model to be used on a user device;

[069] Figure 3 is a flowchart of example steps performed by the automatic enrolment sampling component of the present techniques, during a process to obtain enrolment data (left) and during a process after enrolment data has been obtained (right);

[070] Figure 4 is a schematic diagram illustrating how the automatic enrolment sampling component identifies a subset of the embedding vectors that correspond to a specific speaker;

[071] Figure 5 is a schematic diagram illustrating the OOV detection component of the present techniques;

[072] Figure 6 shows two techniques for detecting OOV samples;

[073] Figure 7 is a block diagram showing the experiments performed to test the automatic enrolment sampling component;

[074] Figure 8 is a table of results from experiments to “train” DBSCAN;

[075] Figure 9 is a table of results from an experiment in which positive utterances are mixed with negative utterances to “train” DBSCAN;

[076] Figure 10 is a block diagram showing the experiments performed to test the combination of the OOV component and the automatic enrolment sampling component;

[077] Figure 11 is a table of results from an experiment using positive utterances mixed with OOV utterances to “train” DBSCAN, without using an OOV classifier;

[078] Figure 12 is a table of results from an experiment using positive utterances mixed with OOV utterances to “train” DBSCAN, using an OOV classifier to filter OOV utterances;

[079] Figure 13 is a flowchart of example steps of a computer-implemented method, performed on a user device, for obtaining enrolment data for use by a speaker verification model; and

[080] Figure 14 is a flowchart of example steps of a computer-implemented method for using, on a user device, a speaker verification model after enrolment data has been obtained. Detailed description of the drawings

[081] Broadly speaking, embodiments of the present techniques relate to a method for obtaining enrolment data for a speaker verification model. In particular, the present application provides a computer-implemented method, performed on a user device, for obtaining enrolment data for use by a speaker verification model, without requiring a user to actively record audio samples.

[082] Figure 1 is a schematic diagram showing an existing technique for gathering data needed to enable a speaker verification model to be used on a user device. As shown, a user of the user device is typically required to actively record several (e.g. five) audio samples of them speaking. As mentioned above, such techniques suffer from needing to ask the user to pre-record a set of “clean” utterances, that are used fortraining personalization. In the existing technique, after the user has recorded the audio samples, the audio samples are all saved on the user device. This may be problematic for resource-constrained - specifically, memory-constrained - devices. The existing techniques then analyse the stored recorded audio samples to determine whether the user has spoken a keyword or key phrase, such as “Hi Bixby”. Such a keyword or phrase may be used to wake-up the user device or cause the user device to become ready to receive an instruction or query from the user (such as “please call Alice” or “what is the weather?”) The stored recorded audio samples in which the user is determined to say the keyword or phrase are used to calculate an embedding vector for the user, which forms the enrolment data for the user that captures features of the user’s voice when saying the specific keyword or phrase. The enrolment data can then be used by the speaker verification model to verify the user in the future, as shown in Figure 1.

[083] Figure 2 is a schematic diagram showing the present techniques for gathering data needed to enable a speaker verification model to be used on a user device. The present techniques do not require any pre-recorded enrolment samples. Instead, advantageously, the present techniques capture audio samples automatically from whenever a user interacts with the user device using their voice. For example, the user may use a keyword or key phrase such as “Hi Bixby” to control their user device or query an Al assistant on the user device. These interactions do not require the speaker verification model to be used, because the interactions do not require the additional security that speaker verification provides. For example, speaker verification may not be needed when the user is simply asking an Al assistant to “call Alice” or “what is the weather?”. These interactions enable the present techniques to collect audio samples via the ordinary interactions between the user and their user device, which overcomes one of the problems with existing techniques. Furthermore, the present techniques select and store certain collected audio samples only, which means only those audio samples that can be used to generate enrolment data for the user are saved on the user device, thereby reducing the memory / storage requirements of the present techniques compared to the existing techniques.

[084] As shown in Figure 2, relative to Figure 1, here the user does not need to record enrolment audio samples before the speaker verification pipeline can be used. Instead, realtime utterances spoken by a user can be used to gather the data needed to enable the speaker verification pipeline. As explained in more detail below, the present techniques also use an out-of-vocabulary, OOV, classifier or detector to filter the real-time utterances which do not contain a predefined keyword or phrase (such as “Hi Bixby”), which means these utterances do not need to be saved on the user device for use in the speaker verification pipeline.

[085] The present techniques comprise two important components. Firstly, as shown in Figure 2, there is an automatic enrolment sampling component. As described in more detail below, this component calculates a centroid of positive audio samples (i.e. those which are to be used to generate enrolment data for the user) and, with it, discards positive samples that are not ideal (e.g. for training personalization).

[086] Secondly, there is an out-of-vocabulary detection component. As described in more detail below, this component identifies audio samples that do not contain the keyword and discards them. This helps the automatic enrolment sampling component to generate a centroid for positive audio samples and thereby generate accurate enrolment data for the user. As noted above, as keyword spotting is used to determine whether a user has spoken a keyword that causes the user device to be ready to receive voice commands, all the audio samples should contain the keyword and no out-of-vocabulary words. However, the speaker verification model is, and is required to be, much more accurate than a model which performs the keyword spotting. Therefore, false positive audio samples (i.e. those which do not contain the keyword) may be passed through to the out-of-vocabulary detection component, but this OOV component is able to more accurately determine whether the audio sample contains the keyword. Any audio samples that reach the OOV component and are determined not to contain the keyword are discarded, such that only positive samples are forwarded to the automatic enrolment sampling component.

[087] Figure 3 is a flowchart of example steps performed by the automatic enrolment sampling component of the present techniques, during a process to obtain enrolment data (left) and during a process after enrolment data has been obtained (right). As noted above, a Speaker Verification model needs enrolment utterances (i.e. audio samples of a speaker’s voice) to perform classification in embedding space (i.e. to determine whether a new audio sample matches enrolment data for a specific speaker). In practice, the speaker verification model measures the distance (in embedding space) between the enrolment audio samples for a specific speaker and a new input audio sample, in order for the model to label the new input audio sample as being a “positive” user sample that has a high probability of belonging to the specific speaker, or a “negative” user sample that has a low probability of belonging to the specific speaker.

[088] The automatic enrolment sampling component of the present techniques generates (i.e. calculates) enrolment data itself in in embedding space, without the need for the user to record “clean” audio samples as reference.

[089] The left side of Figure 3 shows steps performed by the automatic enrolment sampling component of the present techniques during a process to obtain enrolment data. Once a plurality of audio samples of utterances spoken by at least one speaker have been generated, the automatic enrolment sampling component generates an embedding vector for at least some of the captured plurality of audio samples. (As mentioned above, some of the captured audio samples may be discarded before the automatic enrolment sampling component generates enrolment data). The automatic enrolment sampling component: identifies a subset of the embedding vectors which correspond to a specific speaker of the at least one speaker; and calculates an embedding vector for the specific speaker using the identified subset of embedding vectors, the embedding vector representing a voice of the specific speaker and forming enrolment data for the speaker verification ML model.

[090] Figure 4 is a schematic diagram illustrating how the automatic enrolment sampling component identifies a subset of the embedding vectors that correspond to a specific speaker. Utterances / audio samples that are saved in a buffer on the user device are used by the automatic enrolment sampling component to generate enrolment data for the or each specific speaker represented in the audio samples.

[091] The identifying may comprise: clustering, in embedding space, the generated embedding vectors into at least one cluster; and selecting the embedding vectors in one cluster of the at least one cluster as the subset of embedding vectors which correspond to a specific speaker.

[092] In embedding space, the automatic enrolment sampling component may use a clustering algorithm to perform the clustering step. The clustering algorithm may be an unsupervised algorithm. Using an unsupervised clustering algorithm facilitates automatic enrolment. The embeddings in the cluster of embeddings are likely to be highly representative of the specific speaker. The resulting calculated embedding vector may therefore also be highly representative of the specific speaker. This may result in more accurate speaker verification which can, for example, improve security. The clustering algorithm may be any one of: a density-based spatial clustering of applications with noise, DBSCAN; k-means clustering; and / or an Infomap algorithm.

[093] The calculating performed by the automatic enrolment sampling component may comprise: calculating a mean embedding vector using the embedding vectors in the selected cluster. Speaker verification performance may be enhanced by using a “centroid” of the embeddings within a cluster, instead of a centroid of the embeddings of all audio samples. Thus, as shown in Figure 4, a cluster of embeddings obtained from captured audio samples has been identified in embedding space, and it can be seen that there are some other embeddings that appear to not belong to the same user.

[094] Returning to Figure 3, the right side shows a flowchart of example steps performed by the automatic enrolment sampling component during a process after enrolment data has been obtained. Thus, once the centroid has been calculated, the automatic enrolment sampling component can use the centroid as being representative of the specific speaker’s voice, and can compare this centroid with embeddings generated for new audio samples / utterances.

[095] The automatic enrolment sampling component may use a speaker verification ML model to obtain a speaker verification result, by: generating an embedding vector for a new audio sample; determining a distance, in embedding space, between the calculated embedding vector of the enrolment data for the specific speaker and the generated embedding vector for the audio sample; and determining, using the distance, whether the audio sample corresponds to the specific speaker. Preferably, the speaker verification ML model may determine that the audio sample corresponds to the specific speaker when the distance is less than a threshold distance. As such, speaker verification may be performed reliably. By using a threshold, new audio samples of utterances can readily be classified as being from to a device user, or not being from the device user.

[096] The centroid may also be used to personalise the speaker verification ML model, using a triplet loss, to: minimise distances between the calculated embedding vector and embeddings of audio samples of utterances from the specific speaker; and maximise distances between the enrolment sample and embeddings of utterance recordings of utterances that are not from the specific speaker. The triplet loss may be employed to force pseudo-positive utterances to be as close as possible to a cluster centroid, and negative samples as far as possible from the cluster centroid.

[097] Figure 5 is a schematic diagram illustrating the OOV detection component of the present techniques. The Out-Of-Vocabulary (OOV) detection component (also referred to herein as a classifier) is responsible for detecting and discarding utterances containing a keyword that is different from a predefined keyword.

[098] The OOV detection component classifies the plurality of captured audio samples into audio samples that include a predefined keyword, and audio samples that do not include the predefined keyword; wherein the clustering comprises clustering the embedding vectors for audio samples that include the predefined keyword.

[099] OOV may be used to provide enhanced filtering of utterances that wrongly pass a keyword-spotting block. Higher speaker verification performance may be achieved by preventing OOV samples from being used.

[100] Figure 6 shows two techniques for detecting OOV samples. In some cases, as shown on the left side of Figure 6, the classifying may comprise: using a supervised OOV classification ML model, the supervised OOV classification ML model having been trained, using a plurality of training audio samples which are labelled as containing a predefined keyword, to separate the plurality of audio samples into audio samples that include the predefined keyword, and audio samples that do not include the predefined keyword.

[101] Using supervised methods, audio samples containing the predefined keyword can be separated from OOV audio samples. This may be particularly effective where a supervised model can be pre-trained and used on-device.

[102] Using the supervised OOV classification method may comprise: using a linear support vector machine, SVM. Other examples of supervised OOV classification methods include, but are not limited, decision trees and random forests.

[103] In other cases, as shown on the right side of Figure 6, the classifying may comprise: using an unsupervised OOV classification algorithm to identify a cluster of audio samples that do not include the predefined keyword.

[104] Using unsupervised methods enables a cluster of OOV utterances to be found and labelled. This may be particularly effective when running OOV classification in real-time, while a user device is recording new data. Examples of unsupervised methods include, but are not limited to, DBSCAN, Gaussian mixture model (GMM), and k-means clustering.

[105] Some experiments to test the present techniques are now described.

[106] To test the automatic enrolment sampling component of the present techniques, experiments were run using DBSCAN as an unsupervised method to automatically find the enrolment audio samples for a specific speaker from a plurality of audio samples. Figure 7 is a block diagram showing the experiments performed to test the automatic enrolment sampling component. The present techniques were compared to the “baseline”, i.e. the existing techniques which use 5 pre-recorded enrolment samples. Results are compared in terms of False Acceptance Rate (FA) and False Rejection Rate (FR) for speaker verification. Results are shown for different languages, by using a dataset of speakers saying a predefined keyword: “Hi Bixby”.

[107] Averaged across 10 different languages of users saying “Hi Bixby” (keyword), the automatic enrolment sampling component improves the False Rejection Rate (FR) when compared to speaker verification results employing 5 pre-recorded enrolment audio samples. The FR drops more than 4 times, from about 1.2% to about 0.3% (depending on utterance number). However, False Acceptance (FA) slightly increases from about 8% to around 12%. This increase of FA, however, is a good compromise for the large FR improvement (i.e. better security) and the drastic reduction of memory needed for operations such as Speaker Verification personalization.

[108] Figure 8 is a table of results from five different experiments in which the number of utterances / audio samples that are used to “train” DBSCAN (i.e. to find the enrolment sample via DBSCAN) are varied from 5 positive utterances, to 10, 15, 20, and 30. The table in Figure 8 shows that although there is a slight increase in the FA, FR usually improves a lot, with respect to the baseline, which means that although less data is stored in memory, better security is achieved. The greater the number of samples the better the results, with FR very close to 0% (or 0%) for most the languages. It appears that between 10 to 15 positive samples is optimal to have a good balance between the FA and FR results.

[109] Figure 9 is a table of results from an experiment in which positive utterances are mixed with negative utterances to “train” DBSCAN and test its robustness is removing unnecessary samples from memory. The total number of samples is always 30. The experiments tested 10%, 30%, 50%, 70% and 90% outliers (i.e. 10% means 27 positives and 3 negatives in the training set). The table in Figure 9 confirms the trend observed in Figure 8, i.e. that although there is a light increase in the FA, the FR is usually improving a lot, with respect to the baseline. DBSCAN seems to work well even in extreme cases, where the outliers are the 70% of the training set. DBSCAN does less well for the more extreme case of 90% outliers. However, this is understandable as essentially DBSCAN is being asked to find the centroid of 3 positive utterances that are provided together with 27 negative samples. In general, up to 70% outliers FR is (much) better than the baseline, for most of the languages.

[110] To test the OOV detection component of the present techniques, the automatic enrolment sampling component is tested using DBSCAN as unsupervised method to automatically find the enrolment data, while also filtering input samples using a Linear-SVM as the OOV detection component. That is, the testing of the OOV detection component requires testing the performance of the automatic enrolment sampling component when the OOV detection component is used to filter the samples used. Figure 10 is a block diagram showing the experiments performed to test the combination of the OOV component and the automatic enrolment sampling component. To test how the OOV classifier can help the automatic enrolment sampling component, experiments were run with DBSCAN as the unsupervised method to automatically finding the enrolment samples, while filtering input samples using a Linear-SVM as OOV classifier. The present techniques are compared to the baseline, which uses five pre-recorded enrollment samples. Results are compared in terms of False Acceptance Rate (FA) and False Rejection Rate (FR) for speaker verification. Results are shown for different languages, by using a dataset of speakers saying “Hi Bixby”

[111] Averaged across nine different languages of users saying “Hi Bixby” and other Out-Of-Vocabulary keywords, Linear_SVM improves the False Rejection Rate (FR) when compared to the automatic enrolment sampling component alone, in the presence of OOV data. For the extreme case of 70% OOV utterances (where 30% is the “Hi Bixby” data from the positive user), FR drops from about 50% to 0.2%, while FA drops from 61% to 10%. For the very extreme case of 90% OOV utterances (where 10% is the “Hi Bixby” data from the positive user) FR drops from about 89% to 5%, whereas FA from 88% to 13%. This shows that an OOV classifier can dramatically improve the performance of the automatic enrolment sampling component, in the presence of OOV utterances.

[112] Figure 11 is a table of results from an experiment using positive utterances mixed with OOV utterances to “train” DBSCAN. In this experiment, an OOV classifier is not used, in order to determine how the present techniques perform without an OOV classifier. The total number of samples is always 30. The experiments tested 10%, 30%, 50%, 70% and 90% OOVs (i.e. 10% means 27 positives and 3 OOVs in the training set). As shown in the table of Figure 11, the trend observed in Figures 8 and 9 is confirmed: although there is a slight increase in the FA, the FR is usually improving a lot, with respect to the baseline. DBSCAN seems to work fine up to 50% of OOV outliers in the training set. DBSCAN does less well for the more extreme cases of 70% and 90% outliers. In general, up to 50% outliers FR seems to improve almost 7 times with respect to the baseline, for most of the languages.

[113] Figure 12 is a table of results from an experiment using positive utterances mixed with OOV utterances to “train” DBSCAN. In this experiment, an OOV classifier is used, in order to determine how the present techniques perform with an OOV classifier (c.f. Figure 11). The total number of samples is always 30. The experiments tested 10%, 30%, 50%, 70% and 90% OOVs (i.e. 10% means 27 positives and 3 OOVs in the training set). It is clear from Figure 12 that the Linear-SVM used as an OOV classifier works well in identifying and removing from both training and test sets the OOV utterances, in comparison to the results in Figure 11 where an OOV classifier is not used. In fact, results are very good now, for even 70% of OOV outliers in the training set, where, as average FR is more than 2 times better than the baseline. Even in the more extreme case of 90% outliers, the average FR is around 5%, which means that the Linear-SVM is able to remove most of the OOV samples, leaving DBSCAN to find the centroid with about 3 positive samples.

[114] Figure 13 is a flowchart of example steps of a computer-implemented method, performed on a user device, for obtaining enrolment data for use by a speaker verification (e.g. machine learning, ML), model, the method comprising: capturing a plurality of audio samples of utterances spoken by at least one speaker (step S100); generating an embedding vector for at least some of the plurality of audio samples (step S102); identifying a subset of the embedding vectors which correspond to a specific speaker of the at least one speaker (step S104); and calculating an embedding vector for the specific speaker using the identified subset of embedding vectors, the embedding vector representing a voice of the specific speaker and forming enrolment data for the speaker verification model (step S106).

[115] Figure 14 is a flowchart of example steps of a computer-implemented method for using, on a user device, a speaker verification (e.g. machine learning, ML), model. The method comprises: disabling speaker verification before the enrolment data has been calculated using the method for obtaining enrolment data described herein; and enabling speaker verification after the enrolment data has been calculated using the method for obtaining enrolment data described herein. Thus, speaker verification may be enabled once there is sufficient confidence that the obtained enrolment data is highly representative of the specific speaker.

[116] The method comprises: capturing an audio sample of an utterance spoken by a speaker (step S200); and using a speaker verification model to obtain a speaker verification result, by: generating an embedding vector for the audio sample (step S202); determining a distance, in embedding space, between the calculated embedding vector of the enrolment data for the specific speaker and the generated embedding vector for the audio sample (step S204); and determining, using the distance, whether the audio sample corresponds to the specific speaker (step S206). Preferably, the speaker verification model may determine that the audio sample corresponds to the specific speaker when the distance is less than a threshold distance. As such, speaker verification may be performed reliably. By using a threshold, new audio samples of utterances can readily be classified as being from to a device user, or not being from the device user.

[117] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

24 10 241. A computer-implemented method, performed on a user device, for obtaining enrolment data for use by a speaker verification model, the method comprising:capturing a plurality of audio samples of utterances spoken by at least one speaker when the at least one speaker controls the user device with their voice;generating an embedding vector for at least some of the plurality of audio samples;identifying a subset of the embedding vectors which correspond to a specific speaker of the at least one speaker by:clustering, in embedding space, the generated embedding vectors into at least one cluster; andselecting the embedding vectors in one cluster of the at least one cluster as the subset of embedding vectors which correspond to a specific speaker; and calculating an embedding vector for the specific speaker using the identified subset of embedding vectors, the embedding vector representing a voice of the specific speaker and forming enrolment data for the speaker verification model.

2. The method as claimed in claim 1, wherein the clustering comprises using a clustering algorithm, wherein the clustering algorithm is any one of: a density-based spatial clustering of applications with noise, DBSCAN, algorithm; k-means clustering; and an Infomap algorithm.

3. The method as claimed in claim 1 or 2 further comprising:discarding audio samples corresponding to embedding vectors which are not within a cluster.

4. The method as claimed in any one of claims 1 to 3, wherein the calculating comprises: calculating a mean embedding vector using the embedding vectors in the selected cluster.

5. The method as claimed in any one of claims 1 to 4, wherein the identifying comprises: classifying, using an out-of-vocabulary, OOV, classifier, the plurality of audio samples into audio samples that include a predefined keyword, and audio samples that do not include the predefined keyword;wherein the clustering comprises clustering the embedding vectors for audio samples that include the predefined keyword.24 10 246. The method as claimed in claim 5, wherein the classifying comprises:using a supervised OOV classification ML model, the supervised OOV classification ML model having been trained, using a plurality of training audio samples which are labelled as containing a predefined keyword, to separate the plurality of audio samples into audio samples that include the predefined keyword, and audio samples that do not include the predefined keyword.

7. The method as claimed in claim 6, wherein using the supervised OOV classification ML model comprises:using a linear support vector machine, SVM.

8. The method as claimed in claim 7, wherein the classifying comprises:using an unsupervised OOV classification algorithm to identify a cluster of audio samples that do not include the predefined keyword.

9. The method as claimed in any one of claims 5 to 8, further comprising: discarding any audio sample that does not include the predefined keyword.

10. The method as claimed in any one of claims 5 to 9, comprising, before the classifying: detecting, using a keyword detector, the predefined keyword in captured audio samples; andstoring the audio samples in which the predefined keyword is detected for use in the classifying.

11. The method as claimed in any one of claims 1 to 10, wherein capturing a plurality of audio samples comprises:capturing audio samples spoken by at least two speakers.

12. The method as claimed in claim 11 wherein the identifying comprises:clustering, in embedding space, the generated embedding vectors into at least two clusters; andselecting the embedding vectors in a first cluster of the at least two clusters as the subset of embedding vectors which correspond to a first speaker of the at least two speakers, and selecting the embedding vectors in a second cluster of the at least two clusters as the subset of embedding vectors which correspond to a second speaker of the at least two speakers.24 10 2413. The method as claimed in any one of claims 1 to 12 further comprising: buffering the audio samples until a predefined number of audio samples has been captured for the identifying step.

14. The method as claimed in any one of claims 1 to 13 further comprising: discarding the audio samples after calculating the embedding vector.

15. A computer-implemented method for using, on a user device, a speaker verification model, the method comprising:disabling speaker verification before the enrolment data has been calculated using the method of any of claims 1 to 14; andenabling speaker verification after the enrolment data has been calculated using the method of any of claims 1 to 14.

16. The method as claimed in claim 15, wherein enabling speaker verification comprises: enabling speaker verification in response to input from the specific speaker indicating that speaker verification is to be enabled.

17. The method as claimed in any one of claims claim 15 or 16, comprising: capturing an audio sample of an utterance spoken by a speaker; and using a speaker verification model to obtain a speaker verification result, by: generating an embedding vector for the audio sample;determining a distance, in embedding space, between the calculated embedding vector of the enrolment data for the specific speaker and the generated embedding vector for the audio sample; anddetermining, using the distance, whether the audio sample corresponds to the specific speaker.

18. The method as claimed in claim 17 wherein when enrolment data exists for a plurality of specific speakers:determining a distance comprises determining a distance between the calculated embedding vector of the enrolment data for the plurality of specific speakers and the generated embedding vector for the audio sample; anddetermining, using the distance, whether the audio sample corresponds to the specific speaker comprises determining which one specific speaker of the plurality of specific speakersthe audio sample corresponds to.24 10 2419. The method as claimed in claim 17 or 18 wherein the speaker verification model determines that the audio sample corresponds to the specific speaker when the distance is less than a threshold distance.

20. The method as claimed in claim 19, comprising:controlling operation of the user device based on the speaker verification result.

21. The method as claimed in claim 20, wherein the controlling comprises:unlocking the user device in response to the speaker verification result classifying the captured audio sample as being from the specific speaker.

22. The method as claimed in any of claims 15 to 21 further comprising:using a triplet loss to personalise the speaker verification model to:minimise distances between the calculated embedding vector and embeddings of audio samples of utterances from the specific speaker; andmaximise distances between the enrolment sample and embeddings of utterance recordings of utterances that are not from the specific speaker.

23. A user device for obtaining enrolment data for use by a speaker verification model, the device comprising:a microphone arranged to capture a plurality of audio samples of utterances spoken by at least one speaker when the at least one speaker controls the user device with their voice; andat least one processor coupled to a memory, arranged for:generating an embedding vector for at least some of the plurality of audio samples;identifying a subset of the embedding vectors which correspond to a specific speaker of the at least one speaker by:clustering, in embedding space, the generated embedding vectors into at least one cluster; andselecting the embedding vectors in one cluster of the at least one cluster as the subset of embedding vectors which correspond to a specific speaker; and calculating an embedding vector for the specific speaker using the identified subset of embedding vectors, the embedding vector representing a voice of the specificspeaker and forming enrolment data for the speaker verification model.24 10 24

Images