Method and system for personalising speaker verification models

Abstract

A method personalising a trained speaker verification machine learning (ML) model 114 for a specific user, to better distinguish the user’s voice from typical noise in their environment. The speaker’s

Description

Field

[001] The present application generally relates to a method and system for personalising speaker verification models. In particular, the present application provides a method for personalising a trained speaker verification machine learning, ML, model for user-specific noise. Personalisation may take place on-device (i.e. on the end user device which is going to be used to run the personalised ML model) or on a server. Background

[002] Voice authentication systems are increasingly used in banking. For example, unlocking devices (for making payments or otherwise), authorising payments, and other financial activities require high security and therefore, benefit from the accuracy of voice biometrics.

[003] Currently, it is possible to improve speaker verification models (i.e. Artificial Intelligence or machine learning models that verify a speaker via their voice) by personalising the models for individual users. In this way, the models are better able to identify a specific user. However, one of the challenges of on-device deployment of these ASR models arises from the mismatch between training and test conditions. These models suffer from performance degradation due to the speech variability or different acoustic environments (such as additive noise) of the target user.

[004] Personalization of ASR models aims at reducing this performance gap by leveraging limited user data. A variety of techniques have been employed to boost the personalization task, such as contextual biasing, embedding based adaptation or feature transformation adaptation. Separately to personalization, data augmentation (DA) may be used in training of Automatic Speech Recognition (ASR) models. Data augmentation offers increased data variability, robustness and generalization against different noises. Several data augmentation methods have been proposed to improve ASR performance, for example one method optimizes a data augmentation policy by randomly searching over large augmentation space. However, techniques proposed for using personalized data augmentation (DA) in ASR tasks have been scarce.

[005] The applicant has therefore identified the need for an improved method for personalising speaker verification models. Summary

[006] In a first approach of the present techniques, there is provided a computer-implemented method, performed on a user device, for personalising a trained speaker verification machine learning, ML, model for a speaker that uses the user device, the method comprising: obtaining an audio sample that includes a voice of the speaker and a speakerspecific noise; extracting the speaker-specific noise and the voice of the speaker from the audio sample; comparing the extracted speaker-specific noise to noise samples in a plurality of clusters, each cluster containing different types of noise samples, to identify at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise; selecting a set of noise samples from the identified cluster, the set of noise samples comprising at least one noise sample; generating at least one noisy speaker-specific audio sample by adding a noise sample from the selected set of noise samples to the extracted voice of the speaker; and personalising the trained speaker verification ML model using the generated at least one noisy speaker-specific noise audio samples.

[007] The at least one audio sample of the voice of the specific speaker may also be referred to herein as enrolment data or an enrolment sample. The at least one audio sample may be collected once at setup time. For example, the specific speaker may take part in an enrolment process to provide at least one sample of their voice, which can then be used as a reference audio sample(s). This enrolment process may only need to be performed once per user.

[008] Advantageously, the present techniques enable more accurate personalisation of speaker verification models on-device, while also reducing a memory requirement for the personalisation. Specifically, the present techniques isolate and extract noise from speaker utterances / audio samples, and use the extracted noise from at least one speaker’s audio samples to personalise a pre-trained speaker verification model for a specific speaker. In other words, the speaker verification model is personalised for a specific speaker using some augmented speaker-specific audio samples that help the model to learn how to identify that specific user’s voice even in noisy audio samples. The augmented speaker-specific audio samples are audio samples containing the speaker’s voice and noise that has been extracted from noisy audio samples. The noise may be from the speaker’s own noisy audio samples, but preferably may include noise from other speakers’ audio samples. In this way, instead of adding some artificially generated noise to the speaker’s audio samples, the noise is more realistic (e.g. more realistic background noise that is likely to be heard whenever the speaker uses the personalised speaker verification model).

[009] Broadly speaking, the speaker verification model is used to determine whether incoming audio samples contain a specific speaker’s voice (positive sample) or do not contain the specific speaker’s voice (negative sample). As noted above, the present techniques concern personalising a pre-trained speaker verification model. Thus, the speaker verification model has been trained to perform speaker verification, using any suitable training process. The training of the model may have been performed centrally, e.g. on a central server, which stores the (global) trained speaker verification ML model. The central server provides copies of the trained ML model to each user device for use on-device, and for personalisation on-device.

[010] However, because the training process may be performed using audio samples containing artificially-generated noise, or with ‘clean’ audio samples that do not contain any noise, the trained model may not perform well at inference / use time when analysing audio samples that contain real-world / real-life noise. Thus, the personalisation involves taking known positive samples, i.e. those containing a specific speaker’s voice, adding in real noise, and using these noisy samples to personalise the model so it can recognise the speaker’s voice in the presence of real-world noise.

[011] As noted above, generating the augmented audio samples (i.e. audio samples containing noise) involves identifying noise samples that are similar to the noise present within the specific speaker’s noisy audio samples. This is so that the generated augmented noisy audio samples contain the same or similar type(s) of noise that is present in the speaker’s audio samples. This enables the speaker verification model to be trained / personalised using audio samples containing noise that is most likely to be appear in the speaker’s audio samples when they use the personalised model. For example, if a speaker often uses the speaker verification model in an office environment, then it is necessary to be identify the speaker’s voice from all the other sounds that occur in an office, including other voices in the background. Similarly, if a speaker often uses the speaker verification model on a train or while they are outside, it is necessary to identify the speaker’s voice from sounds that occur in such environments. Thus, the present techniques involves finding noise samples that are similar to the speaker-specific noise extracted from the audio samples of the speaker. This involves comparing the speaker-specific noise to noise samples in a plurality of clusters that contain different noise samples.

[012] The step of identifying at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise may comprise: generating an embedding vector for the extracted speaker-specific noise; comparing the generated embedding vector to embedding vectors of each noise sample in the plurality of clusters; and identifying at least one embedding vector of a noise sample in the plurality of clusters that is similar to the generated embedding vector. That is, the present techniques compare the speaker-specific noise to other noise samples in embedding space. This makes it computationally easier to compare the noise and noise samples. An embedding space or latent feature space is a space in which items resembling each other are positioned closer to one another, and further away from items which are different. An embedding vector captures some or all of the features of the corresponding data (i.e. the speaker-specific noise in this case). In embedding space, embedding vectors that encode similar features are located closer together. Embedding vectors convert data (e.g. noise samples) into numbers that capture their meaning or key features.

[013] In one example, the step of comparing the extracted speaker-specific noise to a plurality of clusters may comprise comparing the extracted speaker-specific noise to a plurality of clusters stored on a central server, wherein the noise samples of the plurality of clusters are obtained from other speakers (but may also include the specific speaker’s noise samples). Thus, the present techniques take advantage of the fact this personalisation may be occurring for many users / speakers, as there are now many other examples of noise obtained from other speakers. It is likely that other speakers have the same or similar types of noise in their audio samples, so noise extracted from their audio samples may be used to augment the specific speaker’s audio samples for personalisation of the model.

[014] Additionally or alternatively, the step of comparing the extracted speaker-specific noise to a plurality of clusters may comprise comparing the extracted speaker-specific noise to a plurality of clusters stored on the user device, wherein the noise samples of the plurality of clusters are extracted from audio samples recorded by the speaker that uses the user device. In this scenario, the noise samples in the clusters are obtained from the speaker’s audio samples only. This may preserve user privacy, as no audio samples or noise samples need to leave the user device.

[015] The step of extracting the speaker-specific noise from the audio sample may comprise extracting the speaker-specific noise in embedding space. As noted above, working in embedding space may be computationally more efficient than working with the raw audio signal(s). Specifically, the step of extracting the speaker-specific noise may comprise: generating an embedding vector for the obtained audio sample; retrieving a plurality of audio samples that each include the voice of the speaker; and calculating an embedding vector for the speaker-specific noise by using the generated embedding vector, and a mean and a standard deviation of all embedding vectors generated from the retrieved plurality of audio samples that include the voice of the speaker. An example formula for extracting the noise in embedding space is provided below with reference to the Figures.

[016] Once the noise samples have been selected (via the comparing step described above), the noise samples can be used to generate augmented audio samples containing the voice of the specific speaker and one or more of the selected noise samples. The step of generating at least one noisy speaker-specific audio sample may comprise: retrieving a plurality of audio samples that each include the voice of the speaker; extracting the voice of the speaker from the retrieved plurality of audio samples; and adding at least one noise sample from the selected set of noise samples to each of the extracted voice of the speaker from the obtained audio sample and the retrieved plurality of audio samples. In other words, the speaker’s voice is extracted from audio samples containing their voice to generate speaker audio samples, and at least one of the selected noise samples is added to each speaker audio sample. In some cases, more than one noise sample may be added to a speaker audio sample. In this way, a large number of permutations of noisy audio may be generated. For example, if five audio samples are retrieved, then five speaker audio samples, S1 to S5, may be generated by extracting the speaker’s voice from each retrieved audio sample. If three noise samples, N1 to N3, have been selected, then N1 may be added to each of S1 to S5, N2 may be added to each of S1 to S5, N1 and N3 may both be added to S1, and so on. It will be understood that the number of noisy speaker-specific audio samples generated for the personalisation may depend on how many audio samples and noise samples are obtained, and / or on the minimum number of samples required for the personalisation, and / or on the resources available on the user device for the personalisation (where more samples may require more processing and memory capacity).

[017] The step of adding at least one noise sample to the extracted voice of the speaker may comprise: generating a Mel spectrogram for each extracted voice of the speaker from the obtained audio sample and the retrieved plurality of audio samples; generating a Mel spectrogram for the at least one noise sample; and adding the Mel spectrogram from the at least one noise sample to the Mel spectrogram for each extracted voice of the speaker from the obtained audio sample and retrieved plurality of audio samples. A Mel spectrogram is a spectrogram with the Mel scale as the y axis instead of frequency (and time along the x axis). In a Mel spectrogram, the frequency bands are equally spaced on the Mel scale, which approximates the human auditory system’s response more closely than linearly-spaced frequency bands in a normal spectrum.

[018] The audio samples being retrieved may be obtained in two different ways.

[019] One way may comprise: retrieving audio samples which have been collected during an enrolment period in which the speaker records audio samples of their voice. The plurality of audio samples may also be referred to herein as enrolment data. The plurality of audio samples may be collected once at setup time. For example, the specific speaker may take part in an enrolment process to provide at least one sample of their voice, which can then be used as the audio sample(s). This enrolment process may only need to be performed once per speaker. The audio sample(s) is preferably a sample that only contains the user’s voice and no background noise.

[020] Additionally or alternatively, retrieving a plurality of audio samples may comprise: retrieving audio samples which have been collected when the speaker uses their voice to control the user device. This may be useful because the speaker does not necessarily need to perform an enrolment process (though this could be used in conjunction with the enrolment process). Audio samples may simply be collected during the speaker’s ordinary voice-based interactions with the user device, which allows more samples to be collected and may be more user-friendly.

[021] In addition to or instead of the clustering processes described above, identifying a cluster that is similar to the extracted speaker-specific noise may comprise: determining an environment in which the obtained audio sample was recorded; and identifying a cluster of different types of noise samples by identifying noises samples that are recorded in the same or similar environment to the determined environment. The environment information may be obtained based on when the audio sample is recorded and / or location or GPS data. Knowing when the audio sample is recorded may enable the clustering to be performed because samples recorded during commuting hours are likely to contain similar background sounds (of traffic, for example), and samples recorded between 7pm and 10pm are likely to contain sounds from the speakers’ homes. The location or GPS data may similarly be used to work out that audio samples recorded at train stations are likely to contain similar background noise, and samples recorded inside office buildings are likely to contain similar background noise. (The location or GPS data may need to be combined with map data to determine this).

[022] The environment location needs to be accurate enough to determine the environment where the recording has happened (e.g. public places, offices, homes, etc.). For this, GPS data is likely to be sufficiently accurate. It may also be possible to estimate the exact room (e.g. living room, kitchen, etc) within an environment. For this finer location estimation, triangulation of devices with respect to other devices in a network comprising a Wi-Fi router, may be used.

[023] The trained speaker verification ML model may comprise a pre-trained auto-encoder, the auto-encoder comprising an encoder and a decoder, and wherein: extracting the userspecific noise from the audio sample may comprise using the encoder of the pre-trained autoencoder.

[024] The step of personalising the trained speaker verification ML model may comprise: freezing the encoder of the pre-trained auto-encoder; and training the decoder of the pretrained auto-encoder, using the generated at least one noisy user-specific audio sample, to recognise the voice of the speaker.

[025] The method may further comprise: transmitting the extracted speaker-specific noise to a central server storing a plurality of clusters containing noise samples obtained from other speakers, and wherein identifying at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise comprises obtaining noise samples from the clusters stored by the server. Thus, as noted above, comparing the speaker-specific noise may comprise comparing the speaker-specific noise to noise samples stored by a central server. Thus, in such cases, the extracted speaker-specific noise (or embedding vector(s) thereof) may be transmitted to the central server.

[026] In a second approach of the present techniques, there is provided a user device for personalising a trained speaker verification machine learning, ML, model for a speaker that uses the user device, the user device comprising: at least one processor coupled to memory, for: obtaining an audio sample that includes a voice of the speaker and a speaker-specific noise; extracting the speaker-specific noise and the voice of the speaker from the audio sample; comparing the extracted speaker-specific noise to noise samples in a plurality of clusters, each cluster containing different types of noise samples, to identify at least one cluster that contains noise samples that are most similar to the extracted speaker-specific noise; selecting a set of noise samples from the identified cluster, the set of noise samples comprising at least one noise sample; generating at least one noisy speaker-specific audio sample by adding a noise sample from the selected set of noise samples to the extracted voice of the speaker; and personalising the trained speaker verification ML model using the generated at least one noisy speaker-specific noise audio samples.

[027] The features described above with respect to the first approach apply equally to the second approach and therefore, for the sake of conciseness, are not repeated.

[028] The user device comprises at least one processor and memory. The memory may store instructions that, when executed by the at least one processor individually or collectively, cause the at least one processor to perform the above-described steps.

[029] The user device may be a smart device. The user device may be a smartphone. A smartphone is an example of a smart device. The user device may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV), a smart fridge, a smart oven, a smart vacuum cleaner, a smart robotic device, and so on. More generally, the user device may be a constrained-resource device, but which has the minimum hardware capabilities to personalise a trained ML model, and to use the personalised ML model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart lawn mower, smart oven, etc). It will be understood that this is a non-exhaustive and non-limiting list of example devices.

[030] As described with respect to the first approach, personalising the speaker verification model requires generating some noisy speaker-specific audio samples. Prior to performing the personalisation, a dataset comprising speaker-specific noise samples may be generated to enable the noisy speaker-specific noise samples to be generated on-device. The process to generate, on-device, the dataset of speaker-specific noise is now described. No audio samples need to leave the device, which helps preserve user privacy and security.

[031] In a third approach of the present techniques, there is provided a computer-implemented method for generating, on a user device, a training dataset for personalising a trained speaker verification machine learning, ML, model for a speaker that uses the user device, the method comprising: obtaining a plurality of audio samples that include a voice of 8 the speaker and speaker-specific noise; extracting, using the trained speaker verification ML model, the speaker-specific noise in each of the obtained audio samples to generate speakerspecific noise samples; clustering the extracted speaker-specific noise samples into one or more clusters; and generating a training dataset comprising the one or more clusters of speaker-specific noise samples. Thus, the user device now has noise samples that are clustered, and which can be used to generate the noisy speaker-specific audio samples required for the personalisation.

[032] For each audio sample, extracting the speaker-specific noise may comprise: generating an embedding vector for the obtained audio sample; and calculating an embedding vector for the speaker-specific noise by using the generated embedding vector, and a mean and a standard deviation of all embedding vectors generated from audio samples that include the voice of the speaker.

[033] Clustering the extracted speaker-specific noise samples into one or more clusters may comprise using a clustering algorithm.

[034] Using a clustering algorithm may comprise using any of: an unsupervised clustering algorithm, a data clustering algorithm, a k-means clustering algorithm, a graph-based clustering algorithm, a deep clustering algorithm, and a density-based spatial clustering of applications with noise, DBSCAN, algorithm. It will be understood that these are non-limiting and non-exhaustive examples of clustering techniques.

[035] Clustering the extracted speaker-specific noise samples into one or more clusters may comprise: comparing the extracted speaker-specific noise samples to a plurality of existing clusters, stored on a central server, wherein each cluster comprises noise samples obtained from other speakers; identifying, for each speaker-specific noise sample, a cluster that contains noise samples that are similar to the extracted speaker-specific noise sample; and adding each extracted speaker-specific noise sample into the identified existing cluster.

[036] Additionally or alternatively, clustering the extracted speaker-specific noise samples may further comprise: determining an environment in which each obtained audio sample corresponding to the extracted speaker-specific noise samples was recorded; and clustering the extracted speaker-specific noise samples into at least one cluster based on the determined environment for each audio sample.

[037] The method may further comprise: transmitting the extracted user-specific noise to a central server storing a plurality of existing clusters containing noise samples obtained from other speakers.

[038] Obtaining a plurality of audio samples that include a voice of the speaker and speakerspecific noise may comprise: retrieving audio samples which have been collected during an enrolment period in which the speaker records audio samples of their voice.

[039] Obtaining a plurality of audio samples that include a voice of the speaker and speakerspecific noise may comprise: retrieving audio samples which have been collected when the speaker uses their voice to control the user device.

[040] In a fourth approach of the present techniques, there is provided a user device for generating a training dataset for personalising a trained speaker verification machine learning, ML, model for a speaker that uses the user device, the user device comprising: at least one processor coupled to memory, for performing the method described herein to generate a training dataset (e.g. as described above with respect to the third approach).

[041] The features described above with respect to the third approach apply equally to the fourth approach and therefore, for the sake of conciseness, are not repeated.

[042] The user device comprises at least one processor and memory. The memory may store instructions that, when executed by the at least one processor individually or collectively, cause the at least one processor to perform the above-described steps.

[043] The user device may be a smart device. The user device may be a smartphone. A smartphone is an example of a smart device. The user device may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV), a smart fridge, a smart oven, a smart vacuum cleaner, a smart robotic device, and so on. More generally, the user device may be a constrained-resource device, but which has the minimum hardware capabilities to personalise a trained ML model, and to use the personalised ML model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart lawn mower, smart oven, etc). It will be understood that this is a non-exhaustive and non-limiting list of example devices.

[044] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[045] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[046] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[047] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[048] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[049] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[050] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[051] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[052] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[053] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[054] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[055] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[056] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[057] Figure 1 is a schematic diagram showing how speaker verification, SV, models typically work;

[058] Figure 2 is a schematic diagram showing how the present techniques enable personalisation of a trained speaker verification model for a speaker;

[059] Figure 3 is a flowchart of example steps performed by a speaker verification model;

[060] Figures 4A and 4B are diagrams showing the spread of a data distribution in feature space (Mel-spectrograms);

[061] Figure 5 is a flowchart of example steps to perform personalisation of a speaker verification model;

[062] Figure 6 is a block diagram showing how the noisy speaker-specific audio sample(s) is generated;

[063] Figure 7 is a flowchart of example steps for generating a dataset of noise samples for use during the personalisation of a speaker verification model;

[064] Figure 8 is a block diagram showing how clustering of speaker-specific noise residuals / embeddings may be performed;

[065] Figure 9 is a block diagram showing the pipeline of the present techniques;

[066] Figure 10 is a block diagram of a system 3000 for personalising a trained speaker verification machine learning, ML, model for a speaker. Detailed description of the drawings

[067] Broadly speaking, the present techniques generally relate to a computer-implemented method for personalising general-purpose models using personalised data augmentation. The model may be a speaker verification (SV) model, or another model that is trained to recognise or act on speech data in some way. Advantageously, the present techniques allow for the creation of personalised noisy training data which helps the SV model better identify a particular user even when their input data (e.g. speech) is noisy.

[068] Personalisation aims to “personalise” the ML model to a specific user. In a sense, this could be seen as overfitting the model to enhance the performance for that user. The present techniques may enable on-device personalisation and may be used with existing speaker verification models.

[069] Although it is quite straight forward to collect voice samples (e.g. via users interacting with their devices using their voices), meaningful noisy samples are harder to collect. This is because it is necessary, for each user, to obtain (environmental or other) noises that are quite similar, for robust personalisation of the model for the user. The present techniques provide a method for addressing this problem.

[070] Figure 1 is a schematic diagram showing how speaker verification, SV, models typically work. An SV model 104 is trained, during a training stage, using training data 102. This training data may be generic speaker verification training data, i.e. comprise voice samples of different speakers that have to be identified. Additionally, or alternatively, the SV model may be trained or fine-tuned using user specific voice samples to obtain an SV model that is personalised to recognise a specific user’s voice. Such personalisation may happen on a server or directly on a user device. Thus, while the SV model may be trained on a server, it may also be trained or fine-tuned on a user device.

[071] As noted above, the training data 102 may comprise clean audio samples containing no noise, and / or may contain audio samples that have had artificially-generated noise added to them. Not shown in Figure 1 is that personalisation of the model may occur so that the model can recognise one or more specific speakers.

[072] The trained model 114 is then used, during inference, to process real-life data 112, i.e. real-life audio samples containing the voice of a speaker. The trained model needs to classify each audio sample as either containing the voice of a known speaker or not containing the voice of the known speaker. After training, the SV model is capable of differentiating between different users and verifying a user’s identity. Additionally, the SV model may have been personalised (e.g. by fine tuning) to recognise a specific user’s voice. This may happen during an enrolment procedure in which the user is asked to, for example, repeat specific phrases that allow the SV model to identify the user, for example, by finding and storing a user specific voice profile. A user specific voice profile may, for example, be an embedding of a user’s speech sample or speech samples. An embedding is a latent or feature space representation of the characteristics of that user’s voice. Usually, embeddings are used when the SV model is a transformer, also referred to herein as encoder-decoder model. Such models extract salient features from a user’s voice sample and represent these features in a feature / latent space. This has the advantage that it is possible to extract such features and create a speaker specific voice profile. Additionally, latent space representations / embeddings provide a compact way of storing a speaker profile.

[073] However, because of the above-mentioned issues in the training data, the trained SV model may not be able to accurately classify each audio sample if they contain real-life noise. That is, at inference / use time, when there is noise present in any speech samples of a user, the noise may alter the speaker’s voice profile. For example, when there is noise that is persistently present in a user’s sound recordings, this noise should not be part of the user’s speech profile / embedding but may be included. For example, when a user’s home is next to a busy road, their speech may always include road noises. In another example, a user may often use a voice assistant device when cooking and thus cooking noises may always be heard in the background. It will be appreciated that there are many more examples, and the noise could also only occasionally be present in a user’s speech samples.

[074] A goal of the present techniques is to isolate noise from a user’s speech samples, so that the pre-trained model can learn to distinguish the user’s voice regardless of what noise may appear in their audio samples.

[075] Figure 2 is a schematic diagram showing how the present techniques enable personalisation of a trained speaker verification model for a speaker. (Note, the term “user” is used interchangeably herein with the term “speaker”). It can be seen that the present techniques do not involve altering the training stage. That is, the steps to train the SV model 104 are the same as in Figure 1. Similarly, the inference process is the same as in Figure 1. However, it can be seen that the present techniques provide an additional personalisation stage that enables the trained SV model 114 to be personalised to better recognise a specific speaker’s voice in the presence of real-life noise. Broadly speaking, and as will be explained in more detail below, the present techniques augment audio data containing a specific speaker’s voice with speaker-specific noise. This personalised noisy training data can then be used to improve a speaker verification, or other voice recognition, model. It will be understood that whenever speaker verification is mentioned in this application, this may refer to any type of voice recognition model.

[076] That is, the present techniques aim to extract or isolate a noise residual 106 that represents noise in a specific speaker’s speech samples. Noise residual may refer to the embedding, i.e. feature / latent space representation of noise in a user’s speech recordings. Noise residual may also refer to the actual noise heard in an audio recording of a user’s speech. Noise may comprise environmental noises, such as the ones mentioned above, i.e. any sound present in a user’s sound recording that is not speech. Noise may also comprise background speech from other people that are not the user whose voice is being recognised / verified.

[077] Such a noise residual is then used to find similar noises in a database of noises 108 that have been collected during, for example, training of the SV model from speech samples.

[078] To personalise the trained speaker verification model 114, noise samples 108 are added to speaker audio samples 112 during an augmentation procedure 110. Thus, training samples with user personalised noise are created that can then be used to personalise the trained SV model 114.

[079] Some advantages of the present techniques include: • Robust SV performance even for noise-corrupted utterances; • No need to fine-tune SV personalization for multiple types of noise; • Use of shared noise clusters will help provide a much improved dataset for training and model performance; • Able to find the most meaningful augmentation data for each user, based on both geographic location and past environments; • Works for on-device personalization of SV models.

[080] Figure 3 is a flowchart of example steps performed by a speaker verification model. Typically, a Speaker Verification (SV) model performs classification in the embedding space. At inference / use time, the trained SV model measures the distance (in embedding space) between embedding vectors 208 representing ‘clean’ audio samples 102 (“enrolments” or reference) containing a speaker’s voice, and an embedding vector 212 generated for a new audio sample 204 containing speech (“utterance”). The trained SV model compares the input embedding vector 212 with the reference embedding vectors 208 that are known to contain a specific speaker’s voice, to determine whether the input contains the same specific speaker’s voice. If there are multiple reference embedding vectors 208, then a centroid 210 value may be used for the subsequent comparison. The embedding vector 208 / centroid 210 is compared with the input embedding vector 212 to determine a distance 214 between them in embedding space. The closer the embedding vectors, the more similar the corresponding audio samples are, and vice versa. If the distance 214 is less than a threshold distance 216, then the input embedding vector 212 is considered to be similar to the reference embedding vector / centroid, and therefore, is likely to contain the specific speaker’s voice. The SV model then outputs a classification 218. If the new audio sample contains the specific speaker’s voice, then it is classified as “positive”; similarly, if the new audio sample does not contain the specific speaker’s voice, then it is classified as “negative”.

[081] However, as noted above, when the model is trained on predominantly clean data, or if the enrollments are clean, it will struggle to perform well when presented with noisy data, and vice versa. This is because the training data and test data will have different distributions in the feature space and consequently, in the embedding space.

[082] Figures 4A and 4B are diagrams showing the spread of a data distribution in feature space (Mel-spectrograms). Figure 4A shows clusters of data from distribution 1 (red) and distribution 2 (blue). The overlap between the data is scarce since the data come from different distributions. For example, distribution 1 (red) may contain audio samples that are obtained / recorded in noisy environments, such as the audio samples that may be obtained when a speaker uses a trained speaker verification model. Distribution 2 (blue) may contain audio samples that are obtained / recorded in an acoustically-controlled space, e.g., a quiet environment with no or minimal noise, such as the audio samples that a speaker records during an enrolment phase. This means that if the SV model is trained using distribution 2, it won’t perform well on data from distribution 1. Figure 4B shows the same distributions after adding noise to the data in distribution 2 (blue), as per the present techniques. It can be seen that the distributions overlap more once the noise has been added, which means a speaker verification model that is personalised using the present techniques performs better when processing, during inference, noisy audio samples.

[083] Figure 5 is a flowchart of example steps to perform personalisation of a speaker verification model. The method shown by the flowchart is computer-implemented and performed on a user device. The method is for personalising a trained speaker verification machine learning, ML, model (e.g. model 114 in Figure 2) for a speaker that uses the user device. The method comprises: obtaining at least one audio sample that includes a voice of the speaker and a speaker-specific noise (step S100).

[084] The method comprises extracting the speaker-specific noise and the voice of the speaker from the audio sample (step S102). The step of extracting the speaker-specific noise from the audio sample may comprise extracting the speaker-specific noise in embedding space. As noted above, working in embedding space may be computationally more efficient than working with the raw audio signal(s). Specifically, the step of extracting the speakerspecific noise may comprise: generating an embedding vector for the obtained audio sample; retrieving a plurality of audio samples that each include the voice of the speaker; and calculating an embedding vector for the speaker-specific noise by using the generated embedding vector, and a mean and a standard deviation of all embedding vectors generated from the retrieved plurality of audio samples that include the voice of the speaker. An example formula for extracting the noise in embedding space is provided below.

[085] A noise residual, or user-specific noise, may be extracted using the following formula: c _ p „ _ ^Curr ^Mean £ Noise p cStd where ENoise is the computed embedding of the noise residual, i.e. of the user-specific noise, ECurr is the embedding of the current utterance of the user, EMean is the mean of all available embeddings of the user, also referred to herein as calibration embeddings, Estd is the standard deviation of all available embeddings of the user, i.e. of all calibration embeddings.

[086] That is, isolating a user-specific noise, i.e. noise residual, from a noisy embedding corresponding to the obtained audio sample, may first comprise finding a calibration embedding of the at least one calibration audio sample. That is, the calibration audio samples, i.e. all available audio samples of the user (not including the current noisy audio sample from which are noise embedding is to be extracted) are all transformed into latent space to obtain embeddings of all calibration samples. Thus, the present personalisation process may only begin when a threshold number of noisy audio samples (forming the calibration audio samples) have been obtained.

[087] To then isolate a noise embedding, a mean EMean and standard deviation Estd of the at least one calibration embedding are calculated. The mean of the at least one calibration embedding is subtracted from the noisy embedding to obtain a residual embedding. Then, the residual embedding is divided by the standard deviation of the at least one calibration embedding. The process may be repeated for each obtained audio sample, so that multiple noise embeddings may be generated for a single speaker. These are then added to a dynamically growing database of noise embeddings saved on the user device.

[088] The trained speaker verification ML model may comprise a pre-trained auto-encoder, the auto-encoder comprising an encoder and a decoder. Thus, extracting the user-specific noise from the audio sample may comprise using the encoder of the pre-trained auto-encoder.

[089] The method comprises comparing the extracted speaker-specific noise to noise samples in a plurality of clusters, each cluster containing different types of noise samples, to identify at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise (step S104). Figure 8 show clusters of different noise types. The method to generate these clusters is described below.

[090] Figure 6 is a block diagram showing how the noisy speaker-specific audio sample(s) is generated. The speaker’s speech samples are augmented using noise samples. The noise samples which are added to the speech samples may be obtained in two ways. Firstly, the speaker’s training / enrol utterances (clean or less noisy speech samples) can be augmented with their own noise residuals, i.e. noise residuals extracted from the speaker’s own audio samples. Secondly, the speaker’s training / enrol utterances (clean or less noisy speech samples) can be augmented with other users’ noise residuals. These two ways are described in turn below.

[091] Returning to Figure 5, the step (S104) of identifying at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise may comprise: generating an embedding vector for the extracted speaker-specific noise; comparing the generated embedding vector to embedding vectors of each noise sample in the plurality of clusters; and identifying at least one embedding vector of a noise sample in the plurality of clusters that is similar to the generated embedding vector. That is, the present techniques compare the speaker-specific noise to other noise samples in embedding space. This makes it computationally easier to compare the noise and noise samples. An embedding space or latent feature space is a space in which items resembling each other are positioned closer to one another, and further away from items which are different. An embedding vector captures some or all of the features of the corresponding data (i.e. the speaker-specific noise in this case). In embedding space, embedding vectors that encode similar features are located closer together. Embedding vectors convert data (e.g. noise samples) into numbers that capture their meaning or key features.

[092] In one example, the step (S104) of comparing the extracted speaker-specific noise to a plurality of clusters may comprise comparing the extracted speaker-specific noise to a plurality of clusters stored on a central server, wherein the noise samples of the plurality of clusters are obtained from other speakers (but may also include the specific speaker’s noise samples). Thus, the present techniques take advantage of the fact this personalisation may be occurring for many users / speakers, as there are now many other examples of noise obtained from other speakers. It is likely that other speakers have the same or similar types of noise in their audio samples, so noise extracted from their audio samples may be used to augment the specific speaker’s audio samples for personalisation of the model.

[093] In this case, the method may comprise: transmitting the extracted speaker-specific noise to a central server storing a plurality of clusters containing noise samples obtained from other speakers, and wherein identifying at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise comprises obtaining noise samples from the clusters stored by the server. Thus, as noted above, comparing the speaker-specific noise may comprise comparing the speaker-specific noise to noise samples stored by a central server. Thus, in such cases, the extracted speaker-specific noise (or embedding vector(s) thereof) may be transmitted to the central server.

[094] Additionally or alternatively, the step (S104) of comparing the extracted speakerspecific noise to a plurality of clusters may comprise comparing the extracted speaker-specific noise to a plurality of clusters stored on the user device, wherein the noise samples of the plurality of clusters are extracted from audio samples recorded by the speaker that uses the user device. In this scenario, the noise samples in the clusters are obtained from the speaker’s audio samples only. This may preserve user privacy, as no audio samples or noise samples need to leave the user device.

[095] In addition to or instead of the clustering processes described above, identifying (during step S104) a cluster that is similar to the extracted speaker-specific noise may comprise: determining an environment in which the obtained audio sample was recorded; and identifying a cluster of different types of noise samples by identifying noises samples that are recorded in the same or similar environment to the determined environment. The environment information may be obtained based on when the audio sample is recorded and / or location or GPS data. Knowing when the audio sample is recorded may enable the clustering to be performed because samples recorded during commuting hours are likely to contain similar background sounds (of traffic, for example), and samples recorded between 7pm and 10pm are likely to contain sounds from the speakers’ homes. The location or GPS data may similarly be used to work out that audio samples recorded at train stations are likely to contain similar background noise, and samples recorded inside office buildings are likely to contain similar background noise. (The location or GPS data may need to be combined with map data to determine this).

[096] The method comprises selecting a set of noise samples from the identified cluster, the set of noise samples comprising at least one noise sample (step S106). Any suitable technique may be used to select the at least one noise sample. For example, the noise sample that is most similar to the noise embedding may be selected, and / or the centroid of the cluster may be selected, and / or the noise embedding(s) in the cluster that is most dissimilar from the noise embedding may be selected.

[097] The method comprises generating at least one noisy speaker-specific audio sample by adding a noise sample from the selected set of noise samples to the extracted voice of the speaker (step S108). Turning again to Figure 6, as shown the augmenting may comprise summing the Mel-spec features of the speaker’s training / enrol utterance with the noise residual. The SV model can now use the at least one noisy speaker-specific audio sample to learn how to separate the speaker from the augmented noise, which thereby improves the SV model’s classification accuracy.

[098] Returning to Figure 5, the step (S108) of generating at least one noisy speaker-specific audio sample may comprise: retrieving a plurality of audio samples that each include the voice of the speaker; extracting the voice of the speaker from the retrieved plurality of audio samples; and adding at least one noise sample from the selected set of noise samples to each of the extracted voice of the speaker from the obtained audio sample and the retrieved plurality of audio samples. In otherwords, the speaker’s voice is extracted from audio samples containing their voice to generate speaker audio samples, and at least one of the selected noise samples is added to each speaker audio sample. In some cases, more than one noise sample may be added to a speaker audio sample. In this way, a large number of permutations of noisy audio may be generated. For example, if five audio samples are retrieved, then five speaker audio samples, S1 to S5, may be generated by extracting the speaker’s voice from each retrieved audio sample. If three noise samples, N1 to N3, have been selected, then N1 may be added to each of S1 to S5, N2 may be added to each of S1 to S5, N1 and N3 may both be added to S1, and so on. It will be understood that the number of noisy speaker-specific audio samples generated for the personalisation may depend on how many audio samples and noise samples are obtained, and / or on the minimum number of samples required for the personalisation, and / or on the resources available on the user device for the personalisation (where more samples may require more processing and memory capacity).

[099] The step (at S108) of adding at least one noise sample to the extracted voice of the speaker may comprise: generating a Mel spectrogram for each extracted voice of the speaker from the obtained audio sample and the retrieved plurality of audio samples; generating a Mel spectrogram for the at least one noise sample; and adding the Mel spectrogram from the at least one noise sample to the Mel spectrogram for each extracted voice of the speaker from the obtained audio sample and retrieved plurality of audio samples. A Mel spectrogram is a spectrogram with the Mel scale as the y axis instead of frequency (and time along the x axis). In a Mel spectrogram, the frequency bands are equally spaced on the Mel scale, which approximates the human auditory system’s response more closely than linearly-spaced frequency bands in a normal spectrum.

[100] The audio samples being retrieved during step S108 may be obtained in two different ways.

[101] One way may comprise: retrieving audio samples which have been collected during an enrolment period in which the speaker records audio samples of their voice. The plurality of audio samples may also be referred to herein as enrolment data. The plurality of audio samples may be collected once at setup time. For example, the specific speaker may take part in an enrolment process to provide at least one sample of their voice, which can then be used as the audio sample(s). This enrolment process may only need to be performed once per speaker. The audio sample(s) is preferably a sample that only contains the user’s voice and no background noise.

[102] Additionally or alternatively, retrieving (during step S108) a plurality of audio samples may comprise: retrieving audio samples which have been collected when the speaker uses their voice to control the user device. This may be useful because the speaker does not necessarily need to perform an enrolment process (though this could be used in conjunction with the enrolment process). Audio samples may simply be collected during the speaker’s ordinary voice-based interactions with the user device, which allows more samples to be collected and may be more user-friendly.

[103] The method comprises personalising the trained speaker verification ML model using the generated at least one noisy speaker-specific noise audio samples (step S110).

[104] The step (S110) of personalising the trained speaker verification ML model may comprise: freezing the encoder of the pre-trained auto-encoder; and training the decoder of the pre-trained auto-encoder, using the generated at least one noisy user-specific audio sample, to recognise the voice of the speaker.

[105] As described above with respect to Figure 5, personalising the speaker verification model requires generating some noisy speaker-specific audio samples. Prior to performing the personalisation, a dataset comprising speaker-specific noise samples may be generated to enable the noisy speaker-specific noise samples to be generated on-device. The process to generate, on-device, the dataset of speaker-specific noise is now described. No audio samples need to leave the device, which helps preserve user privacy and security.

[106] Figure 7 is a flowchart of example steps for generating a dataset of noise samples for use during the personalisation of a speaker verification model. The method is computer-implemented and performed on a user device. The method comprises obtaining a plurality of audio samples that include a voice of the speaker and speaker-specific noise (step S200).

[107] Obtaining, at step S200, a plurality of audio samples that include a voice of the speaker and speaker-specific noise may comprise: retrieving audio samples which have been collected during an enrolment period in which the speaker records audio samples of their voice.

[108] Obtaining, at step S200, a plurality of audio samples that include a voice of the speaker and speaker-specific noise may comprise: retrieving audio samples which have been collected when the speaker uses their voice to control the user device.

[109] The method comprises extracting, using the trained speaker verification ML model, the speaker-specific noise in each of the obtained audio samples to generate speaker-specific noise samples (step S202). The techniques described above with respect to Figure 5, step 102 may apply equally here.

[110] Thus, for each audio sample, extracting (at step S202) the speaker-specific noise may comprise: generating an embedding vector for the obtained audio sample; and calculating an embedding vector for the speaker-specific noise by using the generated embedding vector, and a mean and a standard deviation of all embedding vectors generated from audio samples that include the voice of the speaker.

[111] The method comprises clustering the extracted speaker-specific noise samples into one or more clusters (step S204). Figure 8 is a block diagram showing how the clustering may be done. In order to organise the noise residuals better for the augmentation process, clustering is performed on the extracted noise residuals. This provides greater flexibility to pick and augment a specific type of noise during incremental learning / personalisation. Each extracted noise residual 402 is input into an algorithm 404 for unsupervised clustering (e.g. DBSCAN, Infomap, K-Means etc.). The algorithm 404 determines whether each noise residual 402 is similar to other noise residual(s) or different from them. The clustering process may take place once a threshold number of noise residuals have been generated. The algorithm 404 places each noise residual 402 into a cluster 406, 408 or 410 that contains other similar noise residuals. If a noise residual 402 is different from the noise residuals in all the existing clusters, a new cluster may be created.

[112] Returning to Figure 7, clustering, at step S204, the extracted speaker-specific noise samples into one or more clusters may comprise using a clustering algorithm.

[113] Using a clustering algorithm may comprise using any of: an unsupervised clustering algorithm, a data clustering algorithm, a k-means clustering algorithm, a graph-based clustering algorithm, a deep clustering algorithm, and a density-based spatial clustering of applications with noise, DBSCAN, algorithm. It will be understood that these are non-limiting and non-exhaustive examples of clustering techniques.

[114] Clustering the extracted speaker-specific noise samples into one or more clusters may comprise: comparing the extracted speaker-specific noise samples to a plurality of existing clusters, stored on a central server, wherein each cluster comprises noise samples obtained from other speakers; identifying, for each speaker-specific noise sample, a cluster that contains noise samples that are similar to the extracted speaker-specific noise sample; and adding each extracted speaker-specific noise sample into the identified existing cluster.

[115] Additionally or alternatively, clustering the extracted speaker-specific noise samples may further comprise: determining an environment in which each obtained audio sample corresponding to the extracted speaker-specific noise samples was recorded; and clustering the extracted speaker-specific noise samples into at least one cluster based on the determined environment for each audio sample.

[116] The method comprises generating a training dataset comprising the one or more clusters of speaker-specific noise samples (step S206). Thus, the user device now has noise samples that are clustered, and which can be used to generate the noisy speaker-specific audio samples required for the personalisation.

[117] The method may further comprise: transmitting the extracted user-specific noise to a central server storing a plurality of existing clusters containing noise samples obtained from other speakers. This enables a central database of noise residuals to be generated, for use to personalise speaker verification models for all users.

[118] Figure 9 is a block diagram showing the pipeline of the present techniques. Specifically, the diagram shows the steps performed by the trained SV model during inference / use, and how this links in with incremental training of the SV model to personalise the model to better separate a specific speaker’s voice from noise.

[119] As shown in Figure 9, during inference, the trained SV model is used to classify incoming audio samples to determine if they contain a specific speaker’s voice. As described above, the SV model separates the noise in the audio samples from the speech / voice. The extracted noise is added to a growing database of noise samples. Specifically, the embeddings corresponding to the extracted noise may be stored in a database of embeddings. (The extracted speech, or the speech embeddings, do not need to be stored). The embeddings may be clustered, as described above.

[120] Once the noise residuals are extracted and sorted into their clusters, they are ready to be picked for augmentation during the next iteration of incremental training / personalisation of the SV model on-device. As shown in Figure 9, the embeddings can be added to clean speech samples of the specific speaker, and used to personalise the SV model. Specifically, to perform this augmentation, the noise embeddings / residuals need to be converted back to their corresponding Mel-spectrogram. In this way, the Mel-spectrogram of next training utterance can be augmented by adding to it the Mel-spectrogram of the picked noise residual.

[121] The training may involve introducing a pre-trained Auto-Encoder (AE) into the incremental learning pipeline. The noise embeddings computed by the SV model can be considered as the encoder part in the AE. The decoder part of the AE computes the Mel-spectrogram from this noise embedding. When incremental learning is performed on the SV model, the embeddings generated by the SV model are different from that of the previous iteration. Therefore the AE should be able to generate the Mel-spectrogram from the new SV embeddings. To achieve this the AE is trained incrementally alongside the SV model -whenever the SV model is updated, the AE is updated as well. When incrementally training the AE, the encoder is frozen, and only the decoder is trained

[122] Figure 10 is a block diagram of a system 3000 for personalising a trained speaker verification machine learning, ML, model for a speaker that uses a user device 2000. The system 3000 comprises a central server 1000. It will be understood that the system 3000 may comprise many user devices 2000 that are all coupled to the same central server 1000, but only one user device 2000 is shown for the sake of simplicity.

[123] The central server 1000 comprises at least one processor 1002 coupled to memory 1004. The central server 1000 trains a speaker verification, SV, machine learning, ML, model using a training dataset 1008, thereby generating a pre-trained SV ML model 1006. The central server may provide the pre-trained SV model 1006 to multiple user devices 2000 for use on-device to perform speaker verification. The central server may also store a database of clustered noise samples 1010 which are obtained from multiple users. That is, when speaker-specific noise is extracted from audio samples on-device (during inference, as shown in Figure 9), the extracted noise may be stored locally on the user device but may also be sent to the central server 1000 for storage. This allows other users to utilise the noise samples when personalising their SV models.

[124] The at least one processor 1002 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The memory 1004 may comprise volatile memory, such as random access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example.

[125] Each user device 2000 comprises at least one processor 2002 coupled to memory 2004 for obtaining, from the central server 1000, and storing the pre-trained speaker verification ML model 1006.

[126] The user device 2000 may be a constrained-resource device, but which has the minimum hardware capabilities to use and personalise a trained neural network / ML model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge). It will be understood that this is a non-exhaustive and non-limiting list of example user devices.

[127] The user device 2000 may comprise at least one interface 2012. For example, the user device 2000 may comprise a microphone or audio capture device 2012 for receiving utterances of the user of the user device 2000. The utterances may be received during an enrolment phase to record the at least one reference audio sample, and / or when the user is controlling an application (e.g. a virtual assistant) on the user device 2000 using their voice.

[128] The at least one processor 2002 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The memory 2004 may comprise volatile memory, such as random access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example.

[129] The at least one processor 2002 may be configured for: obtaining an audio sample that includes a voice of the speaker and a speaker-specific noise; extracting the speaker-specific noise and the voice of the speaker from the audio sample; comparing the extracted speakerspecific noise to noise samples (stored in database 2010 and / or in database 1010) in a plurality of clusters, each cluster containing different types of noise samples, to identify at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise; selecting a set of noise samples from the identified cluster, the set of noise samples comprising at least one noise sample; generating at least one noisy speaker-specific audio sample by adding a noise sample from the selected set of noise samples to the extracted voice of the speaker; and personalising the trained speaker verification ML model using the generated at least one noisy speaker-specific noise audio samples. The process leads to a personalised SV model 2006.

[130] The at least one processor 2002 also generates a training dataset for personalising a trained speaker verification machine learning, ML, model 1006 for a speaker that uses the user device 2000. The at least one processor 2002 is configured for: obtaining a plurality of audio samples that include a voice of the speaker and speaker-specific noise (e.g. via interface 2012); extracting, using the trained speaker verification ML model 1006, the speaker-specific noise in each of the obtained audio samples to generate speaker-specific noise samples; clustering the extracted speaker-specific noise samples into one or more clusters; and generating a training dataset 2010 comprising the one or more clusters of speaker-specific noise samples. Thus, the user device now has noise samples 2010 that are clustered, and which can be used to generate the noisy speaker-specific audio samples required for the personalisation.

[131] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

1. A computer-implemented method, performed on a user device, for personalising a trained speaker verification machine learning, ML, model for a speaker that uses the user device, the method comprising:obtaining an audio sample that includes a voice of the speaker and a speaker-specific noise;extracting the speaker-specific noise and the voice of the speaker from the audio sample;comparing the extracted speaker-specific noise to noise samples in a plurality of clusters, each cluster containing different types of noise samples, to identify at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise;selecting a set of noise samples from the identified cluster, the set of noise samples comprising at least one noise sample;generating at least one augmented speaker-specific audio sample by adding a noise sample from the selected set of noise samples to the extracted voice of the speaker; andpersonalising the trained speaker verification ML model using the generated at least one augmented speaker-specific audio samples.

2. The method as claimed in claim 1 wherein identifying at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise comprises:generating an embedding vector for the extracted speaker-specific noise;comparing the generated embedding vector to embedding vectors of each noise sample in the plurality of clusters; andidentifying at least one embedding vector of a noise sample in the plurality of clusters that is similar to the generated embedding vector.

3. The method as claimed in claim 1 or 2 wherein comparing the extracted speakerspecific noise to a plurality of clusters comprises comparing the extracted speaker-specific noise to a plurality of clusters stored on a central server, wherein the noise samples of the plurality of clusters are obtained from other speakers.

4. The method as claimed in claim 1 or 2 wherein comparing the extracted speakerspecific noise to a plurality of clusters comprises comparing the extracted speaker-specific noise to a plurality of clusters stored on the user device, wherein the noise samples of theplurality of clusters are extracted from audio samples recorded by the speaker that uses the user device.

5. The method as claimed in any preceding claim wherein extracting the speaker-specific noise from the audio sample comprises extracting the speaker-specific noise in embedding space, by:generating an embedding vector for the obtained audio sample;retrieving a plurality of audio samples that each include the voice of the speaker; and calculating an embedding vector for the speaker-specific noise by using the generated embedding vector, and a mean and a standard deviation of all embedding vectors generated from the retrieved plurality of audio samples that include the voice of the speaker.

6. The method as claimed in any preceding claim wherein generating at least one augmented speaker-specific audio sample comprises:retrieving a plurality of audio samples that each include the voice of the speaker;extracting the voice of the speaker from the retrieved plurality of audio samples; andadding at least one noise sample from the selected set of noise samples to each of the extracted voice of the speaker from the obtained audio sample and the retrieved plurality of audio samples.

7. The method as claimed in claim 6 wherein adding at least one noise sample comprises: generating a Mel spectrogram for each extracted voice of the speaker from the obtained audio sample and the retrieved plurality of audio samples;generating a Mel spectrogram for the at least one noise sample; andadding the Mel spectrogram from the at least one noise sample to the Mel spectrogram for each extracted voice of the speaker from the obtained audio sample and retrieved plurality of audio samples.

8. The method as claimed in claim 5, 6 or 7 wherein retrieving a plurality of audio samples comprises:retrieving audio samples which have been collected during an enrolment period in which the speaker records audio samples of their voice.

9. The method as claimed in claim 5, 6, 7 or 8 wherein retrieving a plurality of audio samples comprises:retrieving audio samples which have been collected when the speaker uses their voice to control the user device.

10. The method as claimed in any preceding claim wherein identifying a cluster that is similar to the extracted speaker-specific noise comprises:determining an environment in which the obtained audio sample was recorded; andidentifying a cluster of different types of noise samples by identifying noises samples that are recorded in the same or similar environment to the determined environment.

11. The method as claimed in any preceding claim wherein the trained speaker verification ML model comprises a pre-trained auto-encoder, the auto-encoder comprising an encoder and a decoder, and wherein:extracting the speaker-specific noise from the audio sample comprises using the encoder of the pre-trained auto-encoder.

12. The method as claimed in claim 11 wherein personalising the trained speaker verification ML model comprises:freezing the encoder of the pre-trained auto-encoder; andtraining the decoder of the pre-trained auto-encoder, using the generated at least one augmented speaker-specific audio sample, to recognise the voice of the speaker.

13. The method as claimed in any preceding claim further comprising:transmitting the extracted speaker-specific noise to a central server storing a plurality of clusters containing noise samples obtained from other speakers, andwherein identifying at least one cluster that contains noise samples that are similar to the extracted speaker-specific noise comprises obtaining noise samples from the clusters stored by the server.

14. A user device for personalising a trained speaker verification machine learning, ML, model for a speaker that uses the user device, the user device comprising:at least one processor coupled to memory, for:obtaining an audio sample that includes a voice of the speaker and a speakerspecific noise;extracting the speaker-specific noise and the voice of the speaker from the audio sample;comparing the extracted speaker-specific noise to noise samples in a plurality of clusters, each cluster containing different types of noise samples, to identify at least one cluster that contains noise samples that are most similar to the extracted speakerspecific noise;selecting a set of noise samples from the identified cluster, the set of noise samples comprising at least one noise sample;generating at least one augmented speaker-specific audio sample by adding a noise sample from the selected set of noise samples to the extracted voice of the speaker; andpersonalising the trained speaker verification ML model using the generated at least one augmented speaker-specific audio samples.

15. A computer-implemented method for generating, on a user device, a training dataset for personalising a trained speaker verification machine learning, ML, model for a speaker that uses the user device, the method comprising:obtaining a plurality of audio samples that include a voice of the speaker and speakerspecific noise;extracting, using the trained speaker verification ML model, the speaker-specific noise in each of the obtained audio samples to generate speaker-specific noise samples;clustering the extracted speaker-specific noise samples into one or more clusters; and generating a training dataset comprising the one or more clusters of speaker-specific noise samples.

16. The method as claimed in claim 15 wherein, for each audio sample, extracting the speaker-specific noise comprises:generating an embedding vector for the obtained audio sample; andcalculating an embedding vector for the speaker-specific noise by using the generated embedding vector, and a mean and a standard deviation of all embedding vectors generated from audio samples that include the voice of the speaker.

17. The method as claimed in any of claims 15 or 16 wherein clustering the extracted speaker-specific noise samples into one or more clusters comprises using a clustering algorithm.

18. The method as claimed in claim 17 wherein using a clustering algorithm comprises using any of: an unsupervised clustering algorithm, a data clustering algorithm, a k-meansclustering algorithm, a graph-based clustering algorithm, a deep clustering algorithm, and a density-based spatial clustering of applications with noise, DBSCAN, algorithm.

19. The method as claimed in any of claims 15 or 16 wherein clustering the extracted speaker-specific noise samples into one or more clusters comprises:comparing the extracted speaker-specific noise samples to a plurality of existing clusters, stored on a central server, wherein each cluster comprises noise samples obtained from other speakers;identifying, for each speaker-specific noise sample, a cluster that contains noise samples that are similar to the extracted speaker-specific noise sample; andadding each extracted speaker-specific noise sample into the identified existing cluster.

20. The method as claimed in any of claims 15 to 19 wherein clustering the extracted speaker-specific noise samples further comprises:determining an environment in which each obtained audio sample corresponding to the extracted speaker-specific noise samples was recorded; andclustering the extracted speaker-specific noise samples into at least one cluster based on the determined environment for each audio sample.

21. The method as claimed in any of claims 15 to 20 further comprising:transmitting the extracted speaker-specific noise to a central server storing a plurality of existing clusters containing noise samples obtained from other speakers.

22. The method as claimed in any of claims 15 to 21 wherein obtaining a plurality of audio samples that include a voice of the speaker and speaker-specific noise comprises:retrieving audio samples which have been collected during an enrolment period in which the speaker records audio samples of their voice.

23. The as claimed in any of claims 15 to 21 wherein obtaining a plurality of audio samples that include a voice of the speaker and speaker-specific noise comprises:retrieving audio samples which have been collected when the speaker uses their voice to control the user device.

24. A user device for generating a training dataset for personalising a trained speaker verification machine learning, ML, model for a speaker that uses the user device, the user device comprising:

25. A computer-readable storage medium comprising instructions which, when executed 5 by a processor, causes the processor to carry out the method of any of claims 1 to 13 and 15 to 23.s