Multitask Learning for Personalized Keyword Spotting

Abstract

Systems and techniques are provided for processing audio data. For example, the systems and techniques may be used for personalized keyword spotting with multi-task learning (PK-MTL). The process may include obtaining an audio sample, generating a representation of a keyword based on the audio sample, and generating a representation of a speaker based on the audio sample. The speaker may be associated with the keyword. A first similarity score may be determined based on the reference representation and one or more of the representation of the keyword and the representation of the speaker. The reference representation may be associated with one or more of the keyword and the speaker. A keyword spotting (KWS) output may be generated based on analyzing the first similarity score against at least a first threshold, the KWS output accepting or rejecting the audio sample as including the target keyword.

Description

[Technical field]

[0001]

[0001] The present disclosure relates generally to audio signal processing. For example, aspects of the present disclosure relate to systems and techniques for performing personalized keyword spotting (KWS) using one or more machine learning systems. [Background technology]

[0002]

[0002] Electronic devices such as smartphones, tablet computers, wearable electronic devices, and smart TVs are becoming increasingly popular among consumers. These devices can provide voice and / or data communication capabilities over wireless or wired networks. In addition, such electronic devices can include other features that provide various functions designed to enhance user convenience. The electronic devices can include speech recognition capabilities for receiving voice commands from a user. Such capabilities enable the electronic devices to perform functions associated with the voice command (e.g., a keyword) when the voice command from the user is received and recognized. For example, the electronic device can activate a voice assistant application, play an audio file, or take a photo in response to a voice command from the user.

[0003]

[0003] In electronic devices with speech recognition capabilities, manufacturers or carriers can often equip the devices with predefined keywords and associated sound models that can be used in detecting keywords in input sounds. Some electronic devices can also allow users to specify keywords as voice commands. For example, the electronic device can receive multiple utterances of keywords from a user and generate a keyword model for the specified keywords from the utterances. The detection performance of the keyword model can be related to the number of utterances for which the keyword model is generated. The detection performance of the keyword model can improve as the number of utterances increases. For example, a manufacturer can provide an electronic device with keyword models generated from thousands or more utterances.

[0004]

[0004] However, in some electronic devices, the number of keyword utterances received from a user is relatively small (e.g., less than 5). Therefore, a keyword model generated from such a limited number of utterances may not provide sufficient detection performance. However, receiving a significant number of utterances from a user to generate a keyword model that can provide sufficient detection performance may be time-consuming and inconvenient for the user. Summary of the Invention

[0005]

[0005] In some examples, systems and techniques for personalized keyword spotting through multi-task learning (PK-MTL) are described. According to at least one example, a method for processing one or more data samples is provided, the method including: acquiring an audio sample; generating a representation of a keyword, the representation of the keyword being generated based on the audio sample; generating a representation of a speaker, the speaker being associated with the keyword, the speaker representation being generated based on the audio sample; determining a first similarity score based on a reference representation associated with one or more of the keyword and the speaker, and one or more of the representation of the keyword and the speaker representation; and generating a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, the KWS output accepting or rejecting the audio sample as including the target keyword.

[0006]

[0006] In another example, an apparatus is provided for processing one or more data samples, the apparatus including a memory (e.g., configured to store data such as audio data) and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured to, and can: obtain an audio sample; generate a representation of a keyword, where the representation of the keyword is generated based on the audio sample; generate a representation of a speaker, where the speaker is associated with the keyword, where the representation of the speaker is generated based on the audio sample; determine a first similarity score based on a reference representation associated with one or more of the keyword and the speaker and one or more of the representation of the keyword and the speaker representation; and generate a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, the KWS output accepting or rejecting the audio sample as including the target keyword.

[0007]

[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to obtain an audio sample, generate a representation of a keyword, where the representation of the keyword is generated based on the audio sample, generate a representation of a speaker, where the speaker is associated with the keyword, where the speaker representation is generated based on the audio sample, determine a first similarity score based on a reference representation associated with one or more of the keywords and speakers and one or more of the representation of the keyword and the speaker representation, and generate a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, where the KWS output accepts or rejects the audio sample as containing the target keyword.

[0008]

[0008] In another example, an apparatus is provided for processing one or more data samples, the apparatus including: means for acquiring an audio sample; means for generating a representation of a keyword, the representation of the keyword being generated based on the audio sample; generating a representation of a speaker, the speaker being associated with the keyword, the representation of the speaker being generated based on the audio sample; determining a first similarity score based on a reference representation associated with one or more of the keyword and the speaker, and one or more of the representation of the keyword and the speaker representation; and generating a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, the KWS output accepting or rejecting the audio sample as including the target keyword.

[0009]

[0009] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0010]

[0010] The above, together with other objects and advantages associated with the aspects disclosed herein, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief description of the drawings]

[0011]

[0011] The accompanying drawings are presented to aid in the description of various aspects of the present disclosure and are provided only to illustrate the aspects and not to limit the aspects. [Figure 1]

[0012] FIG. 1 illustrates an exemplary implementation of a System-on-a-Chip (SoC), in accordance with some examples. [Figure 2A]

[0013] 1 shows an example of a fully connected neural network with some examples. [Figure 2B] 1 shows an example of a fully connected neural network with some examples. [Diagram 3]

[0014] FIG. 1 illustrates an exemplary keyword spotting (KWS) scenario, according to some examples. [Figure 4A]

[0015] 1 illustrates an example categorization that may be applied to inputs of a keyword spotting system, according to some examples. [Figure 4B]

[0016] FIG. 1 illustrates exemplary accuracy percentages of a keyword spotting (KWS) system for a conventional KWS (C-KWS) task, a target-user biased KWS (TB-KWS) task, and a target-user only KWS (TO-KWS) task, in accordance with some examples. [Figure 5A]

[0017] FIG. 1 illustrates an exemplary architecture of a personalized keyword spotting with multi-task learning (PK-MTL) network, in accordance with some examples. [Figure 5B]

[0018] FIG. 1 illustrates an example architecture of a task-specific module for episodic learning of TO-KWS and TB-KWS tasks, according to some examples. [Figure 6]

[0019] 1 is a flow diagram illustrating an example of a process for processing one or more audio samples, according to some examples. [Figure 7]

[0020] FIG. 1 is a block diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012]

[0021] Some aspects and examples of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the aspects and examples of the present application. However, it will be apparent that the various aspects and examples can be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0013]

[0022] The following description provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an effective description for implementing the examples and embodiments of the present disclosure. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0014]

[0023] Voice activity detection can refer to the detection of a human voice by a computing device to perform some function. For example, a computing device can use voice activity detection, such as by recognizing certain keywords, to wake or activate the device, interrogate the device, and / or cause the device to perform various other functions. Voice activity detection can also be used in more complex functions, such as far-field voice detection (e.g., from mobile devices positioned across a room), user identity verification (e.g., by voiceprint), voice detection during other audio output (e.g., detecting a voice command while music is playing on a device or detecting an interrupt command while a smart assistant is speaking), and / or voice interaction in complex noisy environments, such as in a moving vehicle.

[0015]

[0024] Keyword Spotting (KWS) is the task of detecting one or more keywords in an audio signal (e.g., an audio signal containing human speech or spoken words). For example, KWS can be used to distinguish activation phrases or specific commands from other speech and noise in the audio signal. In some cases, KWS systems can be targeted to or utilized by edge devices such as mobile phones and smart speakers. Detected keywords can include single words, compound words, phrases containing multiple words, etc. In some cases, KWS can be implemented based on a set of predefined keywords and / or a set of user-defined keywords. In some cases, the user-defined keywords can include one or more adaptations, adjustments, etc. determined based on certain characteristics of a given user's voice or speech.

[0016]

[0025] The audio signal may be a streaming audio signal, and in some examples, the KWS may be implemented on the streaming audio signal in real time. The streaming audio signal may be recorded by or obtained from a microphone associated with the computing device. The KWS may be implemented locally or remotely. For example, the KWS may be implemented locally using one or more processors of the same computing device that collects or obtains the streaming audio signal. In some examples, the KWS may be implemented remotely by transmitting the streaming audio signal (or a representation thereof) from the local computing device to a remote computing device (e.g., the local computing device records the audio signal but offloads the KWS processing tasks to the remote computing device). Implementing the KWS locally may result in lower overall latency or computation time, but may result in reduced accuracy. Implementing the KWS remotely may result in higher latency, but may result in increased accuracy.

[0017]

[0026] For example, a local computing device (e.g., a smartphone) often has less computational power than a remote computing device (e.g., a cloud computing system) and may therefore generate KWS results with lower accuracy or overall performance, especially when subject to time constraints associated with providing KWS results in real-time or near real-time. For example, a local computing device may implement a KWS detection model with lower complexity than one implemented on a remote computing device to provide real-time KWS results. Lower accuracy KWS results may include false positives (e.g., identifying a keyword that does not actually exist), false negatives (e.g., failing to identify a keyword that does exist), and classification errors (e.g., identifying a first keyword as some other keyword).

[0018]

[0027] However, implementing a KWS remotely can introduce communication latencies that can offset the increased accuracy associated with a remote KWS. For example, a remote KWS can introduce latencies along the communication path from the local computing device to the remote computing device (e.g., the time to transmit a streaming audio signal or a representation thereof to the remote computing device) and along the return communication path from the remote computing device to the local computing device (e.g., the time to transmit the KWS results back from the remote computing device to the local computing device).

[0019]

[0028] As mentioned above, in some examples, KWSs are often implemented in real-time (or near real-time) to enable user interaction with one or more computing devices. The lag between the time a user speaks a keyword (e.g., an activation phrase or a specific command) and the time the computing device provides a corresponding response or action can be a significant factor in the user's willingness to utilize the spoken command. In some cases, a delay of several seconds can frustrate or otherwise discourage a user from using verbal commands. Thus, there is a need for improved KWS performance in local and / or remote KWS implementations, since both local and remote KWS implementations are often time-bound processes.

[0020]

[0029] In some approaches to KWS, one or more machine learning networks are used to detect keywords in an audio signal based on a set of predefined keyword classifications. For example, a KWS can be implemented using one or more neural networks trained over a set of predefined keyword classifications. The resulting neural network can achieve accurate performance when evaluated against a closed set of keywords seen in training, but is often limited to detecting only keywords seen in training (e.g., keywords that correspond to the keyword classifications in the training dataset).

[0021]

[0030] In some cases, a few-shot KWS can be used to detect or classify keywords that were not originally seen during training. In an exemplary FSL-based approach to KWS, a neural network classifier is trained on a training set that includes labeled samples representing various predefined classes or categories of keywords. During training, the neural network classifier absorbs knowledge from the labeled samples of the training set. For example, the neural network classifier can learn an embedding or embedding space for relating training samples to their corresponding labels. The trained neural network classifier can then be adapted or generalized (e.g., in inference) to the task of evaluating new classes (e.g., unseen classes) that were not seen in the training set.

[0022]

[0031] For example, given only a small number of support examples for each new class, the FSL-based neural network classifier can predict the class of an unlabeled point (e.g., a query set) using an attention mechanism on the learned embeddings of a set of labeled support examples (e.g., a support set). In a query-by-example process, a trained FSL-based neural network classifier can receive a support set including M support examples for each of N first-time classes and a query set including one or more query samples. The neural network classifier can determine a prototype representation of each first-time class N (e.g., using the M support examples associated with each first-time class N). Each unlabeled query sample can be classified into one of the previously unseen classes N based on the calculated distance between the query sample and each prototype representation. In some cases, this inference process may be referred to as an N-ary M-shot episode, and the goal of the FSL-based neural network classifier is to correctly classify the query set into N classes that were not seen during training but are known using the M support examples.

[0023]

[0032] In real-world scenarios in which a KWS may be utilized, the input audio signal may, of course, contain unexpected new classes of utterances, words, phrases, etc. for which no supporting examples have been provided. Unexpected new classes may be classes for which no supporting examples were included in the initial training set or support set. These unexpected samples and / or classes may be referred to as an "open set" (e.g., as opposed to a closed set of classes from the initial training set and support set).

[0024]

[0033] Neural network classifiers may exhibit overconfidence and misclassify first-time open set samples into one of the FSL classes. In some examples, the accuracy and / or performance of a KWS system may be improved by rejecting open set utterances rather than classifying them into one of the predefined keyword classifications (e.g., of a closed set). For example, Open-Set Recognition (OSR) may be used to detect first-time open set classes and then process and / or reject query samples that belong to the detected open set classes.

[0025]

[0034] Described herein are systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") for personalized keyword spotting (KWS) using multi-task learning (MTL). As previously described, KWS can be used to implement speech-based user interaction on various computing devices and / or smart devices. In some cases, conventional KWS (C-KWS) systems can focus on detecting predefined keywords that are user-agnostic. However, speech-based user interaction can come from or otherwise be associated with a target user (e.g., an owner) registered with the device. Described herein are systems and techniques for personalized keyword spotting with multi-task learning (PK-MTL), which can be used to further reduce detection errors. In some aspects, a target-user biased KWS (TB-KWS) can be used to provide a personalized KWS with reduced detection errors. In some aspects, a target-user dedicated KWS (TO-KWS) can be used to provide a personalized KWS with reduced detection errors.

[0026]

[0035] In one illustrative example, the systems and techniques described herein may implement or provide personalized keyword spotting with multi-task learning (PK-MTL). PK-MTL may include a multi-task learning portion and a task adaptation portion, as described in more detail below. In some aspects, multi-task learning may be used for keyword spotting and speaker verification to leverage user information (e.g., speaker characteristics) into the keyword spotting system. For example, the PK-MTL framework may include a shared encoder that generates features or embeddings based on receiving one or more utterances, sound samples, audio data, etc. as input. The output of the shared encoder may be provided to a first sub-network used to generate keyword embeddings (e.g., may be provided to a keyword sub-network or a KWS sub-network). The output of the shared encoder may additionally be provided to a second sub-network used to generate speaker embeddings (e.g., may be provided to a speaker sub-network or a SV sub-network).

[0027]

[0036] In some examples, the keyword sub-network may be trained on or associated with a supervised keyword classification task. The speaker sub-network may be trained on or associated with a supervised speaker classification task. In some examples, the first cosine classifier may receive one or more keyword embeddings (e.g., from the keyword sub-network) as input and output one or more keyword classifications. The second cosine classifier may receive one or more speaker embeddings (e.g., from the speaker sub-network) as input and output one or more speaker classifications. In one illustrative example, a total loss function may be used to train the PK-MTL described herein, where the total loss function includes a loss for the task of supervised keyword classification and a loss for the task of supervised speaker classification.

[0028]

[0037] In some cases, the score combination module and / or task representation module can be used to fully adapt the multi-task learning architecture to personalized KWS scenarios (e.g., TB-KWS and TO-KWS scenarios, etc.). In some aspects, PK-MTL can be used to selectively perform TB-KWS and TO-KWS while achieving comparable or better results than the original C-KWS scenario. In some examples, the systems and techniques can be used to implement personalized KWS systems with reduced false alarm rates in real keyword spotting scenarios.

[0029]

[0038] For example, a Score Combination Module (SCM) may be used to determine a linear combination of the two scores based at least in part on one or more keyword embeddings and speaker embeddings. The personalized keyword spotting based on the linear combination determined by the SCM may be associated with a reduced false alarm rate. In some examples, the keyword embeddings and speaker embeddings from the two respective sub-networks may be used to determine one or more task-specific embeddings for each task by a learnable Task Representation Module (TRM). The personalized keyword spotting based on the task-specific embeddings may be associated with a reduced false alarm rate.

[0030]

[0039] Various aspects of the present disclosure are described with reference to the figures. Figure 1 illustrates an example implementation of a system on chip (SoC) 100 that may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., neural networks with weights), delays, frequency bin information, task information, among other information, may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, and / or may be distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 or from the memory block 118.

[0031]

[0040] The SoC 100 may also include a connectivity block 110 that may include a GPU 104, a DSP 106, fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., and additional processing blocks tailored to specific functions, such as a multimedia processor 112 that may detect and recognize gestures, speech, and / or other interactive user actions or inputs. In one implementation, the NPU is implemented within the CPU 102, the DSP 106, and / or the GPU 104. The SoC 100 may also include a navigation module 120 that may include a sensor processor 114, image signal processors (ISPs) 116, and / or a global positioning system. In some examples, the sensor processor 114 may be associated with or connected to one or more sensors for providing sensor input to the sensor processor 114. For example, one or more sensors and the sensor processor 114 may be provided within, coupled to, or otherwise associated with the same computing device.

[0032]

[0041] In some examples, the one or more sensors may include one or more microphones for receiving sounds (e.g., audio input), including sounds or audio input that may be used to perform keyword spotting (KWS) as described herein. In some cases, sounds or audio input received by the one or more microphones (and / or other sensors) may be digitized into data packets for analysis and / or transmission. The audio input may include ambient sounds in the vicinity of the computing device associated with the SoC 100 and / or may include speech from a user of the computing device associated with the SoC 100. In some cases, the computing device associated with the SoC 100 may additionally or alternatively be communicatively coupled to one or more peripheral devices (not shown) and / or configured to communicate with one or more remote computing devices or external resources using a communication network, such as, for example, a wireless transceiver and a cellular communication network.

[0033]

[0042] The SoC 100 may be based on an ARM instruction set. In one aspect of the disclosure, the instructions loaded into the CPU 102 may comprise code for searching a stored multiplication result in a lookup table (LUT) corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into the CPU 102 may also comprise code for disabling a multiplier during a multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. Additionally, the instructions loaded into the CPU 102 may comprise code for storing a calculated multiplication product of an input value and a filter weight when a lookup table miss of the multiplication product is detected.

[0034]

[0043] The SoC 100 and / or components thereof may be configured to perform audio signal processing using one or more machine learning networks and / or techniques according to aspects of the present disclosure and as described herein. For example, the SoC 100 and / or components thereof may be configured to perform keyword spotting according to aspects of the present disclosure. In some examples, by using one or more neural network classifiers to perform few-shot learning (FSL) and / or open set recognition (OSR), aspects of the present disclosure may improve the accuracy and / or efficiency of keyword spotting (KWS).

[0035]

[0044] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences without explicit instructions. One example of an ML system is a neural network (also called an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks can be used for a variety of applications and / or devices, such as speech analysis, audio signal analysis, image and / or video coding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, among others.

[0036]

[0045] Individual nodes in a neural network can emulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data may be selectively passed to other neurons. A weight value is associated with each vector and node in the network, and these values ​​constrain how the input data relates to the output data. For example, the input data of each node may be multiplied by the corresponding weight value and the products may be summed. The summed products may be adjusted by an optional bias, and an activation function may be applied to the result to obtain the output signal or "output activation" of the node (sometimes called a feature map or activation map). The weight values ​​may initially be determined by an iterative flow of training data through the network (e.g., the weight values ​​are established during a training phase in which the network learns how to identify particular classes by their typical input data characteristics).

[0037]

[0046] There are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multilayer perceptron (MLP) neural networks, and transformer neural networks, among others. For example, convolutional neural networks (CNNs) are a type of feed-forward artificial neural network. A convolutional neural network may include a collection of artificial neurons, each with a receptive field (e.g., a spatially localized region of the input space) and tiling the input space together. RNNs work in a way that they save the output of a layer and feed this output back to the input to help predict the outcome of the layer. GANs are a form of generative neural network that can learn patterns in the input data so that the neural network model can generate new synthetic outputs that can reasonably be from the original dataset. A GAN can include two neural networks working together, including a generative neural network that generates a synthesized output and a discriminative neural network that evaluates the output for reliability. In an MLP neural network, data can be fed into an input layer, and one or more hidden layers provide a level of abstraction to the data. Predictions can then be made in the output layer based on the abstracted data.

[0038]

[0047] Deep learning (DL) is an example of a machine learning technique and can be considered a subset of ML. Many DL approaches are based on neural networks, such as RNNs or CNNs, and utilize multiple layers. The use of multiple layers in a deep neural network can allow increasingly higher levels of features to be extracted from a given input of raw data. For example, the output of a first layer of artificial neurons becomes the input to a second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input to a third layer of artificial neurons, and so on. The layers located between the input and output of the entire deep neural network are often referred to as hidden layers. Hidden layers learn (e.g., are trained) to transform intermediate inputs from previous layers into slightly more abstract and synthetic representations that can be provided to subsequent layers until a final or desired representation is obtained as the final output of the deep neural network.

[0039]

[0048] As mentioned above, a neural network is an example of a machine learning system, and may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processing is performed by hidden nodes in one or more hidden layers, and output is generated through output nodes in the output layer. A deep learning network typically includes multiple hidden layers. Each layer of a neural network may include a feature map or activation map, which may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. The nodes may include one or more weights that are used to indicate the importance of one or more nodes of the layer. In some cases, a deep learning network may have a series of many hidden layers, with early layers being used to determine simple, low-level characteristics of the input, and later layers building a hierarchy of more complex and abstract characteristics.

[0040]

[0049] Deep learning architectures may learn a hierarchy of features. If presented with visual data, for example, a first layer may learn to recognize relatively simple features such as edges in the input stream. In another example, if presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases. Deep learning architectures may perform particularly well when applied to problems that have a natural hierarchical structure. For example, classification of electric vehicles may benefit from first learning to recognize wheels, windshields, and other features. These features may be combined in different ways in higher layers to recognize cars, trucks, and planes.

[0041]

[0050] Neural networks may be designed with various connectivity patterns. In feedforward networks, each neuron in a given layer communicates with neurons in a higher layer, so that information is passed from lower layers to higher layers. As described above, hierarchical representations may be constructed in successive layers of a feedforward network. Neural networks may also have recurrent or feedback (also called top-down) connections. In recurrent connections, the output from a neuron in a given layer may be transmitted to another neuron in the same layer. Recurrent architectures may be useful in recognizing patterns across two or more of the input data chunks delivered in a sequence to the neural network. Connections from neurons in a given layer to neurons in a lower layer are called feedback (or top-down) connections. Networks with many feedback connections may be useful when recognition of high-level concepts can help distinguish certain low-level features of the input.

[0042]

[0051] The connections between layers of a neural network may be fully connected or locally connected. FIG. 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, a neuron in a first layer may transmit its output to every neuron in a second layer, so that each neuron in the second layer receives input from every neuron in the first layer. FIG. 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in a layer has the same or similar connectivity pattern, but with connection strengths that may have different values ​​(e.g., 210, 212, 214, and 216). Because higher layer neurons in a given region may receive inputs that are tuned through training to the properties of a limited subset of all inputs to the network, the connectivity patterns of local connections may give rise to spatially distinct receptive fields within the higher layers.

[0043]

[0052] As mentioned above, systems and techniques for personalized keyword spotting (KWS) are described herein. Keyword spotting (KWS) can be used to implement speech-based user interactions on various computing devices and / or smart devices. In some cases, conventional KWS (C-KWS) systems can focus on detecting predefined keywords that are user-agnostic. However, many speech-based user interactions may come from or be associated with a target user (e.g., an owner) registered with the device. Described herein are systems and techniques for improved personalized KWS that can be used to further reduce detection errors. In some aspects, a target-user biased KWS (TB-KWS) can be used to provide a personalized KWS with reduced detection errors. In some aspects, a target-user dedicated KWS (TO-KWS) can be used additionally or alternatively to provide a personalized KWS with reduced detection errors.

[0044]

[0053] In one illustrative example, the systems and techniques described herein may implement or provide personalized keyword spotting with multi-task learning (PK-MTL) for keyword spotting and speaker verification, which may enable the systems and techniques to leverage user information into the keyword spotting system. PK-MTL may include a multi-task learning portion and a task adaptation portion, as described in more detail below.

[0045]

[0054] Task-specific modules for TB-KWS and TO-KWS can be constructed independently. The task-specific modules can be trained using episodes that mimic the test cases of each task, as described in more detail below. In some examples, the score combination module and / or the task representation module can be used to fully adapt the multi-task learning architecture to personalized KWS scenarios (e.g., TB-KWS and TO-KWS scenarios). PK- can be used to selectively perform TB-KWS and TO-KWS while achieving results comparable to or better than the original C-KWS scenario. In some examples, the system and techniques can be used to implement a personalized KWS system with reduced false alarm rates in real keyword spotting scenarios.

[0046]

[0055] In some cases, an always-on lightweight keyword spotting system may be used to wake up a smart audio device. For example, when a keyword is detected by the system, an audio stream including the detected keyword (or an audio stream following the detected keyword) may be uploaded or otherwise provided to one or more speech recognition systems. Conventional keyword spotting (C-KWS) aims to detect a small set of predefined speech signals from any speaker and is therefore speaker-independent (e.g., does not consider speaker identity). C-KWS is often used in always-on keyword spotting systems. In some examples, the use of C-KWS in always-on keyword spotting systems for personal devices may result in a high false alarm rate.

[0047]

[0056] In some examples, the majority of user interaction may originate from the target user. In some examples, existing speech recognition systems may require user interaction to come from a target user registered with the device. In some cases, speech recognition systems may require that the target used be explicitly registered with the device. However, C-KWS may not reflect this requirement. In some examples, the performance of the C-KWS system may degrade in various scenarios. FIG. 3 is a diagram 300 illustrating various scenarios in which a keyword spotting system may operate, including a desired usage scenario in which a target user utterance with a keyword wakes up the electronic device with a true positive. FIG. 3 also illustrates scenarios in which a keyword spotting system may operate in undesirable usage scenarios, such as when streaming audio 304 from a TV, audio 306 for an online meeting, audio 302 from a conversation, etc., are continuously fed into an always-on keyword spotting system 308 that causes the electronic device to wake up with one or more false alarms. For example, the streaming audio 304 may include a variety of words, including a target keyword or a keyword that has a similar pronunciation to a target keyword of a C-KWS system (e.g., speaker-independent and does not consider speaker identity). There is a need for more practical personalized keyword spotting with lower false alarm rates. In one illustrative example, the systems and techniques described herein may be used to perform personalized keyword spotting (and / or personalized keyword spotting tasks) by recognizing keywords according to the user. This process may reduce power consumption while maintaining a high recall rate and a low false alarm rate.

[0048]

[0057] FIG. 4A illustrates an example categorization 400 that can be applied to the input of a keyword spotting system. For example, the table on the left side of FIG. 4A illustrates an example categorization with four categories. In some examples, the input of a KWS system can be classified into four categories based on whether the speaker is a target speaker or a non-target speaker and whether the keyword is a target keyword or a non-target keyword. As shown in FIG. 4A, the four categories are "ts-tk" (utterance from a target speaker containing a target keyword), "nts-tk" (utterance from a non-target speaker containing a target keyword), "ts-ntk" (utterance from a target speaker containing a non-target keyword), and "nts-ntk" (utterance from a non-target speaker containing a non-target keyword).

[0049]

[0058] In some cases, the category "nts-tk" may be the most vague or ambiguous because the "nts-tk" category represents utterances whose keywords are the target to be detected but come from non-target users. In one illustrative example, the two personalized keyword spotting tasks described herein may be used to analyze the "nts-tk" category of utterances in an improved manner. For example, the target-user-biased KWS (TB-KWS) and the target-user-specific KWS (TO-KWS) described herein may consider "nts-tk" utterances differently to better focus on the target user.

[0050]

[0059] In one illustrative example, the TB-KWS can implement a model that is biased toward the target user. For example, by biasing the model toward the target user, the TB-KWS does not explicitly consider utterances that have the target keyword when coming from a non-target user (e.g., does not explicitly consider the "nts-tk" category of utterances).

[0051]

[0060] In another illustrative example, TO-KWS can be used to detect only utterances that contain the target keyword when coming from a target user. For example, TO-KWS can only detect the "ts-tk" category of utterances, but not the "nts-tk" category. In some cases, TO-KWS can be used in scenarios where some applications in a smart device prefer to be activated only for target keywords from a target user of the device.

[0052]

[0061] FIG. 4A also includes a table 405 illustrating the application of C-KWS, TB-KWS, and TO-KWS to the four categories of utterances mentioned above. In the table on the right, an "O" entry indicates positive. A "-" entry indicates neutral, and an "X" entry indicates negative. For example, it can be seen that the TB-KWS considers the "nts-tk" category of utterances as neutral, while the TO-KWS considers the same "nts-tk" category of utterances as negative. Additionally, it can be seen that the C-KWS considers the "nts-tk" category of utterances as positive, with each of the three KWS approaches considering utterances with target keywords (e.g., the "nts-tk" category) from non-target users differently.

[0053]

[0062] FIG. 4B is a diagram with a graph 410 showing example accuracy percentages of different KWS systems for the C-KWS task (e.g., left panel), the TB-KWS task (e.g., center panel), and the TO-KWS task (e.g., right panel). As shown in FIG. 4B, the C-KWS system can show high accuracy for the C-KWS task, but shows a decrease in performance for two personalized tasks (e.g., TB-KWS and TO-KWS) as seen by the increased EER percentage. In some examples, the systems and techniques described herein can be used to provide improved TB-KWS and TO-KWS while performing C-KWS with comparable or better results than the C-KWS system.

[0054]

[0063] The systems and techniques described herein may be used to implement a personalized keyword spotting by multi-task learning (PK-MTL) and an always-on lightweight keyword spotting system that may be selectively utilized for all three of the C-KWS, TB-KWS, and TO-KWS tasks. The PK-MTL described herein may leverage user information into the keyword spotting system. In some examples, the PK-MTL framework may be adapted to fully accommodate the personalized tasks of TB-KWS and TO-KWS, based at least in part on the score combination module (SCM) and task representation module (TRM).

[0055]

[0064] The systems and techniques may include PK-MTL implemented as a two-stage system including a multi-task learning system and a task adaptation system. In one illustrative example, to leverage speaker information into a keyword spotting system, multi-task learning may be first applied to keyword spotting and speaker verification. With regard to parameters and computational efficiency, the systems and techniques and / or PK-MTL described herein may use a hard parameter sharing scheme that shares all parameters in low-level layers, which also helps in learning complementary information from another task (e.g., during training). In some examples, the PK-MTL described herein may obtain keywords and / or speaker representations in a computationally efficient manner.

[0056]

[0065] Then, task-specific features can be introduced to fully adapt the keyword and speaker representations to the TB-KWS and TO-KWS. In some cases, the objectives of the three tasks (e.g., C-KWS, TB-KWS, and TO-KWS) may be different, so a feature that combines information from keyword and speaker representations suitable for each task can be utilized. For example, a score combination module (SCM) and a task representation module (TRM) can be used as task-specific features.

[0057]

[0066] In some examples, SCM can include an optimization-free approach that directly combines similarity scores from keyword and speaker expressions. The optimization-free SCM approach can provide the ability for a network (e.g., a machine learning network or a keyword spotting system) to perform the TB-KWS and TO-KWS.

[0058]

[0067] In some examples, the second approach can include building a trainable task representation module (TRM) designed to take the two representations and extract a new task-specific representation for each task. For example, the TRM can be trained with a metric learning loss in a batch construction manner that mimics the test cases for each task (e.g., for TO-KWS, "ts-tk" utterances are positive and the other three categories of utterances are negative). In some examples, the use of a TRM can cause the final embeddings of positive samples to be clustered closer together and the final embeddings of negative samples to be farther apart.

[0059]

[0068] In some examples, the systems and techniques described herein can be evaluated on the Google Speech Commands dataset. Using multiple keyword spotting backbones, the PK-MTL described herein can improve performance on TB-KWS and TO-KWS while providing comparable results on C-KWS. In some examples, the systems and techniques described herein can be tested in a realistic setting where negative samples come continuously from TV and other noises and / or noise sources. In one illustrative example, WSJ-SI200 and Librispeech can be used as negative samples to reduce the false alarm rate based on speaker expressions.

[0060]

[0069] 5A and 5B show an example architecture according to the systems and techniques described herein. In some cases, a two-part architecture may be used, including multi-task learning for a keyword spotting (KWS) architecture and a speaker verification (SV) part, and one or more task-specific features to adapt to a personalized KWS scenario. For example, FIG. 5A shows an example multi-task learning framework 500 including a shared encoder 504 that receives an input spectrogram 502 as an input. A shared representation of keyword spotting and speaker verification may be encoded in a low-level layer by the shared encoder 504. Sub-networks including a KWS subnetwork 506 and an SV subnetwork 508 may be used to learn the characteristics of each task in high-level features. In some examples, cosine classifiers including a cosine classifier 510 and a cosine classifier 512 may be trained to classify keyword classes 511 and speaker classes 513, as described in more detail below.

[0061]

[0070] 5B shows an example of task-specific features. As shown, the learned keywords and speaker embeddings 514 are fully adapted to the TB-KWS and TO-KWS by a task-specific module 516 that implements the task-specific features to generate task-specific embeddings 518.

[0062]

[0071] In some examples, the training data D train is a labeled sample

[0063]

number

[0064] where x i are the input audio features,

[0065]

number

[0066] For example,

[0067]

number

[0068] can represent the corresponding keywords and speaker labels, respectively. In some cases, to exploit speaker characteristics in a keyword spotting system, the multi-task learning architecture of FIG. 5A can be implemented using a shared encoder f Φ (·), one or more sub-networks for keyword spotting and speaker verification, respectively.

[0069]

number

[0070] and the classifier, g k (·) and g s It may contain (·).

[0071]

[0072] As previously described, the systems and techniques described herein can be used to implement personalized keyword spotting using keyword and speaker information. In some examples, for a given backbone network f θ For (·)(θ={Φ,φ}), hard parameter sharing can be used for the low-level layers (e.g., the shared encoder f Φ (·)). In some instances, the shared encoder can learn complementary information for both KWS and SV, and the shared representation is efficient in terms of memory and computational complexity compared to separate task designs.

[0072]

[0073] In some cases, KWS and SV features may be antagonistic to each other at a high level of concept (e.g., keyword features are speaker-agnostic and vice versa), and the sub-network

[0073]

number

[0074] For example, the “KWS subnet” shown in Figure 5A is a subnetwork

[0075]

number

[0076] The "SV subnet" shown in FIG. 5A may be the same as or similar to the aforementioned subnetwork.

[0077]

number

[0078] It may be the same as or similar to.

[0079] In one illustrative example, keywords and speaker features (e.g., embeddings) may be obtained from each one of the two sub-networks. For example, keyword features (e.g., keyword embeddings) may be

[0080]

number

[0081] and the speaker features (e.g., speaker embeddings) can be determined as

[0082]

number

[0083] In some examples, both the keyword features and the speaker features can be determined by a shared encoder f Φ (·), which may be generated based on the output from

[0084]

number

[0085] is the shared encoder output f Φ (x i ) to the KWS subnetwork

[0086]

number

[0087] Similarly, speaker features are generated by providing

[0088]

number

[0089] is the shared encoder output f Φ (x i ) to the SV subnetwork

[0090]

number

[0091] The signal may be generated by providing the signal

[0092]

[0075] Keyword features

[0093]

number

[0094] and speaker characteristics

[0095]

number

[0096] may be provided as input to one or more cosine classifiers, shown in FIG. 5A as a pair of cosine classifiers with one cosine classifier for each of the two sub-networks (e.g., a first cosine classifier for the keyword sub-network and a second cosine classifier for the speaker sub-network). In one illustrative example, the cosine similarity based classifier may be applied as follows:

[0097]

number

[0098]

[0076] Here, W k are the learnable weights for keyword classification, sim is the cosine similarity,

[0099]

number

[0100] where s and b denote the scale and bias scalar parameters, respectively.

[0101]

[0077] The keyword classification loss can be determined by minimizing the negative log-probability of the true class.

[0102]

number

[0103] In some examples, the speaker classification loss L s can be determined for the speaker sub-network branch in the same way as equation (2) for the keyword sub-network branch above. Combining the keyword classification loss and the speaker classification loss (e.g., combining two task-specific loss functions) gives: L mtl =L k +λL s Formula (3)

[0104]

[0079] where λ is a parameter that can be used to indicate the importance of speaker information. In one illustrative example, the PK-MTL architecture described herein can be trained according to the loss function given in equation (3) above. Examples of the use of the learned representations for TB-KWS and TO-KWS are described below.

[0105] In some examples, keywords and speaker scores may be obtained initially in a testing phase and used to make decisions regarding accepting or rejecting asserted utterances based on predefined thresholds. For example,

[0106]

number

[0107] Registered utterance x ref is provided or otherwise obtained. Then, the same target user

[0108]

number

[0109] Test input sample for x i The keyword and speaker scores for can be calculated as follows:

[0110]

number

[0111]

[0081] Here,

[0112]

number

[0113] is keyword embedding

[0114]

number

[0115] and your target keywords

[0116]

number

[0117] is the cosine similarity score between the classifier weights of

[0118]

number

[0119] are the learnable weights in the keyword classifier, and the trained weights can be regarded as the most representative keyword embeddings.

[0120]

number

[0121] is the input

[0122]

number

[0123] Speaker embedding and registered target user

[0124]

number

[0125] and the cosine similarity score between them. Note that in some examples, the target keywords may be predefined, but the speakers do not overlap between training and testing, and reference embeddings from the target users may be utilized in the testing phase. Two exemplary approaches for adapting the learned PK-MTL architecture to TB-KWS or TO-KWS scenarios, respectively, are described below.

[0126] As mentioned above, in one illustrative example, a score combination module (SCM) is used to combine two scores ψ k and ψ s can be directly combined. The objective of keyword spotting may be to minimize the false rejection rate (FRR) at a given false alarm rate (FAR), where the FAR may be given as the percentage of cases where negative samples are incorrectly accepted, while the FRR may be given as the percentage of cases where positive samples are incorrectly rejected. In some cases, the systems and techniques described herein can minimize the FRR at a given FAR by selecting SCM(·,·;α) as follows: α*=arg α minFRR(SCM(ψ k , ψ s ;α)),stFAR=c Equation (5.1)

[0127]

[0083] where α represents a parameter of the SCM for combining the two scores, and c(%) represents the target FAR percentage. SCM(·,·;α) can be provided as various functions. For example, in some cases, SCM(·,·;α) can be a linear combination function. SCM(ψ k , ψ s ;α)=α ψ k +(1-α) ψ s Equation (5.2)

[0128]

[0084] Note that in some examples, the parameter α can be used to represent the importance weight of keyword spotting for each task. In some examples, the parameter α can be selected from a grid search on the validation set.

[0129] In one illustrative example, a task representation module (TRM) may be utilized, the TRM including one or more trainable neural networks. For example, the systems and techniques may include a trainable neural network, the TRM TB (·,·) and TRM TO (·,·), whose inputs are keyword and speaker embeddings and whose output are task-specific embeddings. As mentioned above, FIG. 5B illustrates an example architecture including a task-specific module that generates task-specific embeddings based on receiving one or more keyword and speaker embeddings as input. In some examples, the keyword embeddings and / or speaker embeddings utilized by the task-specific module of FIG. 5B may be the same or similar to the keyword and speaker embeddings from the multi-task learning framework illustrated in FIG. 5. Returning to FIG. 5B, in some examples, the keyword and speaker embeddings may be provided as input to a task-specific module that includes one or more task-specific features for fully adapting the learned keyword and speaker embeddings to the TB-KWS and TO-KWS. For example, the task-specific embeddings generated by the task-specific module may be used to perform episodic learning of the TO-KWS and TB-KWS, as illustrated in FIG. 5B.

[0130] In some examples, the TRM can be trained to form discriminative embeddings for the TB-KWS and the TO-KWS. For example, a metric learning objective (e.g., a metric learning loss) for the TB-KWS and the TO-KWS can be used to cause the module to form distinctive embeddings for the TB-KWS and the TO-KWS. In some cases, the metric learning objective and / or the metric learning loss can be applied using a batch construction that mimics the test cases for each task.

[0131] As illustrated in the right panel of FIG. 4A, positive and negative sample pairs can be defined for each task (e.g., positive pairs are represented as open circles "O" in FIG. 4A, and negative pairs are represented as "X"). In some examples, positive and negative sample pairs can be defined based on anchor samples in a mini-batch, with the goal of moving positive samples closer to each other and negative samples farther away from each other.

[0132] Each mini-batch may contain a prototype set and a query set. The distance (e.g., similarity) between the prototype and query samples may be measured as follows:

[0133]

number

[0134] Here, the weights of the trainable classifier are

[0135]

number

[0136] In training, the prototype

[0137]

number

[0138] can be used as, respectively.

[0139]

number

[0140] is the prototype of the corresponding keyword of the jth sample. Note that z and p can be obtained from the test sample and the reference sample, respectively, at the time of testing. The task-specific loss function can be defined as follows:

[0141]

number

[0142]

[0090] In some examples, the same task-specific loss function (e.g., Equation (7)) can be used for TB-KWS and TO-KWS. In some cases, the mini-batch construction method can be different for TB-KWS and TO-KWS. For example, for TB-KWS, target keywords from non-target users are not selected for batch construction. Through the objective function, the task-specific module can extract discriminative features for each task. In one illustrative example, a multi-task learning architecture with TRM as described herein may be referred to as PK-MTL (e.g., personalized keyword spotting through multi-task learning).

[0143]

[0091] An exemplary experimental setup and experimental results are described below. For example, the conventional personalized KWS system can be evaluated on the Google Speech Commands v1 dataset, which includes 64,727 utterances of 30 words from 1,881 speakers. A conventional 12-class classification setup can be used, which consists of 10 classes: "yes", "no", "up", "down", "left", "right", "on", "off", "stop", and "start", and two additional classes, "unknown" and "silence", which respectively indicate the remaining 20 words and no speech. Each utterance is 1 second long, and the sampling rate is 16 kHz. The dataset can be split into a training set, a validation set, and a test set, and the "unknown" and "silence" classes can be rebalanced in the split. A standard test set can be used for evaluation. In one illustrative example, for TB-KWS and TO-KWS, the samples in the test set can be used to generate pairs between samples to include positive pairs and negative pairs. In some cases, the anchor samples can be selected randomly, and ts-tk, nts-tk, ts-ntk, and nts-ntk samples (e.g., as described with respect to FIG. 4A) can be selected from the anchors. To reduce the performance variability, 10 test splits can be obtained, with each test split including 16,000 sample-to-sample pairs. In some cases, the experimental results can be reported based on the average performance from the 10 test splits. Note that in some cases, the keywords "silence" and "unknown" can be selected for the non-target keywords, but not for the target keywords.

[0144]

[0092] In some examples, the KWS system can be evaluated in realistic scenarios. For example, a scenario can be simulated in which speech comes continuously from a news broadcast or conversation that contains many words (e.g., a common negative source for the KWS system). In some cases, the WSJ-SI200 and Librispeech datasets can be used as negatives. The negative samples from WSJ-SI200 can be segmented into 1-second long parts from the entire audio stream. For Librispeech, the entire audio stream can be segmented into 1-second long parts in the public clean and noisy test set.

[0145] In some examples, three keyword spotting architectures can be used as the backbone network for the PK-MTL described herein (e.g., BC-ResNet, Res15, and / or DS-ResNet). For BC-ResNet, the input features of a 40-dimensional logarithmic Mel spectrogram with a window length of 30 ms and a frame shift of 10 ms can be used, and one or more data augmentations (e.g., time shift, background noise, SpecAugment) are applied. For Res15 and DS-ResNet, noise and random shifts can be added to each segment. In some aspects, 40-dimensional Mel-frequency cepstral coefficient features can then be extracted and used as input. The PK-MTL can include a shared encoder and sub-network for KWS and SV, as shown in FIG. 5A and described above. The shared encoder can be designed using a complete backbone, except for the last two conv layers for BC-ResNet and the last conv block for Res15 and DS-ResNet. In some examples, the KWS and SV sub-networks may be provided downstream of the shared encoder and may include the remaining layers of each backbone and an additional fully connected layer. The formal training strategy of each baseline (e.g., backbone) network may be utilized (e.g., learning rate, optimization method, and mini-batch size, respectively). In some examples, the parameter λ in Equation (3) may be 0.1. For the baselines (e.g., BC-ResNet, Res15, and DS-ResNet), an additional fully connected layer may be added before the classifier, and the classifier may be changed to a cosine classifier (e.g., the cosine classifier described above with respect to FIG. 5A). In some examples, these modifications may improve performance at the cost of additional parameters and computational costs.

[0146] In one illustrative example, the score combination module (SCM) can be defined as a linear combination function, whose parameters can be selected on a validation set with a target FARc set to 1%. For the task representation module (TRM), a task-specific module can be added consisting of two fully connected layers. To reduce the model size, the intermediate feature size can be set to 2. The module can be trained for 50 epochs using the same training strategy described above for learning the PK-MTL architecture.

[0147]

[0095] In some examples, ablation studies can be performed on C-KWS, TB-KWS, and TO-KWS. For example, to analyze the impact of leveraging speaker information into a keyword spotting system, four methods (e.g., Vanilla, Naive MTL, Sep.w / SCM, and PK-MTL described herein) can be compared. Vanilla uses only the loss function for keyword spotting. The multi-task learning framework described herein but does not include task-specific features (e.g., those described with respect to FIG. 5B) is denoted as Naive MTL. Sep. shows that it utilizes keywords and speaker representations from two separate networks (e.g., KWS and SV networks). In Table 1, we can see that Naive MTL can perform both KWS and SV tasks with a small increase in computational cost, and also boosts performance on keyword spotting (e.g., C-KWS and TB-KWS) by learning complementary information from SV. Sep.w / SCM may require significant computation due to solving one task per forward pass. Compared to Sep., the systems and techniques described herein can take advantage of the hard parameter sharing of most of the network for KWS and SV, which can significantly reduce computational costs. It can be seen that the PK-MTL described herein significantly outperforms TB-KWS and TO-KWS, which have roughly the same number of parameters, but is associated with a much smaller computational footprint or overhead than Sep.w / SCM.

[0148]

[0096] With a simple score combination function, PK-MTL w / SCM significantly boosts the performance of TB-KWS and TO-KWS compared to naive MTL. However, the performance improvement is limited because the representation cannot explicitly learn task-specific characteristics. Through TRM (e.g., task representation module), PK-MTL can fully adapt the keywords and speaker representations to TB-KWS and TO-KWS, and Table 1 shows that it outperforms other comparisons for all metrics.

[0149] In one illustrative example, the PK-MTL framework described herein can be applied to one or more existing conventional keyword spotting architectures. For example, as shown in Table 1, PK-MTL based on all three backbones, BC-ResNet, Res15, and DS-ResNet, improves the performance of C-KWS, as well as TB-KWS and TO-KWS, using speaker representations and the TRM module described herein. Although the performance of SV and the impact of SV information on KWS may vary depending on the structure and size of the baseline, PK-MTL consistently outperforms the baseline for all tasks.

[0150]

[0098] As mentioned above, the KWS system can be evaluated in realistic scenarios such as simulated scenarios where speech continuously arrives from news streams or conversations that contain words that are generally negative for the KWS system. In one illustrative example, on the device, the keyword spotting system can select a threshold based on the target FRR, at which positive samples are falsely rejected. Using the threshold, conventional personalized keyword spotting systems can be evaluated based on their performance in rejecting general negatives from news and conversations. For example, Table 2 shows the FAR at specific operating points FRR1% and FRR10% on the Librispeech and WSJ datasets. In some examples, vanilla methods that use only loss functions for KWS cannot correctly reject general negatives because general negatives include target keywords or keywords with similar pronunciation to the target, which results in a high score for negatives. The systems and techniques described herein utilize both keyword information and speaker information to more accurately and efficiently reject general negatives. In some examples, the systems and techniques described herein may utilize PK-MTL to selectively implement one or more of the three tasks (e.g., C-KWS, TB-KWS, TO-KWS) according to, for example, target device requirements, use cases, user preferences, etc.

[0151]

[0099] Figure 6 is a flow chart illustrating an example of a process 600 for processing one or more audio samples. Although the example process 600 illustrates a particular sequence of operations, the sequence may be modified without departing from the scope of the present disclosure. For example, some of the illustrated operations may be performed in parallel or in a different sequence that does not substantially affect the functionality of the process 600. In other examples, different components of an example device or system implementing the process 600 may perform functions substantially simultaneously or in a particular sequence.

[0152]

[0100] At block 602, the process 600 includes acquiring an audio sample. For example, the audio sample may be acquired using an electronic device, including an electronic device implementing the SOC 100 of FIG. 1 and / or an electronic device implementing the computing device architecture 700 of FIG. 7. In some cases, the electronic device may include one or more of a smartphone, a table computer, a wearable electronic device, a smart TV, a smart speaker, and the like. For example, the audio sample may be acquired using an electronic device including a speech recognition function for receiving a voice command from a user. For example, the audio sample may be acquired using one or more of the electronic devices (e.g., voice assistant devices) 308 having a keyword spotting system, as shown in FIG. 3. The one or more audio samples may include one or more of the example audio samples 302, 304, 306 of FIG. 3. In some cases, the audio sample may be acquired using one or more microphones and / or other audio sensors. The one or more microphones and / or audio sensors may be included in an electronic device, including an electronic device implementing SOC 100 of FIG. 1 and / or an electronic device implementing computing device architecture 700 of FIG.

[0153] In some examples, audio samples may be obtained using one or more sensors associated with the sensor processor 114 of FIG. 1. For example, the sensor processor 114 may be associated with or connected to one or more sensors for providing sensor input to the sensor processor 114. The one or more sensors associated with or connected to the sensor processor 114 may include one or more microphones and / or other audio sensors for obtaining audio samples. In some cases, the one or more sensors may include one or more microphones for receiving sounds (e.g., audio input), including sounds or audio input that may be used to perform keyword spotting (KWS) as described herein. In some cases, sounds or audio input received by the one or more microphones (and / or other sensors) may be digitized into data packets for analysis and / or transmission. The audio input may include ambient sounds in the vicinity of the computing device associated with the SoC 100 of FIG. 1 and / or may include speech from a user of the computing device associated with the SoC 100 of FIG. 1. In some cases, a computing device associated with SoC 100 of FIG. 1 may additionally or alternatively be communicatively coupled to one or more peripheral devices and / or may be configured to communicate with one or more remote computing devices or external resources using, for example, a wireless transceiver and a communications network, such as a cellular communications network.

[0154] In another example, the audio samples may be obtained using the input device 745 of the computing device architecture 700 of FIG. 7. For example, the input device 745 may be an input mechanism such as a microphone for speech. In some examples, the input device 745 of the computing device architecture 700 of FIG. 7 may be the same as or similar to one or more sensors described above with respect to the sensor processor 114 of FIG.

[0155]

[0103] At block 704, the process 700 includes generating a representation of a keyword, where the representation of the keyword is generated based on the audio sample. For example, the representation of the keyword can be a keyword embedding generated based on the audio sample. In some cases, the representation of the keyword can be the same as or similar to the keyword embedding shown in FIG. 5A. In some examples, the representation of the keyword can be the same as or similar to the keyword and speaker embedding 514 of FIG. 5B. In some examples, the representation of the keyword can be generated using a multi-task learning (MTL) machine learning network. For example, the representation of the keyword can be the same as or similar to the keyword embedding of FIG. 5A (e.g., as described above), and can be generated based on the input spectrogram 502 of the audio sample using the shared encoder 504 and the KWS Sub-Net 506, as also shown in FIG. 5A. In some cases, the representation of the keyword can be generated based on an audio sample obtained using one of the voice assistant devices 308 with keyword spotting system shown in FIG. 3. For example, a representation of a keyword may be generated based on one or more of the audio samples 302, 304, 306 of FIG.

[0156]

[0104] At block 706, the process 700 includes generating a speaker representation, where the speaker is associated with the keyword, and the speaker presentation is generated based on the audio sample. For example, the speaker representation can be a speaker embedding generated based on the audio sample. In some cases, the speaker representation can be the same as or similar to the speaker embedding shown in FIG. 5A. In some examples, the speaker representation can be the same as or similar to the keyword and speaker embedding 514 of FIG. 5B. In some examples, the speaker representation can be generated using a multi-task learning (MTL) machine learning network. For example, the speaker representation can be the same as or similar to the speaker embedding of FIG. 5A (e.g., as described above), and can be generated based on the input spectrogram 502 of the audio sample using the shared encoder 504 and the SV subnet 508, also as shown in FIG. 5A. In some cases, the speaker representation can be generated based on the audio sample obtained using one of the voice assistant devices 308 with keyword spotting system shown in FIG. 3. For example, a speaker representation may be generated based on one or more of the audio samples 302, 304, 306 of Figure 3. In some examples, the speaker representation may be associated with a speaker of the audio sample (e.g., a target user associated with audio sample 302 of Figure 3).

[0157] At block 708, the process 700 includes determining a first similarity score based on the reference expression and one or more of the keyword expression and the speaker expression, where the reference expression is associated with one or more of the keyword and the speaker. For example, the reference expression can be the same as or similar to the task-specific embedding 518 of FIG. 5B. In some cases, the reference expression can be generated using a task-specific machine learning module, such as the task-specific machine learning module 516 of FIG. 5B. In some cases, the first similarity score can be generated using a task-adaptive machine learning network.

[0158]

[0106] Determining the first similarity score may include generating a keyword similarity score between the representation of the keyword and the reference representation of the target keyword. The target keyword may be a target keyword, such as a predefined wake-up command or an activation command associated with a voice-activated device implementing keyword spotting. Determining the first similarity score may further include generating a speaker similarity score between the representation of the speaker and the reference representation of the speaker. The first similarity score may be determined as a combined similarity score generated based at least in part on the keyword similarity score and the speaker similarity score. For example, one or more (or both) of the keyword similarity score and the speaker similarity score may be a cosine similarity score. For example, the cosine similarity score used as the keyword similarity score may be the same as or similar to the cosine similarity score associated with the cosine classifier 510 for keyword embedding of FIG. 5A. In some cases, the cosine similarity score used as the speaker similarity score may be the same as or similar to the cosine similarity score associated with the cosine classifier 512 for speaker embedding of FIG. 5A.

[0159]

[0107] In some examples, the combined similarity score may be generated using a score combination function. For example, the score combination function may be a linear combination function between the keyword similarity scores and the speaker similarity scores. The linear combination function may include at least a first adjustable weighting parameter. In some cases, the score combination function may be associated with one or more neural networks, and the first adjustable weighting parameter may be adjusted to train to minimize a keyword spotting (KWS) false rejection rate (FRR) associated with the one or more neural networks. In some examples, the first adjustable weighting parameter may be set to a first value to implement target user biased keyword spotting (TB-KWS). In another example, the first adjustable weighting parameter may be set to a second value to implement target user dedicated keyword spotting (TO-KWS), the first value being greater than the second value.

[0160] In some cases, determining the first similarity score between the reference expression and one or more of the keyword representations and the speaker representations includes generating a task-specific embedding using the keyword representations and the speaker representations. For example, the task-specific embedding can be the same as or similar to the task-specific embedding 518 of FIG. 5B. The first similarity score can be generated as a task-specific embedding using the keyword and speaker embedding 514 of FIG. 5B (for example, the keyword representations and the speaker representations can be the same as or similar to the keyword and speaker embedding 514 of FIG. 5B).

[0161] In some cases, the task-specific embeddings may be generated based on the output of a first neural network, where the output of the first neural network includes a target-user biased keyword spotting (TB-KWS) task-specific embedding. For example, the task-specific embeddings may be generated using the task-specific neural network module 516 of FIG. 5B, where the output of the task-specific neural network module 516 (e.g., task-specific embeddings 518) may include a TB-KWS task-specific embedding.

[0162] In some examples, determining the first similarity score further includes determining the first similarity score as a cosine similarity score between the task-specific embedding and the reference representation, where the reference representation is a learnable weight for keyword classification. In some cases, the task-specific embedding can be generated based on the output of a second neural network, where the output of the second neural network includes a target user dedicated keyword spotting (TO-KWS) task-specific embedding. For example, the task-specific module 516 of FIG. 5B can generate as an output a task-specific embedding 518 including a TO-KWS task-specific embedding.

[0163]

[0111] At block 710, the process 700 includes generating a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, the KWS output accepting or rejecting the audio sample as containing the target keyword. In some cases, the KWS output can be generated using a task-adaptive machine learning network. The task-adaptive machine learning network used to generate the KWS output can be the same as the task-adaptive machine learning network used to generate the first similarity score.

[0164] In some examples, the processes described herein (e.g., process 600, and / or any other processes described herein) may be performed by a computing device, apparatus, or system. In one example, process 600 may be performed by a computing device or system having the computing device architecture 700 of FIG. 7. The computing device, apparatus, or system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an autonomous vehicle or autonomous vehicle computing device, a robotic device, a laptop computer, a smart television, a camera, and / or any other computing device having resource capabilities to perform the processes described herein, including process 600 and / or any other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0165]

[0113] Components of a computing device may be implemented with circuits. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform various operations described herein.

[0166]

[0114] The process 600 is illustrated as a logical flow diagram, whose operations represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0167] Additionally, process 600 and / or any other process described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes collectively, by hardware, or a combination thereof on one or more processors. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0168]

[0116] Figure 7 illustrates an exemplary computing device architecture 700 of an exemplary computing device that can implement various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other device. The components of the computing device architecture 700 are shown to be in electrical communication with each other using a connection 705 such as a bus. The exemplary computing device architecture 700 includes a processing unit (CPU or processor) 710 and computing device connections 705 that couple various computing device components, including a computing device memory 715 such as a read only memory (ROM) 720 and a random-access memory (RAM) 725, to the processor 710.

[0169]

[0117] The computing device architecture 700 may include a cache of high-speed memory directly connected to, adjacent to, or integrated as part of the processor 710. The computing device architecture 700 may copy data from the memory 715 and / or storage device 730 to the cache 712 for quick access by the processor 710. In this manner, the cache may provide a performance boost that avoids delays in the processor 710 while waiting for data. These and other engines may control or be configured to control the processor 710 to perform various actions. Other computing device memories 715 may also be available for use. The memory 715 may include multiple different types of memory with different performance characteristics. The processor 710 may include any general-purpose processor, hardware or software services such as service 1 732, service 2 734, and service 3 736 stored in the storage device 730 configured to control the processor 710, and dedicated processors with software instructions built into the processor design. The processor 710 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. A multi-core processor may be symmetric or asymmetric.

[0170] To enable user interaction with the computing device architecture 700, the input device 745 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. The output device 735 can also be one or more of some output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 700. The communication interface 740 can generally govern and manage user input and computing device output. There is no constraint to operate on any particular hardware configuration, and therefore the basic functions herein can be easily replaced with improved hardware or firmware configurations as they are developed.

[0171]

[0119] The storage device 730 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAMs) 725, a read-only memory (ROM) 720, and hybrids thereof. The storage device 730 may include services 732, 734, 736 for controlling the processor 710. Other hardware or software modules or engines are also contemplated. The storage device 730 may be connected to a computing device connection 705. In one aspect, a hardware module performing a particular function may include software components stored in a computer-readable medium in association with necessary hardware components, such as the processor 710, the connection 705, the output device 735, etc., to perform the function.

[0172] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, a smartphone, a tablet, a laptop computer, a vehicle, a drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0173]

[0121] The term "device" is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system, etc.). A device as used herein may be any electronic device having one or more parts that can implement at least some parts of the present disclosure. The following description and examples use the term "device" to describe various aspects of the present disclosure, but the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or a specific aspect. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. The following description and examples use the term "system" to describe various aspects of the present disclosure, but the term "system" is not limited to a specific configuration, type, or number of objects.

[0174]

[0122] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that the aspects can be practiced without these specific details. For ease of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method embodied in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the aspects in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the aspects.

[0175]

[0123] Each aspect may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0176]

[0124] The processes and methods according to the examples described above can be implemented using computer-executable instructions stored in or otherwise available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary or intermediate format instructions such as assembly language, firmware, source code, etc.

[0177]

[0125] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, store, or convey instructions and / or data. Computer-readable media may include non-transitory media on which data may be stored and that do not include carrier waves and / or transitory electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as flash memory, memories or memory devices, magnetic or optical disks, flash memories, USB devices provided with non-volatile memory, networked storage devices, compact disks (CDs) or digital versatile disks (DVDs), among others, or any suitable combination thereof. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, an engine, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0178] In some aspects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when mentioned, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0179]

[0127] A device implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) that perform the necessary tasks may be stored in a computer-readable medium or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-footprint personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-in cards. Such functions may also be implemented on a circuit board among different chips, or on different processes running in a single device, as further examples.

[0180]

[0128] The instructions, media for propagating such instructions, computing resources for executing such instructions, and other structures supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0181]

[0129] In the foregoing description, the aspects of the present application have been described with reference to certain aspects thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary aspects of the present application have been described in detail herein, it should be understood that the inventive concept may be variously implemented and utilized in other ways, and the appended claims are intended to be construed to include such variations, except as limited by conventional techniques. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the aspects may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative aspects, the methods may be performed in an order different from that described.

[0182]

[0130] Those skilled in the art will understand that the symbols or terminology used in this specification such as less than ("<") and greater than (">") can be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this specification.

[0183]

[0131] When a component is described as being "configured to" perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0184]

[0132] The phrase "coupled to" refers to any component that is physically connected to another component, either directly or indirectly, and / or any component that is in communication with another component, either directly or indirectly (e.g., connected to the other component via a wired or wireless connection and / or other suitable communications interface).

[0185]

[0133] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, a claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, a claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include unrecited items in the set of A and B.

[0186]

[0134] The various exemplary logic blocks, modules, engines, circuits, and algorithm steps described in relation to the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this compatibility of hardware and software, the various exemplary components, blocks, modules, engines, circuits, and steps are described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such a determination of the implementation form should not be interpreted as a cause for departing from the scope of this application.

[0187]

[0135] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random-access memory (SDRAM), read-only memory (ROM), non-volatile random-access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, which carries or communicates program code in the form of instructions or data structures and which can be accessed, read and / or executed by a computer.

[0188]

[0136] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0189]

[0137] Exemplary aspects of the present disclosure include the following.

[0190]

[0138] Aspect 1. A method for processing one or more audio samples, the method comprising: obtaining the audio sample; generating a representation of a keyword, where the representation of the keyword is generated based on the audio sample; generating a representation of a speaker, where the speaker is associated with the keyword, and the speaker representation is generated based on the audio sample; determining a first similarity score based on a reference representation associated with one or more of the keywords and speakers and one or more of the representation of the keyword and the speaker representation; and generating a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, the KWS output accepting or rejecting the audio sample as containing the target keyword.

[0191]

[0139] Aspect 2. The method of aspect 1, wherein the representation of the keyword is a keyword embedding generated based on the audio sample, and the representation of the speaker is a speaker embedding generated based on the audio sample.

[0192]

[0140] Aspect 3. The method of aspect 1 or 2, wherein one or more of the keyword representations or speaker representations are generated using a multi-task learning (MTL) machine learning network.

[0193]

[0141] Aspect 4. A method according to any one of aspects 1 to 3, wherein the first similarity score and the KWS output are generated using a task-adaptive machine learning network.

[0194]

[0142] Aspect 5. A method according to any one of aspects 1 to 4, wherein determining the first similarity score includes generating a keyword similarity score between the representation of the keyword and a reference representation of the target keyword, generating a speaker similarity score between the representation of the speaker and the reference representation of the speaker, and determining the first similarity score as a combined similarity score generated based at least in part on the keyword similarity score and the speaker similarity score.

[0195]

[0143] Aspect 6. The method of aspect 5, wherein one or more of the keyword similarity scores or the speaker similarity scores is a cosine similarity score.

[0196]

[0144] Aspect 7. The method of aspect 5 or 6, wherein the combined similarity score is generated using a score combination function.

[0197]

[0145] Aspect 8. The method of aspect 7, wherein the score combination function is a linear combination function between the keyword similarity scores and the speaker similarity scores, the linear combination function including at least a first adjustable weighting parameter.

[0198]

[0146] Aspect 9. The method described in aspect 8, wherein the score combination function is associated with one or more neural networks and trained to adjust a first adjustable weighting parameter to minimize a keyword spotting (KWS) false rejection rate (FRR) associated with the one or more neural networks.

[0199]

[0147] Aspect 10. The method of aspect 8 or 9, further comprising setting a first adjustable weighting parameter to a first value to perform target user biased keyword spotting (TB-KWS) and setting the first adjustable weighting parameter to a second value to perform target user specific keyword spotting (TO-KWS), wherein the first value is greater than the second value.

[0200]

[0148] Aspect 11. A method according to any one of aspects 1 to 10, wherein determining the first similarity score includes generating a task-specific embedding using a keyword representation and a speaker representation, and determining the first similarity score as a cosine similarity score between the task-specific embedding and a reference representation, wherein the reference representation is a learnable weight for keyword classification.

[0201]

[0149] Aspect 12. The method of aspect 11, further comprising generating a task-specific embedding based on the output of the first neural network, the output of the first neural network comprising a target-user biased keyword spotting (TB-KWS) task-specific embedding.

[0202]

[0150] Aspect 13. The method of aspect 12, further comprising generating a task-specific embedding based on the output of a second neural network, the output of the second neural network comprising a target user-specific keyword spotting (TO-KWS) task-specific embedding.

[0203]

[0151] Aspect 14. An apparatus for processing one or more audio samples, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor obtains the audio samples, generates a representation of a keyword, where the representation of the keyword is generated based on the audio sample, generates a representation of a speaker, where the speaker is associated with the keyword, where the speaker representation is generated based on the audio sample, determines a first similarity score based on a reference representation associated with one or more of the keywords and speakers and one or more of the representation of the keyword and the speaker representation, and generates a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, where the KWS output accepts or rejects the audio sample as containing the target keyword.

[0204]

[0152] Aspect 15. The apparatus of aspect 14, wherein the representation of the keyword is a keyword embedding generated based on the audio sample, and the representation of the speaker is a speaker embedding generated based on the audio sample.

[0205]

[0153] Aspect 16. An apparatus described in aspect 14 or 15, wherein one or more of the keyword representations or speaker representations are generated using a multi-task learning (MTL) machine learning network.

[0206]

[0154] Aspect 17. An apparatus described in any one of aspects 14 to 16, wherein the first similarity score and the KWS output are generated using a task-adaptive machine learning network.

[0207]

[0155] Aspect 18. An apparatus as described in any one of aspects 14 to 17, wherein to determine the first similarity score, at least one processor is configured to generate a keyword similarity score between the representation of the keyword and a reference representation of the target keyword, generate a speaker similarity score between the representation of the speaker and the reference representation of the speaker, and determine the first similarity score as a combined similarity score generated at least in part based on the keyword similarity score and the speaker similarity score.

[0208]

[0156] Aspect 19. The apparatus of aspect 18, wherein one or more of the keyword similarity scores or the speaker similarity scores is a cosine similarity score.

[0209]

[0157] Aspect 20. An apparatus according to aspect 18 or 19, wherein the combined similarity score is generated using a score combination function.

[0210]

[0158] Aspect 21. The apparatus of aspect 20, wherein the score combination function is a linear combination function between the keyword similarity scores and the speaker similarity scores, the linear combination function including at least a first adjustable weighting parameter.

[0211]

[0159] Aspect 22. The apparatus described in aspect 21, wherein the score combination function is associated with one or more neural networks and trained to adjust a first adjustable weighting parameter to minimize a keyword spotting (KWS) false rejection rate (FRR) associated with the one or more neural networks.

[0212]

[0160] Aspect 23. The device described in Aspect 21 or 22, wherein at least one processor is further configured to set a first adjustable weighting parameter to a first value to perform target-user biased keyword spotting (TB-KWS) and to set the first adjustable weighting parameter to a second value to perform target-user specific keyword spotting (TO-KWS), the first value being greater than the second value.

[0213]

[0161] Aspect 24. An apparatus described in any one of aspects 14 to 23, wherein to determine the first similarity score, at least one processor is configured to generate a task-specific embedding using the keyword representation and the speaker representation, and to determine the first similarity score as a cosine similarity score between the task-specific embedding and a reference representation, the reference representation being a learnable weight for keyword classification.

[0214]

[0162] Aspect 25. The apparatus described in Aspect 24, wherein at least one processor is further configured to generate a task-specific embedding based on the output of the first neural network, the output of the first neural network including a target-user biased keyword spotting (TB-KWS) task-specific embedding.

[0215]

[0163] Aspect 26. The apparatus described in aspect 25, wherein at least one processor is further configured to generate a task-specific embedding based on the output of the second neural network, the output of the second neural network including a target user dedicated keyword spotting (TO-KWS) task-specific embedding.

[0216]

[0164] Aspect 27. A computer-readable storage medium storing instructions which, when executed by one or more processors, cause at least one processor to obtain an audio sample, generate a representation of a keyword, where the representation of the keyword is generated based on the audio sample, generate a representation of a speaker, where the speaker is associated with the keyword, where the speaker representation is generated based on the audio sample, determine a first similarity score based on a reference representation associated with one or more of the keywords and speakers and one or more of the representation of the keyword and the speaker representation, and generate a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, where the KWS output accepts or rejects the audio sample as containing the target keyword.

[0217]

[0165] Aspect 28. A computer-readable storage medium as described in aspect 27, wherein the keyword representation is a keyword embedding generated based on the audio sample, and the speaker representation is a speaker embedding generated based on the audio sample.

[0218]

[0166] Aspect 29. A computer-readable storage medium as described in aspect 27 or 28, wherein one or more of the keyword representations or speaker representations are generated using a multi-task learning (MTL) machine learning network.

[0219]

[0167] Aspect 30. A computer-readable storage medium according to any one of aspects 27 to 29, wherein the first similarity score and the KWS output are generated using a task-adaptive machine learning network.

[0220]

[0168] Aspect 31. The apparatus of any one of aspects 27 to 30, wherein to determine the first similarity score, the instructions further configure at least one processor to generate a keyword similarity score between the representation of the keyword and a reference representation of the target keyword, generate a speaker similarity score between the representation of the speaker and the reference representation of the speaker, and determine the first similarity score as a combined similarity score generated at least in part based on the keyword similarity score and the speaker similarity score.

[0221]

[0169] Aspect 32. The computer-readable storage medium of aspect 31, wherein one or more of the keyword similarity scores or the speaker similarity scores is a cosine similarity score.

[0222]

[0170] Aspect 33. A computer-readable storage medium according to aspect 31 or 32, wherein the combined similarity score is generated using a score combination function.

[0223]

[0171] Aspect 34. A computer-readable storage medium as described in aspect 33, wherein the score combination function is a linear combination function between the keyword similarity scores and the speaker similarity scores, the linear combination function including at least a first adjustable weighting parameter.

[0224]

[0172] Aspect 35. A computer-readable storage medium as described in Aspect 34, wherein the score combination function is associated with one or more neural networks and trained to adjust a first adjustable weighting parameter to minimize a keyword spotting (KWS) false rejection rate (FRR) associated with the one or more neural networks.

[0225]

[0173] Aspect 36. A computer-readable storage medium as described in aspect 34 or 35, wherein the instructions further cause at least one processor to set a first adjustable weighting parameter to a first value to perform target-user biased keyword spotting (TB-KWS) and to set the first adjustable weighting parameter to a second value to perform target-user specific keyword spotting (TO-KWS), the first value being greater than the second value.

[0226]

[0174] Aspect 37. A computer-readable storage medium as described in any one of aspects 27 to 36, wherein to determine the first similarity score, the instructions cause at least one processor to generate a task-specific embedding using a representation of the keyword and a representation of the speaker, and determine the first similarity score as a cosine similarity score between the task-specific embedding and a reference representation, the reference representation being a learnable weight for keyword classification.

[0227]

[0175] Aspect 38. A computer-readable storage medium as described in aspect 37, wherein the instructions further cause at least one processor to generate a task-specific embedding based on the output of the first neural network, the output of the first neural network including a target-user biased keyword spotting (TB-KWS) task-specific embedding.

[0228]

[0176] Aspect 39. A computer-readable storage medium as described in aspect 38, wherein the instructions further cause at least one processor to generate a task-specific embedding based on the output of the second neural network, the output of the second neural network including a target user dedicated keyword spotting (TO-KWS) task-specific embedding.

[0229]

[0177] Aspect 40. A non-transitory computer-readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to perform the operations described in any one of aspects 1 to 13.

[0230]

[0178] Aspect 41. A non-transitory computer-readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to perform the operations described in any one of aspects 14 to 26.

[0231]

[0179] Aspect 42. A non-transitory computer-readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to perform the operations described in any one of aspects 27 to 39.

[0232]

[0180] Aspect 43. An apparatus for processing one or more audio samples, the apparatus comprising one or more means for performing the operations recited in any one of aspects 1-13.

[0233]

[0181] Aspect 44. An apparatus for processing one or more audio samples, the apparatus comprising one or more means for performing the operations recited in any one of aspects 14 to 26.

[0234]

[0182] Aspect 45. An apparatus for processing one or more audio samples, the apparatus comprising one or more means for performing the operations recited in any one of aspects 27 to 39.

Claims

1. 1. A method for processing one or more audio samples, comprising: Obtaining an audio sample; generating a representation of a keyword, the representation of the keyword being generated based on the audio sample; generating a speaker representation, the speaker associated with the keyword, the representation of the speaker generated based on the audio sample; i) generating a keyword similarity score between the representation of the keyword and a reference representation of a target keyword; generating a speaker similarity score between the representation of the speaker and a reference representation of the speaker; determining a first similarity score as a combined similarity score generated based at least in part on the keyword similarity score and the speaker similarity score; or ii) generating task-specific embeddings using the representations of the keywords and the representations of the speakers; determining the first similarity score as a cosine similarity score between the task-specific embedding and the reference representation, the reference representation being a learnable weight for keyword classification; determining a first similarity score by either: generating a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, the KWS output accepting or rejecting the audio sample as containing a target keyword.

2. the representation of the keyword is a keyword embedding generated based on the audio sample; The method of claim 1 , wherein the representation of the speaker is a speaker embedding generated based on the audio samples.

3. The method of claim 1 , wherein one or more of the representations of the keywords or the representations of the speakers are generated using a multi-task learning (MTL) machine learning network.

4. The method of claim 1 , wherein the first similarity score and the KWS output are generated using a task-adaptive machine learning network.

5. The method of claim 1 , wherein the combined similarity score is generated using a score combination function.

6. the score combination function is a linear combination function between the keyword similarity scores and the speaker similarity scores; The method of claim 5 , wherein the linear combination function includes at least a first adjustable weighting parameter.

7. 7. The method of claim 6, wherein the score combination function is associated with one or more neural networks and trained to adjust the first adjustable weighting parameters to minimize a KWS false rejection rate (FRR) associated with the one or more neural networks.

8. setting the first adjustable weighting parameter to a first value to perform target-user-biased keyword spotting (TB-KWS); 8. The method of claim 7, further comprising: setting the first adjustable weighting parameter to a second value to perform target-user-specific keyword spotting (TO-KWS), wherein the first value is greater than the second value.

9. generating the task-specific embedding based on an output of a first neural network, wherein the output of the first neural network comprises a target-user-biased keyword spotting (TB-KWS) task-specific embedding; or 10. The method of claim 1, further comprising generating the task-specific embedding based on an output of a second neural network, the output of the second neural network comprising a target-user-specific keyword spotting (TO-KWS) task-specific embedding.

10. 1. An apparatus for processing one or more audio samples, comprising: at least one memory; at least one processor coupled to the at least one memory; wherein the at least one processor: Get the audio samples, generating a representation of a keyword, the representation of the keyword being generated based on the audio sample; generating a representation of a speaker, the speaker being associated with the keyword, the representation of the speaker being generated based on the audio sample; i) generating a keyword similarity score between the representation of the keyword and a reference representation of a target keyword; generating a speaker similarity score between the representation of the speaker and a reference representation of the speaker; determining a first similarity score as a combined similarity score generated based at least in part on the keyword similarity score and the speaker similarity score; or ii) generating task-specific embeddings using the representations of the keywords and the representations of the speakers; determining the first similarity score as a cosine similarity score between the task-specific embedding and the reference representation, the reference representation being a learnable weight for keyword classification; determining a first similarity score by either The apparatus is configured to generate a keyword spotting (KWS) output based on analyzing the first similarity score against at least a first threshold, the KWS output accepting or rejecting the audio sample as containing a target keyword.

11. The method of claim 1 or the apparatus of claim 10, wherein one or more of the keyword similarity scores or the speaker similarity scores is a cosine similarity score.

12. The at least one processor is configured to generate the combined similarity score using a score combination function; the score combination function is a linear combination function between the keyword similarity scores and the speaker similarity scores; The apparatus of claim 10 , wherein the linear combination function includes at least a first adjustable weighting parameter.

13. the score combination function is associated with one or more neural networks and trained to adjust the first adjustable weighting parameters to minimize a KWS false rejection rate (FRR) associated with the one or more neural networks; or The at least one processor setting the first adjustable weighting parameter to a first value to perform target-user-biased keyword spotting (TB-KWS); 13. The apparatus of claim 12, further configured to set the first adjustable weighting parameter to a second value, the first value being greater than the second value, to perform target-user-specific keyword spotting (TO-KWS).

14. The at least one processor further configured to generate the task-specific embedding based on an output of a first neural network, the output of the first neural network comprising a target-user biased keyword spotting (TB-KWS) task-specific embedding; or 11. The apparatus of claim 10, further configured to generate the task-specific embedding based on an output of a second neural network, the output of the second neural network comprising a target-user-specific keyword spotting (TO-KWS) task-specific embedding.

Images