Dummy Prototype Networks for Few-Shot Open Set Keyword Spotting

Abstract

Systems and techniques are provided for processing audio data. For example, a dummy prototype network can be used to perform few-shot open-set keyword spotting (FSOS-KWS). The process can include determining one or more prototype representations based on a plurality of support samples associated with one or more classes. Each prototype representation can be associated with one of the classes. The dummy prototype representation can be determined in the same learned metric space as the prototype representation. One or more distance metrics can be determined for each query sample of the one or more query samples. The distance metric can be based on the prototype representation and the dummy prototype representation. Each query sample can be classified based on the distance metric. Each query sample can be classified into one of the classes associated with the prototype representation or into an open-set class associated with the dummy prototype representation.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 321,677, filed March 19, 2022, the entire contents of which are incorporated by reference into this specification for all purposes.

[0002]

[0002] The present disclosure generally relates to audio signal processing. For example, aspects of the present disclosure relate to systems and techniques for performing open-set recognition (OSR) keyword spotting (KWS) using one or more machine learning systems (e.g., including one or more neural network-based models). [Background technology]

[0003]

[0003] Electronic devices such as smartphones, tablet computers, wearable electronic devices, and smart TVs are becoming increasingly popular among consumers. These devices can provide voice and / or data communication capabilities over wireless or wired networks. In addition, such electronic devices can include other features that provide various functions designed to enhance user convenience. The electronic device can include a speech recognition function for receiving a voice command from a user. Such a function enables the electronic device to perform a function associated with the voice command (e.g., a keyword) when the voice command from the user is received and recognized. For example, the electronic device can activate a voice assistant application, play an audio file, or take a photo in response to a voice command from the user.

[0004]

[0004] In electronic devices with speech recognition capabilities, manufacturers or carriers can often equip the devices with predefined keywords and associated sound models that can be used in detecting keywords in input sounds. Some electronic devices can also allow users to specify keywords as voice commands. For example, the electronic device can receive multiple utterances of keywords from a user and generate a keyword model for the specified keywords from the utterances. The detection performance of the keyword model can be related to the number of utterances for which the keyword model is generated. The detection performance of the keyword model can improve as the number of utterances increases. For example, a manufacturer can provide an electronic device with keyword models generated from thousands or more utterances.

[0005]

[0005] However, in some electronic devices, the number of keyword utterances received from a user is relatively small (e.g., less than 5). Therefore, a keyword model generated from such a limited number of utterances may not provide sufficient detection performance. However, receiving a significant number of utterances from a user to generate a keyword model that can provide sufficient detection performance may be time-consuming and inconvenient for the user. Summary of the Invention

[0006] In some examples, systems and techniques for few-shot open-set recognition (FSOSR) are described. In some examples, the systems and techniques can be used to perform keyword spotting with recognition and rejection of open set queries that do not belong to known or seen classes. In some aspects, the systems and techniques can utilize dummy prototype networks to perform few-shot open-set keyword spotting (FSOS-KWS).

[0007]

[0007] According to at least one example, a method for processing one or more data samples is provided, the method including: determining one or more prototype representations based on a plurality of support samples associated with the one or more classes, where each prototype representation is associated with one of the one or more classes; determining a dummy prototype representation, where the one or more prototype representations and the dummy prototype representation are determined in the same learned metric space; determining one or more distance metrics for each query sample of the one or more query samples, where the one or more distance metrics are determined based on the one or more prototype representations and the dummy prototype representation; and classifying each query sample based on the one or more distance metrics, where each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the dummy prototype representation.

[0008]

[0008] In another example, an apparatus is provided for processing one or more data samples, the apparatus including a memory (e.g., configured to store data such as audio data) and one or more processors coupled to the memory (e.g., implemented in circuitry). The one or more processors may be configured to: determine, based on a plurality of support samples associated with the one or more classes, one or more prototype representations, each prototype representation associated with one of the one or more classes; determine a dummy prototype representation, where the one or more prototype representations and the dummy prototype representation are determined in a same learned metric space; determine, for each query sample of the one or more query samples, one or more distance metrics, where the one or more distance metrics are determined based on the one or more prototype representations and the dummy prototype representation; and classify each query sample based on the one or more distance metrics, where each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the dummy prototype representation.

[0009]

[0009] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, can cause the one or more processors to determine, based on a plurality of support samples associated with the one or more classes, one or more prototype representations, where each prototype representation is associated with one of the one or more classes; determine a dummy prototype representation, where the one or more prototype representations and the dummy prototype representation are determined in a same learned metric space; determine, for each query sample of the one or more query samples, one or more distance metrics, where the one or more distance metrics are determined based on the one or more prototype representations and the dummy prototype representation; and classify each query sample based on the one or more distance metrics, where each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the dummy prototype representation.

[0010]

[0010] In another example, an apparatus is provided for processing one or more data samples, the apparatus including: determining one or more prototype representations based on a plurality of support samples associated with one or more classes, each prototype representation being associated with one of the one or more classes; determining a dummy prototype representation, the one or more prototype representations and the dummy prototype representation being determined in the same learned metric space; determining one or more distance metrics for each query sample of the one or more query samples, the one or more distance metrics being determined based on the one or more prototype representations and the dummy prototype representation; and classifying each query sample based on the one or more distance metrics, each query sample being classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the dummy prototype representation.

[0011]

[0011] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0012]

[0012] The above, together with other objects and advantages associated with the aspects disclosed herein, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief description of the drawings]

[0013]

[0013] The accompanying drawings are presented to aid in the description of various aspects of the present disclosure and are provided only to illustrate the aspects and not to limit the aspects. [Figure 1]

[0014] FIG. 1 illustrates an exemplary implementation of a System-on-a-Chip (SoC), in accordance with some examples. [Figure 2A]

[0015] 1 shows an example of a fully connected neural network with some examples. [Figure 2B] 1 shows an example of a fully connected neural network with some examples. [Diagram 3]

[0016] FIG. 1 illustrates an example of a few-shot learning (FSL) scenario, with some examples. [Figure 4A]

[0017] FIG. 1 illustrates another example of a few-shot learning (FSL) scenario, with some examples. [Figure 4B]

[0018] FIG. 2 illustrates a first example episode of a few-shot open set recognition (FSOSR) episode, in accordance with some examples. [Figure 4C]

[0019] FIG. 1 illustrates a second example episode of a few-shot open set recognition (FSOSR) episode, in accordance with some examples. [Diagram 5]

[0020] FIG. 1 illustrates an example architecture of a dummy prototype network, in accordance with some examples. [Figure 6]

[0021] FIG. 1 is a flow diagram illustrating an example of a process for processing one or more data samples (eg, of an audio signal) to perform keyword spotting (KWS). [Figure 7]

[0022] FIG. 1 is a block diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014]

[0023] Some aspects and examples of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the aspects and examples of the present application. However, it will be apparent that the various aspects and examples can be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0015]

[0024] The following description provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an effective description for implementing the examples and embodiments of the present disclosure. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0016]

[0025] Voice activity detection can refer to the detection of a human voice by a computing device to perform some function. For example, a computing device can use voice activity detection, such as by recognizing certain keywords, to wake or activate the device, interrogate the device, and / or cause the device to perform various other functions. Voice activity detection can also be used in more complex functions, such as far-field voice detection (e.g., from mobile devices positioned across a room), user identity verification (e.g., by voiceprint), voice detection during other audio output (e.g., detecting a voice command while music is playing on a device or detecting an interrupt command while a smart assistant is speaking), and / or voice interaction in complex noisy environments, such as in a moving vehicle.

[0017]

[0026] Keyword Spotting (KWS) is the task of detecting one or more keywords in an audio signal (e.g., an audio signal containing human speech or spoken words). For example, KWS can be used to distinguish activation phrases or specific commands from other speech and noise in the audio signal. In some cases, KWS systems can be targeted to or utilized by edge devices such as mobile phones and smart speakers. Detected keywords can include single words, compound words, phrases containing multiple words, etc. In some cases, KWS can be implemented based on a set of predefined keywords and / or a set of user-defined keywords. In some cases, the user-defined keywords can include one or more adaptations, adjustments, etc. determined based on certain characteristics of a given user's voice or speech.

[0018]

[0027] The audio signal may be a streaming audio signal, and in some examples, the KWS may be implemented on the streaming audio signal in real time. The streaming audio signal may be recorded by or obtained from a microphone associated with the computing device. The KWS may be implemented locally or remotely. For example, the KWS may be implemented locally using one or more processors of the same computing device that collects or obtains the streaming audio signal. In some examples, the KWS may be implemented remotely by transmitting the streaming audio signal (or a representation thereof) from the local computing device to a remote computing device (e.g., the local computing device records the audio signal but offloads the KWS processing tasks to the remote computing device). Implementing the KWS locally may result in lower overall latency or computation time, but may result in reduced accuracy. Implementing the KWS remotely may result in higher latency, but may result in increased accuracy.

[0019]

[0028] For example, a local computing device (e.g., a smartphone) often has less computational power than a remote computing device (e.g., a cloud computing system) and may therefore generate KWS results with lower accuracy or overall performance, especially when subject to time constraints associated with providing KWS results in real-time or near real-time. For example, a local computing device may implement a KWS detection model with lower complexity than one implemented on a remote computing device to provide real-time KWS results. Lower accuracy KWS results may include false positives (e.g., identifying a keyword that does not actually exist), false negatives (e.g., failing to identify a keyword that does exist), and classification errors (e.g., identifying a first keyword as some other keyword).

[0020]

[0029] However, implementing a KWS remotely can introduce communication latencies that can offset the increased accuracy associated with a remote KWS. For example, a remote KWS can introduce latencies along the communication path from the local computing device to the remote computing device (e.g., the time to transmit a streaming audio signal or a representation thereof to the remote computing device) and along the return communication path from the remote computing device to the local computing device (e.g., the time to transmit the KWS results back from the remote computing device to the local computing device).

[0021]

[0030] As mentioned above, in some examples, KWSs are often implemented in real-time (or near real-time) to enable user interaction with one or more computing devices. The lag between the time a user speaks a keyword (e.g., an activation phrase or a specific command) and the time the computing device provides a corresponding response or action can be a significant factor in the user's willingness to utilize the spoken command. In some cases, a delay of several seconds can frustrate or otherwise discourage a user from using verbal commands. Thus, there is a need for improved KWS performance in local and / or remote KWS implementations, since both local and remote KWS implementations are often time-bound processes.

[0022]

[0031] In some approaches to KWS, one or more machine learning networks are used to detect keywords in an audio signal based on a set of predefined keyword classifications. For example, a KWS can be implemented using one or more neural networks trained over a set of predefined keyword classifications. The resulting neural network can achieve accurate performance when evaluated against a closed set of keywords seen in training, but is often limited to detecting only keywords seen in training (e.g., keywords that correspond to the keyword classifications in the training dataset).

[0023]

[0032] In some cases, a few-shot KWS can be used to detect or classify keywords that were not originally seen during training. In an exemplary FSL-based approach to KWS, a neural network classifier is trained on a training set that includes labeled samples representing various predefined classes or categories of keywords. During training, the neural network classifier absorbs knowledge from the labeled samples of the training set. For example, the neural network classifier can learn an embedding or embedding space for relating training samples to their corresponding labels. The trained neural network classifier can then be adapted or generalized (e.g., in inference) to the task of evaluating new classes (e.g., unseen classes) that were not seen in the training set.

[0024]

[0033] For example, given only a small number of support examples for each new class, the FSL-based neural network classifier can predict the class of an unlabeled point (e.g., a query set) using an attention mechanism on the learned embeddings of a set of labeled support examples (e.g., a support set). In a query-by-example process, a trained FSL-based neural network classifier can receive a support set including M support examples for each of N first-time classes and a query set including one or more query samples. The neural network classifier can determine a prototype representation of each first-time class N (e.g., using the M support examples associated with each first-time class N). Each unlabeled query sample can be classified into one of the previously unseen classes N based on the calculated distance between the query sample and each prototype representation. In some cases, this inference process may be referred to as an N-ary M-shot episode, and the goal of the FSL-based neural network classifier is to correctly classify the query set into N classes that were not seen during training but are known using the M support examples.

[0025]

[0034] In real-world scenarios in which a KWS may be utilized, the input audio signal may, of course, contain unexpected new classes of utterances, words, phrases, etc. for which no supporting examples have been provided. Unexpected new classes may be classes for which no supporting examples were included in the initial training set or support set. These unexpected samples and / or classes may be referred to as an "open set" (e.g., as opposed to a closed set of classes from the initial training set and support set).

[0026]

[0035] Neural network classifiers may exhibit overconfidence and misclassify first-time open set samples into one of the FSL classes. In some examples, the accuracy and / or performance of a KWS system may be improved by rejecting open set utterances rather than classifying them into one of the predefined keyword classifications (e.g., of a closed set). For example, Open-Set Recognition (OSR) may be used to detect first-time open set classes and then process and / or reject query samples that belong to the detected open set classes.

[0027]

[0036] Systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to as "systems and techniques") are described herein for performing few-shot open set recognition (FSOSR) and / or few-shot open set keyword spotting (FSOS-KWS) using a machine learning classifier. In some examples, the machine learning classifier may be a neural network classifier. The few-shot neural network classifier may be used to perform improved keyword spotting (KWS) using OSR by predicting episode-specific (e.g., episode-known) dummy classes based on the supporting examples in each episode. In some examples, the few-shot neural network classifier may classify the predicted dummy classes as open sets for rejection from downstream or subsequent KWS processes, as described in more detail below. In some aspects, the neural network described herein may be referred to and / or included in a dummy prototypical network (D-ProtoNet).

[0028]

[0037] In some examples, the systems and techniques can generate a set of prototype representations based on a support set provided as an input during inference. The support set can include a plurality of support examples associated with known keyword classes. For example, the support set can include example utterances labeled with keyword classes corresponding to the utterance. The machine learning encoder can generate embedded support examples from the plurality of support examples included in the support set. The machine learning encoder can be a neural network encoder. In some examples, the embedded support examples for each known class (e.g., each class found in the labeled support set) can be used to generate or otherwise determine a prototype representation for each class. In some cases, the prototype representation can be determined as an average of the embedded support examples for each class.

[0029]

[0038] In some examples, the machine learning (e.g., neural network) encoder may also receive a query set as input, where the query set includes a combination of query samples belonging to known classes and query samples belonging to unknown classes. The neural network encoder may generate embeddings for the query samples (e.g., generate embedded query samples), which may then be analyzed against the prototype representations to perform classification.

[0030]

[0039] In some examples, one or more dummy prototypes (e.g., dummy prototype representations) may be generated based on the set of prototype representations for each of the known classes. The dummy prototypes may be used to classify the open set query samples as not belonging to any of the known classes. For example, the open set query samples may be rejected or removed as outliers. In some examples, the dummy prototypes may be used to perform Few-Shot Open Set Keyword Spotting (FSOS-KWS), as described in more detail below.

[0031]

[0040] An illustrative example of a dummy protocol network for few-shot open set keyword spotting is provided in Appendix A attached hereto.

[0032]

[0041] Various aspects of the present disclosure are described with reference to the figures. Figure 1 illustrates an example implementation of a system on chip (SoC) 100 that may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., neural networks with weights), delays, frequency bin information, task information, among other information, may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, and / or may be distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 or from the memory block 118.

[0033]

[0042] The SoC 100 may also include a connectivity block 110 that may include a GPU 104, a DSP 106, fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., and additional processing blocks tailored to specific functions, such as a multimedia processor 112 that may detect and recognize gestures, speech, and / or other interactive user actions or inputs. In one implementation, the NPU is implemented within the CPU 102, the DSP 106, and / or the GPU 104. The SoC 100 may also include a navigation module 120 that may include a sensor processor 114, image signal processors (ISPs) 116, and / or a global positioning system. In some examples, the sensor processor 114 may be associated with or connected to one or more sensors for providing sensor input to the sensor processor 114. For example, one or more sensors and the sensor processor 114 may be provided within, coupled to, or otherwise associated with the same computing device.

[0034]

[0043] In some examples, the one or more sensors may include one or more microphones for receiving sounds (e.g., audio input), including sounds or audio input that may be used to perform keyword spotting (KWS) as described herein. In some cases, sounds or audio input received by the one or more microphones (and / or other sensors) may be digitized into data packets for analysis and / or transmission. The audio input may include ambient sounds in the vicinity of the computing device associated with the SoC 100 and / or may include speech from a user of the computing device associated with the SoC 100. In some cases, the computing device associated with the SoC 100 may additionally or alternatively be communicatively coupled to one or more peripheral devices (not shown) and / or configured to communicate with one or more remote computing devices or external resources using a communication network, such as, for example, a wireless transceiver and a cellular communication network.

[0035]

[0044] The SoC 100 may be based on an ARM instruction set. In one aspect of the disclosure, the instructions loaded into the CPU 102 may comprise code for searching a stored multiplication result in a lookup table (LUT) corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into the CPU 102 may also comprise code for disabling a multiplier during a multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. Additionally, the instructions loaded into the CPU 102 may comprise code for storing a calculated multiplication product of an input value and a filter weight when a lookup table miss of the multiplication product is detected.

[0036]

[0045] The SoC 100 and / or components thereof may be configured to perform audio signal processing using one or more machine learning networks and / or techniques according to aspects of the present disclosure and as described herein. For example, the SoC 100 and / or components thereof may be configured to perform keyword spotting according to aspects of the present disclosure. In some examples, by using one or more neural network classifiers to perform few-shot learning (FSL) and / or open set recognition (OSR), aspects of the present disclosure may improve the accuracy and / or efficiency of keyword spotting (KWS).

[0037]

[0046] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences without explicit instructions. One example of an ML system is a neural network (also called an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks can be used for a variety of applications and / or devices, such as speech analysis, audio signal analysis, image and / or video coding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, among others.

[0038]

[0047] Individual nodes in a neural network can emulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data may be selectively passed to other neurons. A weight value is associated with each vector and node in the network, and these values ​​constrain how the input data relates to the output data. For example, the input data of each node may be multiplied by the corresponding weight value and the products may be summed. The summed products may be adjusted by an optional bias, and an activation function may be applied to the result to obtain the output signal or "output activation" of the node (sometimes called a feature map or activation map). The weight values ​​may initially be determined by an iterative flow of training data through the network (e.g., the weight values ​​are established during a training phase in which the network learns how to identify particular classes by their typical input data characteristics).

[0039]

[0048] There are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multilayer perceptron (MLP) neural networks, and transformer neural networks, among others. For example, convolutional neural networks (CNNs) are a type of feed-forward artificial neural network. A convolutional neural network may include a collection of artificial neurons, each with a receptive field (e.g., a spatially localized region of the input space) and tiling the input space together. RNNs work in a way that they save the output of a layer and feed this output back to the input to help predict the outcome of the layer. GANs are a form of generative neural network that can learn patterns in the input data so that the neural network model can generate new synthetic outputs that can reasonably be from the original dataset. A GAN can include two neural networks working together, including a generative neural network that generates a synthesized output and a discriminative neural network that evaluates the output for reliability. In an MLP neural network, data can be fed into an input layer, and one or more hidden layers provide a level of abstraction to the data. Predictions can then be made in the output layer based on the abstracted data.

[0040]

[0049] Deep learning (DL) is an example of a machine learning technique and can be considered a subset of ML. Many DL approaches are based on neural networks, such as RNNs or CNNs, and utilize multiple layers. The use of multiple layers in a deep neural network can allow increasingly higher levels of features to be extracted from a given input of raw data. For example, the output of a first layer of artificial neurons becomes the input to a second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input to a third layer of artificial neurons, and so on. The layers located between the input and output of the entire deep neural network are often referred to as hidden layers. Hidden layers learn (e.g., are trained) to transform intermediate inputs from previous layers into slightly more abstract and synthetic representations that can be provided to subsequent layers until a final or desired representation is obtained as the final output of the deep neural network.

[0041]

[0050] As mentioned above, a neural network is an example of a machine learning system, and may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processing is performed by hidden nodes in one or more hidden layers, and output is generated through output nodes in the output layer. A deep learning network typically includes multiple hidden layers. Each layer of a neural network may include a feature map or activation map, which may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. The nodes may include one or more weights that are used to indicate the importance of one or more nodes of the layer. In some cases, a deep learning network may have a series of many hidden layers, with early layers being used to determine simple, low-level characteristics of the input, and later layers building a hierarchy of more complex and abstract characteristics.

[0042]

[0051] Deep learning architectures may learn a hierarchy of features. If presented with visual data, for example, a first layer may learn to recognize relatively simple features such as edges in the input stream. In another example, if presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases. Deep learning architectures may perform particularly well when applied to problems that have a natural hierarchical structure. For example, classification of electric vehicles may benefit from first learning to recognize wheels, windshields, and other features. These features may be combined in different ways in higher layers to recognize cars, trucks, and planes.

[0043]

[0052] Neural networks may be designed with various connectivity patterns. In feedforward networks, each neuron in a given layer communicates with neurons in a higher layer, so that information is passed from lower layers to higher layers. As described above, hierarchical representations may be constructed in successive layers of a feedforward network. Neural networks may also have recurrent or feedback (also called top-down) connections. In recurrent connections, the output from a neuron in a given layer may be transmitted to another neuron in the same layer. Recurrent architectures may be useful in recognizing patterns across two or more of the input data chunks delivered in a sequence to the neural network. Connections from neurons in a given layer to neurons in a lower layer are called feedback (or top-down) connections. Networks with many feedback connections may be useful when recognition of high-level concepts can help distinguish certain low-level features of the input.

[0044]

[0053] The connections between layers of a neural network may be fully connected or locally connected. FIG. 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, a neuron in a first layer may transmit its output to every neuron in a second layer, so that each neuron in the second layer receives input from every neuron in the first layer. FIG. 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in a layer has the same or similar connectivity pattern, but with connection strengths that may have different values ​​(e.g., 210, 212, 214, and 216). Because higher layer neurons in a given region may receive inputs that are tuned through training to the properties of a limited subset of all inputs to the network, the connectivity patterns of local connections may give rise to spatially distinct receptive fields within the higher layers.

[0045]

[0054] 3 is a diagram illustrating an example of a few-shot learning (FSL) scenario 300. In some examples, FSL and FSL-based approaches can be used to perform few-shot classification, where a classifier (e.g., a neural network or other machine learning classifier) ​​can generalize and extend inference to include new classes not seen in the training set, given only a few examples of each new class. For example, an FSL-based neural network classifier can be used to perform few-shot KWS for detecting keywords in audio streams or audio data, as previously described.

[0046]

[0055] As shown, Figure 3 illustrates an example of a 3-way 5-shot FSL scenario, where a neural network classifier (not shown) receives as input a support set including a total of 15 labeled support examples that are divided into three previously unseen classes (e.g., 310, 320, and 330) with 5 support examples each. In one illustrative example, the 3-way 5-shot FSL scenario 300 can be associated with a prototype network that learns a metric space in which classification can be performed by computing distances to prototype representations of each class.

[0047]

[0056] The use of a prototype network for FSL may be based on the idea that there exists an embedding in which points cluster around a single prototype representation for each class. As shown in FIG. 3, the five support examples in class 310 are found to be clustered around prototype representation 315, the five support examples in class 320 are found to be clustered around prototype representation 325, the five support examples in class 330 are found to be clustered around prototype representation 335, etc. The prototype network may include one or more neural networks that learn (e.g., during training) a non-linear mapping of inputs to an embedding space. Using the learned embedding space, the prototype network may render a prototype of each previously unseen class the average of its support set in the embedding space. For example, the prototype representation 315 may be determined as the average of the five embedded support examples for class 310, the prototype representation 325 may be determined as the average of the five embedded support examples for class 320, the prototype representation 335 may be determined as the average of the five embedded support examples for class 330, etc.

[0048]

[0057] Classification may then be performed on the embedded query point by determining the class prototype that is closest to the query point. For example, embedded query point 370 may be classified into class 320 based on a determination that the distance from embedded query point 370 to prototype representation 325 is less than the distance from embedded query point 370 to any of the remaining prototype representations 315 and 335. In some examples, embedded query points (e.g., embedded query point 370) may be classified based on the Euclidean distance between the embedded query point and each of the prototype representations, although it should be noted that various other distance metrics and / or determinations may also be utilized without departing from the scope of this disclosure.

[0049]

[0058] FIG. 4A illustrates another example of an FSL scenario 400a that uses the prototypical networks (ProtoNets) described above with respect to FIG. 3. As previously described, an N-ary M-shot episode (e.g., N * In one round of inference on a support set containing M support examples), FSL may be performed with the goal of correctly classifying one or more queries in a query set into N known classes based on receiving M support examples for each class not seen during training but not previously seen. In some examples, a pure FSL-based approach does not consider open-set classes from the N classes, meaning that a pure FSL-based approach such as that of FSL scenario 400a only classifies queries into one of N closed-set classes.

[0050]

[0059] In one illustrative example, FSL scenario 400a is a 3-way 5-shot FSL scenario, meaning that classification is performed across three classes 410, 420, and 430 in an evaluation (e.g., inference) setting, and respective prototype representations 415, 425, and 435 are generated using five embedded supporting examples for each of the three classes. In the context of an FSL-based KWS, each of the three classes 410, 420, and 430 can represent a keyword or keyword classification. For example, class 410 can represent keyword "1," class 420 can represent keyword "2," and class 430 can represent keyword "3." Because FSL scenario 400a is a 3-way 5-shot FSL scenario, support set S (not shown) may include a total of 15 support examples (e.g., five example utterances of “1” labeled with keyword classification 410, five example utterances of “2” labeled with keyword classification 420, and five example utterances of “3” labeled with keyword classification 430).

[0051]

[0060] As shown in FIG. 4A, the query set Q of test utterances can include a combination of closed-set queries and open-set queries. The closed-set queries can represent test utterances from known classes (e.g., test utterances of “1”, “2”, or “3”, corresponding to support set classes 410, 420, 430, respectively). As shown in FIG. 4A, the query set includes three closed-set test utterances for each class 410, 420, 430. The open-set queries can represent test utterances from unknown classes and are shown as open circles with “?” inside. For example, an open-set query can include test utterances such as “4”, “5”, “6”, etc., but it should be noted that the open-set queries / test utterances can take various other forms with various degrees of similarity to the closed-set test utterances (e.g., since an open set can be defined as including any test utterance that does not match a known class). The open set query test utterances may be associated with one or more unseen, unsupported classes that were not seen in either the initial training dataset (e.g., used to initially train the neural network classifier associated with FSL scenario 400a) or the support set S (e.g., provided to the neural network classifier at inference time and used to implement FSL).

[0052]

[0061] Because the open set classes are new, unknown, or otherwise unsupported by the neural network classifier associated with FSL scenario 400a, each of the open set queries will be misclassified into one of the three known (e.g., closed set) classes 410, 420, 430. For example, the decision boundary shown in FIG. 4A as a dotted line dividing the embedding space can represent the (mis)classified closed set classifications that may be determined for the open set queries. In some examples, misclassification of queries from open set classes may occur due to the design of existing FSL approaches (e.g., FSL scenario 400a, etc.), which are limited to only known closed set classes and cannot consider any open set classes, regardless of the distance between the open set query and the closest prototype representation determined for one of the closed set classes.

[0053]

[0062] In some real-world scenarios and / or KWS implementations, unseen unsupported classes such as "4", "5", and "6" may be regularly encountered and are consistently misclassified by existing FSL approaches as belonging to known or seen classes from a closed set of FSL classes included in the seen or support set S in training. Any unseen unsupported classes need to be handled or rejected during KWS classification to prevent novel open set classes from being misclassified into seen closed set classes.

[0054]

[0063] 4B illustrates a first exemplary episode 400b of a few-shot open set recognition (FSOSR) approach according to one or more aspects of the present disclosure. FIG. 4C illustrates a second exemplary episode 400c of the FSOSR approach according to one or more aspects of the present disclosure. In one illustrative example, the first FSOSR episode 400b and the second FSOSR episode 400c may represent or apply to the same underlying sample set S and query set Q as described above with respect to the exemplary FSL scenario 400a shown in FIG. 4A.

[0055]

[0064] The systems and techniques described herein can implement FSOSR, including FSOSR for KWS (e.g., FSOS-KWS), to distinguish unknown open sets from known closed set classes while implementing FSL. For example, the systems and techniques can receive as input a query set including a combination of closed set test utterances (e.g., that should be correctly classified as belonging to one of the known classes) and open set test utterances (e.g., that should be classified as belonging to one or more unknown classes and not to any of the known classes). As described in more detail below, the systems and techniques can implement FSOSR and / or FSOS-KWS using episode-known dummy prototypes based on metric learning to detect and reject open sets. In some examples, episode-specific (e.g., episode-known) dummy classes are predicted based on support examples included in a support set provided for each episode, and open sets are classified as dummy classes and rejected from inclusion in a subsequent KWS.

[0056]

[0065] In one illustrative example, the open set of test utterances with unknown or unseen classes can change across episodes based on the selection of N classes in the support set S provided as inference input to each episode. For example, FIG. 4B shows a first episode 400b with a first open set of queries with unseen classes, and FIG. 4C shows a second episode 400c with a second open set of queries with unseen classes. The first episode 400b and the second episode 400c are constructed for the same underlying query set (e.g., the same embedding sample points are shown in FIG. 4B and FIG. 4C). Based on the selection of N classes identified or included in the support set provided for each of the episodes 400b and 400c, the composition of the open set changes from episode 400b to 400c. The composition of the closed set classification also changes from episode 400b to 400c. For example, the first classification 410b includes a different set of queries / test utterances than the first classification 410c, the second classification 420b includes a different set of queries / test utterances than the second classification 420c, the third classification 430b includes a different set of queries / test utterances than the third classification 430c, etc.

[0057]

[0066] As described in more detail below, the systems and techniques described herein can implement FSOS-KWS to distinguish unknown open sets from known closed set classes for each episode (e.g., unknown open sets can be uniquely distinguished from known closed sets for any given selection of N classes associated with the support set S provided for a given episode). The systems and techniques predict episode-specific (e.g., episode-known) dummies based on the support examples and / or support set S provided as input to each episode. The predicted episode-specific dummies are then classified as episode-specific open sets, which can then be rejected or otherwise excluded from the KWS classification. In one illustrative example, the systems and techniques can utilize Dummy Prototype Networks (D-ProtoNets), as described in more detail below.

[0058]

[0067] 5 illustrates an example of a Dummy Prototype Network (D-ProtoNet) 500 for implementing FSOSR and / or FSOS-KWS with episode-specific open set recognition and / or rejection. In one illustrative example, the Dummy Prototype Network 500 can implement FSOS-KWS adaptively to changing open sets between episodes. In some aspects, the Dummy Prototype Network 500 can implement FSOS-KWS by using metric-based learning to learn a metric space in which a distance metric can classify samples.

[0059]

[0068] In the context of the following discussion, the FSOSR setting is a set of previously seen training data D that does not overlap with the classes. train And, first-time evaluation data D eval For example, the training data D train is a set of multiple labeled samples.

[0060]

number

[0061] where x i are the input features, and y i are their corresponding labels. In the FSOS-KWS example, the input feature x i may be an utterance or other speech and / or audio data. i is the input feature x i The keyword may be a class or classification of keywords.

[0062]

[0069] In one illustrative example, a machine learning model (e.g., a neural network model) associated with the dummy prototype network 500 may be trained to learn from a number of N-ary, M-shot pseudo-FSOSR episodes. For example, each pseudo-FSOSR episode may include N known classes with M supporting examples per class (e.g., each pseudo-FSOSR episode may include N total classes with M supporting examples per class). * The class may include one or more pseudo-unknown (eg, pseudo-open set) classes that have no supporting examples, as well as one or more pseudo-unknown (eg, pseudo-open set) classes that have no supporting examples.

[0063]

[0070] The pseudo-episodes used for training can be designed to mimic the FSOSR inference task by subsampling the classes and data points. For the FSOSR inference task, inference can be performed on episodes each associated with a support set S and a query set Q. For example, FIG. 5 shows a support set S (denoted as 512) that includes a closed set K (denoted as 513) of known classes for which supporting examples are provided.

[0064]

[0071] 5 also shows a query set Q (denoted as 516) that includes both a closed set K of known classes (denoted as set 517) and an open set U of unknown classes with no supporting examples (denoted as set 519). Note that the set of closed set samples 513 and the closed set of samples 517 correspond to the same underlying known classes, although it is not necessary that the constituent samples in 513 are the same as the constituent samples in closed set 517.

[0065]

[0072] In one illustrative example, the support set S (e.g., 512) can include M samples for each of the N classes, as described above. The query set Q (e.g., 516) can include one or more queries from the N known classes (e.g., in set 517), and N (e.g., in set 519). U The set of queries may further include one or more queries from the unknown classes.

[0066]

number

[0067]

[0073] Here, M Q can represent the number of queries per class. During inference, the evaluation dataset D eval All classes in may be new to the dummy prototype network 500. In some examples, the evaluation dataset D eval can be the same as the query set 516. Inference is performed by comparing N known classes (e.g., set 517) with supporting samples and N U In the inference episode, the evaluation dataset D eval Note that the set (eg, 517) does not necessarily contain M support examples for each of the N classes, as is the case for the support set S (eg, 512).

[0068]

[0074] As discussed above, the dummy prototype network 500 can implement the FSOS-KWS based at least in part on one or more metric-based learning approaches. For example, in an N-ary M-shot episode, the dummy prototype network 500 can train N prototypes using the average of the embedded support samples n for each class.

[0069]

number

[0070] One can determine or otherwise obtain and compute the corresponding prototype for each class, where:

[0071]

number

[0072]

[0075] Here, S n is a subset of the support set S (e.g., 512) whose label is n, and |S n |=M, and f φ In the example of Fig. 5, which is an encoder with φ parameters, the prototype c n can be obtained based on determining the average of the embedded support samples of each class in the averaging engine 530. In one illustrative example, the individual prototypes are n The notation f is used to represent a set of N prototypes, whereas the notation C (e.g., as shown in block 532 of FIG. 5) can represent a set of N prototypes. As shown, φ is shown as an encoder 520, which receives as input each episode of the support set 512 and the query set 516. In one illustrative example, the encoder 520 may be a neural network or other machine learning network that generates embeddings based on the support set 512 and / or the query set 516 (e.g., a prototype c n is the encoder f φ 520).

[0073]

[0076] In one illustrative example, the encoder f φ 520 may include or implement a machine learning backbone. For example, the encoder f φ 520 may include or implement one or more of Conv4-64, ResNet-12, and / or BCResNet-8. φ In an example where 520 uses a Conv4-64 architecture, the output dimension may be 768. φ In an example where 520 uses the ResNet-12 architecture, the output dimension may be 512. φIn an example where 520 uses a BCResNet-8 architecture, the output dimension may be 256. In some cases, the Conv4-64 architecture may be associated with 768 dimensions (e.g., yielding 768 dimensions, which is greater than the number of its input channels, 64), based at least in part on the absence of global average pooling at its output.

[0074]

[0077] Prototype C n Based on this, the dummy prototype network 500 can determine or otherwise obtain a probability distribution over the N known classes.

[0075]

number

[0076]

[0078] where d(·) is a distance metric (eg, shown as distance metric 540 in FIG. 5). In one illustrative example, the distance metric d(·) may be determined based on Euclidean distance. d(z,z ’ )=||zz ’ || 2 Formula (3)

[0077]

[0079] In some aspects, the distance metric d(·) may be determined or otherwise implemented using a distance metric other than Euclidean distance (e.g., the Euclidean distance of Equation (3)). In some examples, based on Equation (2) and / or Equation (3), the systems and techniques described herein may use a dummy prototype network 500 to minimize the negative log-probability, which is the negative log-probability of the true class n.

[0078]

number

[0079] It can be given as:

[0080] In one illustrative example, the probability distribution in equation (2) is n For example, if the input query example x is closest to class number 3 (e.g., of N classes), then a relatively high probability can be determined for class 3 and relatively low probabilities for the remaining N-1 classes. For example, these probabilities can be determined based on Equation (2), which itself can be determined based on a distance metric d(·) (e.g., such as the Euclidean distance metric given in Equation (3)).

[0081] In one illustrative example, the systems and techniques described herein can further determine one or more episode-known dummy prototypes to handle (e.g., reject from the KWS) open sets that vary across multiple FSOS-KWS episodes. For example, the dummy prototype network 500 may include a dummy generator g for determining or otherwise generating episode-known dummy prototypes. φ 550. The dummy generator g φ 550 generates a set of original prototypes C for a given episode (e.g., using equation (1) to generate an encoder f φ Based on receiving as input a set of prototypes C generated by averaging the embeddings output by 520, d As will be described in more detail below, a dummy generator g φ Dummy prototype produced by 550c d is the original prototype C={c1,...,c N}, and the expanded (e.g., dummy-expanded) prototype set {c1,...,c N ,c d} can be generated.

[0082] In some examples, a set of N prototypes,

[0083]

number

[0084] can be used for each inference episode of the dummy prototype network 500, where c ∈ R 1xD and f φ , D. In some cases, the prototypes may be permutation invariant to one another for the dummy prototype network 500. In one illustrative example, the dummy generator g φ 550 may have a parameter φ based on DeepSets. In some aspects, the dummy generator g φ 550 may be permutation invariant. In one illustrative example, the dummy generator g φ 550 is a set of N prototypes C = [c1;c2; ;c N ]∈R NxD or given as input a dummy c d 552 can be generated.

[0085] In one illustrative example, the dummy c d 552 is generated using a set of prototypes C determined for each class of episodes (e.g., uniquely determined based on the support set and / or support examples provided for a given episode), d can be episode-specific and episode-known. As mentioned above, changing the selection and / or composition of N can change the open set (e.g., as described with respect to the first and second episodes shown in Figures 4B and 4C, respectively). In some aspects, the dummy c d 552 is generated based on a set of prototypes C that are uniquely determined for each class, and the dummy c d 552 is not static. For example, dummy c d552 can change with changes in the open set caused by changes in N and / or the support set S.

[0086] In one illustrative example, the dummy expression c d 552 is a dummy generator g φ 550 can be generated as follows:

[0087]

number

[0088] where g1 includes a fully-connected (FC) layer with nonlinearity, and g1(C)∈R NxH has hidden dimension H. For example, g1 may be a two-layer linear layer with a nonlinearity level between the two linear layers. In some examples, the dummy generator g φ 550 may use g1 of FC-ReLU-FC with hidden D=32.

[0089] Following g1, max pooling can be applied. For example, max pooling in equation (4) operates on N (e.g., the number of prototypes / classes) and R 1xH The feature output from max pooling is W g This can be implemented as a learnable HxD matrix. The result of this matrix multiplication is the generated dummy c d 552, c d ∈R 1xD It is.

[0090] As mentioned above, the extended prototype set {c1, . . . , c N ,c d} is a dummy prototype d 552 (e.g., dummy generator g φ 550) into the original prototype set C (e.g., function generator f φ520 and the averaging engine 530, which applies equation (1) to obtain

[0091] In one illustrative example, the label of an open set query is a dummy d The N+1th label y corresponding to 552 d The dummy extended prototype {c1,...,c N ,c d}, equation (2) can be modified to determine or otherwise obtain a probability distribution over the N+1 prototypes as follows:

[0092]

number

[0093] In some examples, the probability distribution over the N+1 prototypes (e.g., as previously described with respect to Equation (5)) may be determined in the same or similar manner as the probability distribution over the N prototypes (e.g., as previously described with respect to Equation (2)). For example, the probability distribution over the N+1 prototypes may be determined using a softmax (e.g., the N+1 softmax 560 shown in FIG. 5) based on the same distance metric d(·) 540. In some examples, the distance metric 540 may be based on Euclidean distance, as previously described (e.g., with respect to Equation (3)). The N+1 softmax 560 may be followed by an N+1 classifier 570, which may generate an N-way classification 580. In some aspects, the N+1 softmax 560 and the N+1 classifier 570 may be used to determine the N classes and the newly generated episode-specific dummy class c d In one example, the N-way classification 580 may be performed on argmax n∈{1,...,N} p(y=n|x)) can be generated based on an argmax function.

[0094] In the example of equation (5), θ may represent or otherwise include the parameters φ of the encoder 520 and the parameters φ of the dummy generator 550. n is the softmax temperature. For example, τ n may be a softmax temperature associated with or implemented by the N+1 softmaxes 560. In some examples, the softmax temperature may be held constant or otherwise the same across different classes N.

[0095] In one illustrative example, N+1 softmaxes 560 are used (e.g., as described above) to reduce the labels of an open set query to the N+1th label y d In some embodiments, other τ n Larger τ compared to ≠ N+1 N+1 (e.g., within or by the N+1 softmax 560), so that the dummy c d 552 does not significantly affect the original N-class classification (e.g., τ N+1 = γ τ n ≠N+1, where γ>1. In some cases, during training, the softmax temperature τ n≠N+1 can be fixed to 1, and the loss function L CE For γ=3 (for example, τ N+1 =3), which is explained below with respect to equation (7).

[0096] In some aspects, the probability distribution of equation (5) can be output or otherwise determined using N+1 softmaxes 560. During training, a cross-entropy loss for known classes and a cross-entropy loss for known classes can be determined and used to generate a combined loss or loss function that drives the training process. In one illustrative example, N+1 class classifications 570 can be learned (e.g., during training) using one or more cross-entropy losses, as given below:

[0097]

number

[0098]

[0093] Here, Q K and Q U are known queries and unknown queries of the query set Q516, respectively. For example, Q K can be a known query from the set of known queries 517, and Q U can be unknown queries from a set of unknown queries 519, both of which are shown in FIG. 5 as being included in query set Q 516.

[0099]

[0094] In some cases, the two losses given in equations (6) and (7) can be balanced by a hyperparameter λ, and the annihilation function (e.g., still the cross-entropy loss) is given as follows:

[0100]

number

[0101] In some examples, the hyperparameter λ may be set equal to 0.1, although other hyperparameter values ​​may also be utilized. Using the cross-entropy loss of equation (8), a dummy prototype network described herein (e.g., dummy prototype network 500) is constructed by multiplying the dummy generator g φ Dummy prototype generated using 550c d and the function generator f φ 520 and the original set of prototypes C generated using

[0102] During inference, the cross-entropy loss (e.g., equations (6)-(8)) is no longer needed or utilized. In one illustrative example, N+1 classes may be identified based on or using the N+1 softmax outputs. For example, the outputs of the N+1 softmaxes 560 may be used to detect and reject open set queries. As shown in FIG. 5, the dummy prototype network 500 may include an outlier detection engine 590 that may be used to detect open set queries for a given FSOS-KWS episode.

[0103] In one illustrative example, the outlier detection engine 590 detects a dummy class c d For example, the outlier detection engine 590 may detect open set queries for a given FSOS-KWS episode based on the probability over i It can be determined whether x belongs to an open set (eg, is therefore an outlier and / or should be rejected).

[0104]

number

[0105] In some examples, the probability p given by equation (9) θ is the sum of the functions for a given query x i Dummy Class C d (e.g., y i is the query x i y represents a label or classification. d is dummy class c d 552, and the label of the open set query is the N+1th label y d can be set to

[0106] In one illustrative example, the outlier detection engine 590 may use the above probabilities (e.g., p θ (yi =y d │x i )) based on a threshold δ. i It is possible to detect or otherwise determine that belongs to an open set.

[0107] As shown in FIG. 5, the classification output of the N+1 classifier 570 with open set rejection applied by the outlier detection engine 590 is argmax n∈{1,...N} p θ (y=n|x i ) is an N-way classification.

[0108]

[0101] In some examples, the dummy prototype network of FIG. 5 can be tested by classifying N known classes by:

[0109]

number

[0110]

[0102] Given x i To verify whether is an open set or not, we use the probability p given by equation (9). θ (y i =y d │x i ) may be compared to or analyzed against a threshold δ (eg, the test may be performed the same as or similar to the reasoning described above).

[0111] In one illustrative example, the systems and techniques described herein may utilize multiple dummies L. For example, the example prototype network 500 shown in FIG. 5 may include a learnable matrix W (e.g., of Equation (4)) g Multiple dummies L can be included or generated based on setting L to be a matrix of dimension H × (L D). In the single dummy example above, the learnable matrix W gNote that L is provided as a matrix of dimension H×D (e.g., L=1).

[0112]

[0104] In some cases, a learnable matrix W with dimensions H × (L D) is g Based at least in part on the use of l (-d(x i ,c l )) for a given input x i We can simply select the most likely one of the L dummies for

[0113] In one illustrative example, a Gumbel softmax can be used to replace non-differentiable samples (e.g., argmax) with differentiable samples. For example, the probability of selecting a dummy l of a set of multiple dummies L using a Gumbel softmax can be given as follows:

[0114]

number

[0115]

[0106] where ∈1, , ∈ L are independent and identically distributed (IID) samples drawn from a standard Gumbel distribution with μ=0 and β=1. L are temporary dummy labels among the L dummies. In some examples, training is performed to

[0116]

number

[0117] is obtained.

[0118] In one illustrative example, during inference, a single dummy yL can be selected for use in performing open set recognition (OSR). For example, a single dummy y L can be selected according to the following formula:

[0119]

number

[0120] For example, the selected dummy y L can be used to classify and reject open set queries as a dummy class in the same or similar manner as described above with respect to the example in which the dummy generator 550 generates a single dummy.

[0121] In one illustrative example, training and / or inference may be performed with the τ term in the Gumbel softmax (e.g., equation (11) above) annealed from 2 to 0.5. In some examples, when multiple dummies L are utilized, the number of dummies may be L=3, although it should be noted that a greater or lesser number of dummies L may also be utilized without departing from the scope of this disclosure.

[0122]

[0110] In some examples, one or more spoken command datasets may be used for training, validation, and / or testing of the systems and techniques described herein. In one illustrative example, the spoken command dataset may include a total of 35 keywords and may be used to support 12 class classification. For example, the 12 classes may include 10 keywords (e.g., "yes", "no", "up", "down", "left", "right", "on", "off", "stop", and "go") and two additional classes. The two additional classes may include an "unknown word" class including the 25 remaining keywords (e.g., the total of 35 keywords in the spoken command dataset minus the above 10 keywords) and a "silence" class. In some cases, the "silence" class may include only background noise.

[0123] In some examples, the speech command dataset can be split by class labels. For example, a split can be performed to establish 15 keywords for the training set, 10 keywords for the validation set, and 10 keywords for the test set, i.e. Training keywords: "happy", "house", "bird", "bed", "backyard", "Sheila", "Marvin", "wow", "tree", "follow me", "dog", "visual", "forward", "learn", and "cat", Verification keywords (e.g., numbers): "0", "1", "2", "3", "4", "5", "6", "7", "8", and "9", Test keywords (10 keyword classes used in the traditional 12-class KWS): "yes", "no", "up", "down", "left", "right", "on", "off", "stop", and "go".

[0124]

[0112] In some cases, the use of fixed keyword splits can prevent possible performance variance from split changes across trials. On top of the split, a specific class "silence" can be added, which can only be included in the open set as a background noise class. For example, in a 5-source 5-shot episode, 5 known classes without "silence" can be randomly selected, and an equal number of open set classes can also be selected from the remaining classes including "silence". In some examples, these settings can be referred to as split speech command datasets.

[0125]

[0113] In some examples, the underlying speech command dataset (e.g., before applying the above-mentioned splitting) may be the Google speech command (GSC) dataset ver2, which includes 105,829 utterances from 2,618 speakers. As described above, the dataset can first be split into a training set, a validation set, and a test set, each having 84,843, 9,981, and 11,005 utterances, using a formula splitting (e.g., using a hash function on the name of each utterance file).

[0126]

[0114] Samples can then be selected based on the split speech command dataset approach described above to obtain 22,916, 3,643, and 4,074 samples for training, validation, and testing, respectively. "Silence" samples can be added to each split by the average number of utterances per class in each split, resulting in a final split speech command dataset containing 24,444, 4,007, and 4,482 utterances for training, validation, and testing, respectively. In some examples, training can be performed by adding the official background noise provided by GSC with a probability of 0.8, using the minimal data augmentation generally used in GSC tasks.

[0127] In some aspects, the encoder f φ 520 may include or implement a machine learning backbone (e.g., as described above). For example, the encoder f φ 520 may include or implement one or more of Conv4-64, ResNet-12, and / or BCResNet-8. φ In an example where 520 uses a Conv4-64 architecture, the output dimension may be 768. φ In an example where 520 uses the ResNet-12 architecture, the output dimension may be 512. φIn an example where 520 uses a BCResNet-8 architecture, the output dimension may be 256. In some cases, the Conv4-64 architecture may be associated with 768 dimensions (e.g., yielding 768 dimensions, which is greater than the number of its input channels, 64), based at least in part on the absence of global average pooling at its output.

[0128]

[0116] In some aspects, each utterance in GSC is 1 second long and the sampling rate is 16 kHz. During training (e.g., using the final split speech command dataset described above), input features of 40-dimensional logarithmic mel-spectrograms with frame shifts and window lengths of 10 and 30 milliseconds (ms), respectively, may be used. In some examples, the dummy prototype network described herein (e.g., the dummy prototype network 500 shown in FIG. 5) may be trained for 100 epochs using an Adam optimizer with an initial learning rate of 0.001. The learning rate may be gradually decayed by multiplying by 0.5 every 20 epochs. Each epoch may include 100 episodes, each episode including 5 known (e.g., 5-way) and 5 open set classes. Five support examples (e.g., 5-shot) may be used or provided per episode. During training, 5 queries may be provided or otherwise utilized for each class. During testing, 15 queries may be provided or otherwise utilized for each class. In some instances, early stopping can be used with few-shot validation accuracy, and the trained model can be evaluated for 1,000 episodes.

[0129]

[0117] FIG. 6 is a flow chart illustrating an example of a process 600 for processing one or more data samples (e.g., one or more data samples of audio data). At block 602, the process 600 includes determining one or more prototype representations based on a plurality of support samples associated with one or more classes, each prototype representation being associated with one of the one or more classes. For example, the one or more prototype representations may be determined using the dummy prototype network 500 illustrated in FIG. 5. In some examples, the one or more prototype representations may be determined by a machine learning and / or neural network encoder, such as the neural network encoder 520 illustrated in FIG. 5. The one or more prototype representations may be determined based on receiving as input a plurality of support samples, each support sample being associated with a known class (e.g., labeled with a known class). In some examples, the encoder (e.g., the encoder 520) may generate a plurality of embeddings for a plurality of support samples associated with each keyword class. Then, for each keyword class, a prototype representation may be determined as the average of its embedded support samples. For example, the average of the embedded support samples for each class can be determined using the averaging engine 530 shown in FIG.

[0130] In some examples, the one or more classes may be keyword classes (e.g., of a keyword spotting (KWS) system). In some cases, each prototype representation may be determined using multiple supporting samples associated with one of the keyword classes. For example, each prototype representation may be determined for one keyword class of multiple keyword classes, and the prototype representation of the keyword class is determined based on supporting samples associated with the same keyword class.

[0131] At block 604, the process 600 includes determining a dummy prototype representation, where the one or more prototype representations and the dummy prototype representation are determined in the same learned metric space. For example, the dummy prototype representation may be determined using the dummy generator 550 shown in FIG. 5. In some examples, the dummy prototype representation may be determined using the prototype representation determined for each known class as input (e.g., as described above with respect to block 602). In some examples, the one or more prototype representations and the dummy prototype representation may be determined in the same learned embedding space of the neural network encoder (e.g., the dummy prototype representation may be determined in the same embedding space associated with the encoder 520).

[0132] In some examples, a dummy generator network (e.g., dummy generator 550) may be used to determine the dummy prototype representation. For example, the dummy generator network may be a machine learning network including one or more linear layers and a max pooling layer. The one or more linear layers may receive one or more prototype representations as inputs (e.g., from the encoder 520 and the averaging engine 530, both shown in FIG. 5). The max pooling layer may receive the output of one or more linear layers as inputs.

[0133] In some examples, determining the dummy prototype representation further includes generating a plurality of dummy prototypes using a dummy generator network (e.g., dummy generator 550). The dummy prototype representation may be a single dummy prototype selected from the plurality of dummy prototypes. For example, the dummy prototype representation may be a single dummy prototype selected from the plurality of dummy prototypes based on a Gumbel softmax.

[0134] At block 606, the process 600 includes determining one or more distance metrics for each query sample of the one or more query samples, where the one or more distance metrics are determined based on the one or more prototype representations and the dummy prototype representation. For example, the one or more distance metrics may be determined using the distance metric 540 as shown in FIG. 5. In some examples, the distance metric may be a Euclidean distance between each query sample and each of the prototype representations and the dummy prototype representations.

[0135]

[0123] At block 608, the process 600 includes classifying each query sample based on one or more distance metrics, where each query sample is classified into one of the classes associated with one or more prototype representations or into one of the open set classes associated with the dummy prototype representation. For example, the query sample can be classified using a softmax, such as the N+1 softmax 560 shown in FIG. 5. In some examples, the softmax classification can be based on or otherwise utilize the distance metric described above with respect to block 606. For example, the softmax classification can classify a given query sample into one of the classes associated with the prototype representations or into an open set class associated with the dummy prototype representation.

[0136] In some examples, classifying each query sample based on one or more distance metrics includes determining a probability distribution over one or more classes and open set classes, where the probability distribution is determined at least in part based on a Euclidean distance metric, such as the Euclidean distance metric described above. For example, a given query sample may have a relatively high probability of being classified into a given one of the classes or into an open set class as the Euclidean distance (or distance metric) determined between the query sample and one of the classes (or open set classes) decreases. In some examples, each query sample may be classified into one of the one or more classes associated with the prototype representation or into an open set class associated with the dummy prototype representation based on the probability distribution. In some examples, open set rejection (OSR) may be performed based on the set of classified query samples classified into the open set class associated with the dummy prototype representation.

[0137] In some examples, the processes described herein (e.g., process 600 and / or any other processes described herein) may be performed by a computing device, apparatus, or system. In one example, process 600 may be performed by a computing device or system having the computing device architecture 700 of FIG. 7. The computing device, apparatus, or system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an autonomous vehicle or autonomous vehicle computing device, a robotic device, a laptop computer, a smart television, a camera, and / or any other computing device having resource capabilities to perform the processes described herein, including process 600 and / or any other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0138]

[0126] Components of a computing device may be implemented with circuits. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform various operations described herein.

[0139]

[0127] The process 600 is illustrated as a logical flow diagram, whose operations represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. In general, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0140] Additionally, process 600 and / or any other process described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes collectively, by hardware, or a combination thereof on one or more processors. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0141]

[0129] Figure 7 illustrates an exemplary computing device architecture 700 of an exemplary computing device that can implement various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other device. The components of the computing device architecture 700 are shown to be in electrical communication with each other using a connection 705 such as a bus. The exemplary computing device architecture 700 includes a processing unit (CPU or processor) 710 and computing device connections 705 that couple various computing device components, including a computing device memory 715 such as a read only memory (ROM) 720 and a random-access memory (RAM) 725, to the processor 710.

[0142]

[0130] The computing device architecture 700 may include a cache of high-speed memory directly connected to, adjacent to, or integrated as part of the processor 710. The computing device architecture 700 may copy data from the memory 715 and / or storage device 730 to the cache 712 for quick access by the processor 710. In this manner, the cache may provide a performance boost that avoids delays in the processor 710 while waiting for data. These and other engines may control or be configured to control the processor 710 to perform various actions. Other computing device memories 715 may also be available for use. The memory 715 may include multiple different types of memory with different performance characteristics. The processor 710 may include any general-purpose processor, hardware or software services such as service 1 732, service 2 734, and service 3 736 stored in the storage device 730 configured to control the processor 710, and dedicated processors with software instructions built into the processor design. The processor 710 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. A multi-core processor may be symmetric or asymmetric.

[0143] To enable user interaction with the computing device architecture 700, the input device 745 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. The output device 735 can also be one or more of some output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 700. The communication interface 740 can generally govern and manage user input and computing device output. There is no constraint to operate on any particular hardware configuration, and therefore the basic functions herein can be easily replaced with improved hardware or firmware configurations as they are developed.

[0144]

[0132] The storage device 730 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAMs) 725, a read-only memory (ROM) 720, and hybrids thereof. The storage device 730 may include services 732, 734, 736 for controlling the processor 710. Other hardware or software modules or engines are also contemplated. The storage device 730 may be connected to a computing device connection 705. In one aspect, a hardware module performing a particular function may include software components stored in a computer-readable medium in association with necessary hardware components, such as the processor 710, the connection 705, the output device 735, etc., to perform the function.

[0145] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, a smartphone, a tablet, a laptop computer, a vehicle, a drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0146]

[0134] The term "device" is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system, etc.). A device as used herein may be any electronic device having one or more parts that can implement at least some parts of the present disclosure. The following description and examples use the term "device" to describe various aspects of the present disclosure, but the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or a specific aspect. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. The following description and examples use the term "system" to describe various aspects of the present disclosure, but the term "system" is not limited to a specific configuration, type, or number of objects.

[0147]

[0135] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that the aspects can be practiced without these specific details. For ease of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method embodied in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the aspects in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the aspects.

[0148]

[0136] Each aspect may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0149]

[0137] The processes and methods according to the examples described above can be implemented using computer-executable instructions stored in or otherwise available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary or intermediate format instructions such as assembly language, firmware, source code, etc.

[0150]

[0138] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, store, or convey instructions and / or data. Computer-readable media may include non-transitory media on which data may be stored and that do not include carrier waves and / or transitory electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as flash memory, memories or memory devices, magnetic or optical disks, flash memories, USB devices provided with non-volatile memory, networked storage devices, compact disks (CDs) or digital versatile disks (DVDs), among others, or any suitable combination thereof. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, an engine, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0151] In some aspects, computer-readable storage devices, media, and memories can include cables or wireless signals containing bit streams, etc. However, when mentioned, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0152]

[0140] A device implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) that perform the necessary tasks may be stored in a computer-readable medium or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-footprint personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board among different chips, or on different processes running in a single device, as further examples.

[0153]

[0141] The instructions, media for propagating such instructions, computing resources for executing such instructions, and other structures supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0154]

[0142] In the foregoing description, the aspects of the present application have been described with reference to certain aspects thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary aspects of the present application have been described in detail herein, it should be understood that the inventive concept may be variously implemented and utilized in other ways, and the appended claims are intended to be construed to include such variations, except as limited by conventional techniques. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the aspects may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative aspects, the methods may be performed in an order different from that described.

[0155]

[0143] Those skilled in the art will understand that the symbols or terminology used in this specification such as less than ("<") and greater than (">") can be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this specification.

[0156]

[0144] When a component is described as being "configured to" perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0157]

[0145] The phrase "coupled to" refers to any component that is physically connected to another component, either directly or indirectly, and / or any component that is in communication with another component, either directly or indirectly (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0158]

[0146] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include unrecited items in the set of A and B.

[0159]

[0147] The various exemplary logic blocks, modules, engines, circuits, and algorithm steps described in relation to the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this compatibility of hardware and software, the various exemplary components, blocks, modules, engines, circuits, and steps are described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such a determination of the implementation form should not be interpreted as a cause for departing from the scope of this application.

[0160]

[0148] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random-access memory (SDRAM), read-only memory (ROM), non-volatile random-access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, which carries or communicates program code in the form of instructions or data structures and which can be accessed, read and / or executed by a computer.

[0161]

[0149] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0162]

[0150] Exemplary aspects of the present disclosure include the following.

[0163]

[0151] Aspect 1: A method for processing one or more data samples, comprising: determining one or more prototype representations based on a plurality of support samples associated with one or more classes, where each prototype representation is associated with one of the one or more classes; determining a dummy prototype representation, where the one or more prototype representations and the dummy prototype representation are determined in the same learned metric space; determining one or more distance metrics for each query sample of the one or more query samples, where the one or more distance metrics are determined based on the one or more prototype representations and the dummy prototype representation; and classifying each query sample based on the one or more distance metrics, where each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the dummy prototype representation.

[0164]

[0152] Aspect 2: The method of aspect 1, wherein classifying each query sample based on one or more distance metrics includes determining a probability distribution over one or more classes and an open set class, where the probability distribution is determined based at least in part on a Euclidean distance metric, classifying each query sample into one of the one or more classes or into an open set class based on the probability distribution, and performing open set rejection (OSR) based on the set of classified query samples classified into open set classes associated with dummy prototype representations.

[0165]

[0153] Aspect 3: The method of aspect 1 or 2, wherein the one or more classes are keyword classes and each prototype representation is determined using a plurality of supporting samples associated with the keyword class.

[0166]

[0154] Aspect 4: The method of aspect 3, further comprising: using a neural network encoder to generate a plurality of embeddings for a plurality of support samples associated with a keyword class, the plurality of embeddings including an embedding for each support sample among the plurality of support samples; and determining each prototype representation based on an average of the plurality of embeddings generated for the plurality of support samples.

[0167]

[0155] Aspect 5: The method of aspect 4, wherein the one or more prototype representations and the dummy prototype representations are determined within the same learned embedding space of the neural network encoder.

[0168]

[0156] Aspect 6: A method according to any one of aspects 1 to 5, further comprising determining a dummy prototype representation using a dummy generator network, the dummy generator network including one or more linear layers and a max pooling layer, the one or more linear layers receiving one or more prototype representations as inputs, and the max pooling layer receiving outputs of the one or more linear layers as inputs.

[0169]

[0157] Aspect 7: The method of aspect 6, wherein determining the dummy prototype representation further includes generating a plurality of dummy prototypes using a dummy generator network, and determining the dummy prototype representation by selecting one of the plurality of dummy prototypes based on Gumbel softmax.

[0170]

[0158] Aspect 8: The method of any one of aspects 1 to 3, wherein the keyword class comprises a closed set of known classes.

[0171]

[0159] Aspect 9: A method described in any one of aspects 1 to 8, wherein the one or more query samples include speech audio data, and the multiple support samples include speech audio data labeled with a corresponding one of one or more classes for each support sample among the multiple support samples.

[0172]

[0160] Aspect 10: A method according to any one of aspects 1 to 9, wherein the multiple support samples are obtained for a single few-shot learning (FSL) episode, and the dummy prototype representation is an episode-known dummy prototype representation.

[0173]

[0161] Aspect 11: An apparatus for processing one or more data samples, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: determine, based on a plurality of support samples associated with the one or more classes, one or more prototype representations, each prototype representation associated with one of the one or more classes; determine a dummy prototype representation, where the one or more prototype representations and the dummy prototype representation are determined in a same learned metric space; determine, for each query sample among the one or more query samples, one or more distance metrics, where the one or more distance metrics are determined based on the one or more prototype representations and the dummy prototype representation; and classify each query sample based on the one or more distance metrics, where each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the dummy prototype representation.

[0174]

[0162] Aspect 12: The apparatus described in aspect 11, wherein to classify each query sample based on one or more distance metrics, at least one processor is configured to determine a probability distribution over one or more classes and an open set class, where the probability distribution is determined based at least in part on a Euclidean distance metric, classify each query sample into one of the one or more classes or into an open set class based on the probability distribution, and perform open set rejection (OSR) based on the set of classified query samples classified into open set classes associated with the dummy prototype representation.

[0175]

[0163] Aspect 13: An apparatus described in aspect 11 or 12, wherein one or more classes are keyword classes, and at least one processor is configured to determine each prototype representation using a plurality of support samples associated with the keyword class.

[0176]

[0164] Aspect 14: The apparatus described in aspect 13, wherein at least one processor is further configured to use a neural network encoder to generate multiple embeddings for multiple support samples associated with a keyword class, the multiple embeddings including an embedding for each support sample among the multiple support samples, and to determine each prototype representation based on an average of the multiple embeddings generated for the multiple support samples.

[0177]

[0165] Aspect 15: The apparatus of aspect 14, wherein at least one processor is configured to determine one or more prototype representations and a dummy prototype representation within the same learned embedding space of the neural network encoder.

[0178]

[0166] Aspect 16: An apparatus described in any one of aspects 11 to 15, wherein at least one processor is further configured to determine a dummy prototype representation using a dummy generator network, the dummy generator network including one or more linear layers and a max pooling layer, the one or more linear layers receiving one or more prototype representations as inputs, and the max pooling layer receiving outputs of the one or more linear layers as inputs.

[0179]

[0167] Aspect 17: The apparatus described in aspect 16, wherein to determine the dummy prototype representation, at least one processor is configured to generate a plurality of dummy prototypes using a dummy generator network and determine the dummy prototype representation by selecting one of the plurality of dummy prototypes based on Gumbel softmax.

[0180]

[0168] Aspect 18: The apparatus of any one of aspects 11 to 13, wherein the keyword class includes a closed set of known classes.

[0181]

[0169] Aspect 19: An apparatus described in any one of aspects 11 to 18, wherein the one or more query samples include speech audio data, and the multiple support samples include speech audio data labeled with a corresponding one of one or more classes for each support sample among the multiple support samples.

[0182]

[0170] Aspect 20: An apparatus described in any one of aspects 11 to 19, wherein the multiple support samples are obtained for a single few-shot learning (FSL) episode, and the dummy prototype representation is an episode-known dummy prototype representation.

[0183]

[0171] Aspect 21: A non-transitory computer-readable storage medium storing instructions which, when executed by at least one processor, cause the at least one processor to determine, based on a plurality of support samples associated with the one or more classes, one or more prototype representations, each prototype representation associated with one of the one or more classes; determine a dummy prototype representation, where the one or more prototype representations and the dummy prototype representation are determined in the same learned metric space; determine, for each query sample of the one or more query samples, one or more distance metrics, where the one or more distance metrics are determined based on the one or more prototype representations and the dummy prototype representation; and classify each query sample based on the one or more distance metrics, where each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the dummy prototype representation.

[0184]

[0172] Aspect 22: A non-transitory computer-readable storage medium as described in aspect 21, wherein, to classify each query sample based on one or more distance metrics, the instructions cause at least one processor to determine a probability distribution over one or more classes and an open set class, the probability distribution being determined at least in part based on a Euclidean distance metric, classify each query sample into one of the one or more classes or into an open set class based on the probability distribution, and perform open set rejection (OSR) based on the set of classified query samples classified into open set classes associated with the dummy prototype representation.

[0185]

[0173] Aspect 23: A non-transitory computer-readable storage medium as described in aspect 21 or 22, wherein one or more classes are keyword classes, and at least one processor is configured to determine each prototype representation using a plurality of support samples associated with the keyword class.

[0186]

[0174] Aspect 24: A non-transitory computer-readable storage medium as described in aspect 23, wherein the instructions further cause at least one processor to generate, using a neural network encoder, a plurality of embeddings for a plurality of support samples associated with a keyword class, the plurality of embeddings including an embedding for each support sample of the plurality of support samples, and determine each prototype representation based on an average of the plurality of embeddings generated for the plurality of support samples.

[0187]

[0175] Aspect 25: A non-transitory computer-readable storage medium as described in aspect 24, wherein the instructions cause at least one processor to determine one or more prototype representations and a dummy prototype representation within the same learned embedding space of a neural network encoder.

[0188]

[0176] Aspect 26: A non-transitory computer-readable storage medium as described in any one of aspects 21 to 25, wherein the instructions further cause at least one processor to determine a dummy prototype representation using a dummy generator network, the dummy generator network including one or more linear layers and a max pooling layer, the one or more linear layers receiving one or more prototype representations as inputs, and the max pooling layer receiving outputs of the one or more linear layers as inputs.

[0189]

[0177] Aspect 27: A non-transitory computer-readable storage medium as described in aspect 26, wherein to determine a dummy prototype representation, the instructions cause at least one processor to generate a plurality of dummy prototypes using a dummy generator network and determine the dummy prototype representation by selecting one of the plurality of dummy prototypes based on a Gumbel softmax.

[0190]

[0178] Aspect 28: The non-transitory computer-readable storage medium according to any one of aspects 23 to 27, wherein the keyword class comprises a closed set of known classes.

[0191]

[0179] Aspect 29: A non-transitory computer-readable storage medium described in any one of aspects 21 to 28, wherein the one or more query samples include speech audio data, and the multiple support samples include speech audio data labeled with a corresponding one of one or more classes for each support sample among the multiple support samples.

[0192]

[0180] Aspect 30: A non-transitory computer-readable storage medium described in any one of aspects 21 to 29, wherein multiple support samples are obtained for a single few-shot learning (FSL) episode and the dummy prototype representation is an episode-known dummy prototype representation.

[0193]

[0181] Aspect 31: A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations recited in any one of aspects 1 to 30.

[0194]

[0182] Aspect 32: An apparatus for processing one or more data samples, comprising one or more means for performing the operations recited in any one of aspects 1 to 30.

Claims

1. A method for processing one or more data samples for few-shot learning open set keyword spotting, comprising: determining one or more prototype representations based on a plurality of support samples associated with one or more keyword classes of a keyword spotting system, each prototype representation being associated with one of the one or more keyword classes, the plurality of support samples including, for each support sample of the plurality of support samples, speech audio data labeled with a corresponding one of the one or more keyword classes; determining a dummy prototype representation, wherein the one or more prototype representations and the dummy prototype representation are determined in the same learned metric space; determining, for each query sample of one or more query samples comprising speech audio data, one or more distance metrics comprising a distance metric between the respective query sample and the one or more prototype representations and a distance metric between the respective query sample and the dummy prototype representations; and classifying each query sample based on the one or more distance metrics, wherein each query sample falls into one of the one or more keyword classes associated with the one or more prototype representations or into an open set class associated with the dummy prototype representation.

2. Classifying each query sample based on the one or more distance metrics includes: determining a probability distribution over the one or more keyword classes and the open set class, the probability distribution determined based at least in part on a Euclidean distance metric; classifying each query sample into one of the one or more keyword classes or into the open set class based on the probability distribution; and performing open set rejection (OSR) based on the set of classified query samples classified into the open set classes associated with the dummy prototype representations.

3. Each prototype representation is determined using a plurality of supporting samples associated with a keyword class, the method comprising: generating a plurality of embeddings for the plurality of support samples associated with the keyword class using a neural network encoder, the plurality of embeddings including an embedding for each support sample of the plurality of support samples; The method of claim 1 , further comprising: determining each prototype representation based on an average of the multiple embeddings generated for the multiple support samples.

4. The method of claim 3 , wherein the one or more prototype representations and the dummy prototype representations are determined within the same learned embedding space of the neural network encoder.

5. determining the dummy prototype representation using a dummy generator network; the dummy generator network includes one or more linear layers and max pooling layers; the one or more linear layers receive as input the one or more prototype representations; The method of claim 1 , wherein the max pooling layer receives as input the outputs of the one or more linear layers.

6. determining the dummy prototype representation generating a plurality of dummy prototypes using the dummy generator network; The method of claim 5 , further comprising: determining the dummy prototype representation by selecting one of the plurality of dummy prototypes based on a Gumbela softmax.

7. An apparatus for processing one or more data samples for few-shot learning open-set keyword spotting, comprising: at least one memory; at least one processor coupled to the at least one memory; wherein the at least one processor: determining one or more prototype representations based on a plurality of support samples associated with one or more keyword classes of a keyword spotting system, each prototype representation being associated with one of the one or more keyword classes, the plurality of support samples including, for each support sample of the plurality of support samples, speech audio data labeled with a corresponding one of the one or more keyword classes; determining a dummy prototype representation, wherein the one or more prototype representations and the dummy prototype representation are determined in the same learned metric space; determining one or more distance metrics for each query sample of one or more query samples comprising speech audio data, the one or more distance metrics comprising a distance metric between the respective query sample and the one or more prototype representations and a distance metric between the respective query sample and the dummy prototype representation; the apparatus is configured to classify each query sample based on the one or more distance metrics, wherein each query sample falls into one of the one or more keyword classes associated with the one or more prototype representations or an open set class associated with the dummy prototype representation.

8. To classify each query sample based on the one or more distance metrics, the at least one processor: determining a probability distribution over the one or more keyword classes and the open set class, the probability distribution determined based at least in part on a Euclidean distance metric; classifying each query sample into one of the one or more keyword classes or into the open set class based on the probability distribution; The apparatus of claim 7 , configured to perform open set rejection (OSR) based on a set of classified query samples classified into the open set classes associated with the dummy prototype representations.

9. The apparatus of claim 7, wherein the at least one processor is configured to determine each prototype representation using a plurality of supporting samples associated with a keyword class.

10. The at least one processor generating a plurality of embeddings for the plurality of support samples associated with the keyword class using a neural network encoder, the plurality of embeddings including an embedding for each support sample of the plurality of support samples; further configured to determine each prototype representation based on an average of the plurality of embeddings generated for the plurality of support samples; 10. The apparatus of claim 9, wherein the at least one processor is configured to determine the one or more prototype representations and the dummy prototype representation within a same learned embedding space of the neural network encoder.

11. the at least one processor is further configured to determine the dummy prototype representation using a dummy generator network; the dummy generator network includes one or more linear layers and max pooling layers; the one or more linear layers receive as input the one or more prototype representations; The apparatus of claim 7 , wherein the max pooling layer receives as input the output of the one or more linear layers.

12. To determine the dummy prototype representation, the at least one processor: generating a plurality of dummy prototypes using the dummy generator network; The apparatus of claim 11 , configured to determine the dummy prototype representation by selecting one of the plurality of dummy prototypes based on a Gumbela softmax.

13. The apparatus of claim 9 , wherein the keyword class comprises a closed set of known classes.

14. The method of claim 1 or the apparatus of claim 7, wherein the plurality of support samples are obtained for a single few-shot learning (FSL) episode, and the dummy prototype representation is an episode-known dummy prototype representation.

15. A computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method according to any one of claims 1 to 8.

Images