Dummy prototype network for small sample open set keyword detection

Through the virtual prototype network, the audio signal keyword detection is solved in the existing technology, the problem of insufficient detection performance is achieved, and higher accuracy and efficiency are achieved, and no classes are effectively identified and rejected.

CN118946929APending Publication Date: 2024-11-12QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380026693.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-12-07
Filing Date
2023-01-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art performs audio signal keyword detection under small sample conditions, and the detection performance is insufficient, especially when the user receives a small number of keyword vocabulary, resulting in low accuracy and efficiency.

Method used

Using the virtual prototype network (D-ProtoNet) combined with the machine learning classifier, the prototype representation and dummy prototype representation are generated based on the support samples, and the distance metric is calculated to query samples in a classified manner, reject open set queries that do not belong to known classes, and improve keyword detection. accuracy and efficiency.

Benefits of technology

Under small sample conditions, the accuracy and efficiency of audio signal keyword detection are significantly improved, and the identification and rejection of unknown classes are effectively handled, reducing false positive and false negative errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118946929A_ABST
    Figure CN118946929A_ABST
Patent Text Reader

Abstract

Systems and techniques for processing audio data are provided. For example, small sample open set keyword detection (FSOS-KWS) may be performed using a dummy prototype network. The process may include determining one or more prototype representations based on a plurality of support samples associated with one or more classes. Each prototype representation may be associated with one of the classes. A dummy prototype representation may be determined in the same learning metric space as the prototype representation. One or more distance metrics may be determined for each of the one or more query samples. The distance metric may be based on the prototype representation and the dummy prototype representation. Each query sample may be classified based on the distance metric. Each query sample may be classified into one of the classes associated with the prototype representation or into an open set class associated with the dummy prototype representation.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 321,677, filed on March 19, 2022, which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to audio signal processing. For example, aspects of the present disclosure relate to systems and techniques for performing open set recognition (OSR) keyword detection (KWS) using one or more machine learning systems (e.g., including one or more neural network-based models). Background Art

[0004] Electronic devices such as smart phones, tablet computers, wearable electronic devices, smart TVs, etc. are becoming more and more popular among consumers. These devices can provide voice and / or data communication functions through wireless or wired networks. In addition, such electronic devices may include other features that provide various functions designed to enhance user convenience. The electronic device may include a voice recognition function for receiving voice commands from the user. When the voice command from the user is received and recognized, such functions allow the electronic device to perform functions associated with the voice command (e.g., keywords). For example, the electronic device may activate a voice assistant application, play an audio file, or take a picture in response to a voice command from the user.

[0005] In electronic devices with speech recognition features, manufacturers or operators can usually equip these devices with predetermined keywords and associated sound models, which can be used to detect keywords in input sounds. Some electronic devices can also allow users to specify keywords as voice commands. For example, an electronic device can receive several utterances of keywords from a user and generate a keyword model for specifying keywords from the utterances. The detection performance of a keyword model can be related to the number of utterances from which the keyword model is generated. The detection performance of a keyword model can improve as the number of utterances increases. For example, a manufacturer can provide a keyword model that has been generated from thousands or more utterances in an electronic device.

[0006] However, in some electronic devices, the number of utterances of keywords received from a user is relatively small (e.g., less than five). Therefore, a keyword model generated from such a limited number of utterances may not produce sufficient detection performance. However, receiving a large number of utterances from a user to generate a keyword model that can provide sufficient detection performance may be time-consuming and inconvenient for the user. Summary of the invention

[0007] In some examples, systems and techniques for Few-Shot Open Set Recognition (FSOSR) are described. In some examples, these systems and techniques can be used to perform keyword detection by identifying and rejecting open set queries that do not belong to known or seen classes. In some aspects, these systems and techniques can utilize a fictitious prototype network to perform Few-Shot Open Set Keyword Detection (FSOS-KWS).

[0008] According to at least one example, a method for processing one or more data samples is provided, the method comprising: determining one or more prototype representations based on a plurality of support samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determining a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; and classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the fictitious prototype representation.

[0009] In another example, a device for processing one or more data samples is provided, the device comprising a memory (e.g., configured to store data, such as audio data, etc.) and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured and capable of: determining one or more prototype representations based on a plurality of support samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determining a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; and classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the fictitious prototype representation.

[0010] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: determine one or more prototype representations based on a plurality of support samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determine a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determine one or more distance metrics for each of one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; and classify each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the fictitious prototype representation.

[0011] In another example, a device for processing one or more data samples is provided. The device includes: a device for: determining one or more prototype representations based on a plurality of support samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determining a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; and classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the fictitious prototype representation.

[0012] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0013] The foregoing, as well as other objects and advantages associated with the aspects disclosed herein, will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are presented to aid in describing various aspects of the present disclosure and are provided solely for illustration and not limitation of the various aspects.

[0015] Figure 1 illustrates example implementations of a system on a chip (SoC) according to some examples;

[0016] Figure 2A and Figure 2B An example of a fully connected neural network according to some examples is illustrated;

[0017] Figure 3 is a diagram illustrating an example of a few-shot learning (FSL) scenario according to some examples;

[0018] Figure 4A is a diagram illustrating another example of a few-shot learning (FSL) scenario according to some examples;

[0019] Figure 4B is a diagram illustrating a first example episode in a few-shot open set recognition (FSOSR) episode according to some examples;

[0020] Figure 4C is a diagram illustrating a second example episode in a few-shot open set recognition (FSOSR) episode according to some examples;

[0021] Figure 5 is a diagram illustrating an example architecture of a fictitious prototype network according to some examples;

[0022] Figure 6 is a flow chart illustrating an example of a process for processing one or more data samples (eg, of an audio signal) to perform keyword detection (KWS); and

[0023] Figure 7 is a block diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0024] Provided below are some aspects and examples of the present disclosure. As will be apparent to those skilled in the art, some of these aspects and examples can be applied independently, and some of them can be applied in combination. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects and examples of the application. However, it is apparent that various aspects and examples can be practiced without these specific details. Each drawing and description are not intended to be restrictive.

[0025] The following description provides example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the exemplary aspects will provide those skilled in the art with an enabling description of various examples and aspects for implementing the present disclosure. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0026] Voice activity detection may refer to the detection of human voice by a computing device in order to perform a function. For example, a computing device may use voice activity detection (such as by identifying specific keywords) to wake up or activate the device, query the device, and / or cause the device to perform various other functions. Voice activity detection may also be used for more complex functions, such as far-field sound detection (e.g., from a mobile device placed on the other side of the room), user authentication (e.g., through a voice signature), sound detection during other audio output (e.g., detecting voice commands when music is playing on the device or detecting interrupt commands when a smart assistant is speaking), and / or sound interaction in complex noise environments (such as in a moving vehicle).

[0027] Keyword detection (KWS) is the task of detecting one or more keywords in an audio signal (e.g., an audio signal including human speech or spoken words). For example, KWS can be used to distinguish activation phrases or specific commands from other speech and noise in the audio signal. In some cases, the KWS system can be targeted at or utilized by edge devices such as mobile phones and smart speakers. The detected keywords can include single words, compound words, phrases including multiple words, etc. In some cases, KWS can be performed based on a set of predetermined keywords and / or a set of user-defined keywords. In some cases, user-defined keywords may include one or more adaptations, adjustments, etc. determined based on specific characteristics of the sound or voice of a given user.

[0028] The audio signal may be a streaming audio signal, and in some examples, KWS may be performed in real time for the streaming audio signal. The streaming audio signal may be recorded by a microphone associated with a computing device or obtained from the microphone. KWS may be performed locally or remotely. For example, KWS may be performed locally using one or more processors of the same computing device that collects or obtains the streaming audio signal. In some examples, KWS may be performed remotely by sending the streaming audio signal (or its representation) from a local computing device to a remote computing device (e.g., the local computing device records the audio signal, but the KWS processing task is offloaded to the remote computing device). Performing KWS locally may reduce the total latency or computation time, but may reduce accuracy. Performing KWS remotely may result in higher latency, but may improve accuracy.

[0029] For example, a local computing device (e.g., a smartphone) typically has less computing power than a remote computing device (e.g., a cloud computing system), and may therefore generate KWS results with lower accuracy or overall performance, particularly under time constraints associated with providing KWS results in real time or near real time. For example, in order to provide real-time KWS results, a local computing device may implement a KWS detection model that is less complex than those models implemented on a remote computing device. Lower accuracy KWS results may include false positives (e.g., identifying keywords that do not actually exist), false negatives (e.g., failing to identify keywords that exist), and classification errors (e.g., identifying a first keyword as some other keyword).

[0030] However, performing KWS remotely may introduce communication delays that may offset the accuracy gains associated with remote KWS. For example, remote KWS may introduce delays along the communication path from the local computing device to the remote computing device (e.g., the time to send a streaming audio signal or a representation of the streaming audio signal to the remote computing device) and along the return communication path from the remote computing device to the local computing device (e.g., the time to send KWS results from the remote computing device back to the local computing device).

[0031] As described above, in some examples, KWS is typically performed in real time (or near real time) to allow a user to interact with one or more computing devices. The lag between the time a user speaks a keyword (e.g., an activation phrase or a particular command) and the time a computing device provides a corresponding response or action can be an important factor in a user's willingness to utilize verbal commands. In some cases, multi-second lags may frustrate users or otherwise dissuade them from using verbal commands. Therefore, there is a need to improve KWS performance in local and / or remote KWS implementations, as both local and remote KWS implementations are typically time-limited processes.

[0032] In some approaches to KWS, one or more machine learning networks are used to detect keywords in an audio signal based on a set of predefined keyword categories. For example, KWS may be performed using one or more neural networks trained on the set of predefined keyword categories. When evaluated against a closed set of keywords seen in training, the resulting neural network may achieve accurate performance, but is typically limited to detecting only those keywords seen in training (e.g., keywords corresponding to the keyword categories in the training dataset).

[0033] In some cases, small-sample KWS can be used to detect or classify keywords that were not initially seen in training. In an example FSL-based approach to KWS, a neural network classifier is trained on a training set that includes labeled samples representing various predefined classes or keyword categories. During training, the neural network classifier absorbs knowledge from the labeled samples of the training set. For example, the neural network classifier can learn an embedding or embedding space for associating training samples with corresponding labels for the training samples. The trained neural network classifier can then (e.g., at inference) adapt or generalize to the evaluation task of new classes (e.g., unseen classes) that were not seen in the training set.

[0034] For example, assuming that there are only a small number of support examples for each new class, an FSL-based neural network classifier can use an attention mechanism on a learned embedding of a labeled set of support examples (e.g., a support set) to predict the class of an unlabeled point (e.g., a query set). In a one-by-one query process, a trained FSL-based neural network classifier can receive a support set including M support examples for each of N unseen classes, and a query set including one or more query samples. The neural network classifier can determine a prototype representation for each unseen class N (e.g., using the M support examples associated with each unseen class N). Each unlabeled query sample can be classified into one of the previously unseen classes N based on the calculated distance between the query sample and each prototype representation. In some cases, this inference process can be referred to as an N-class M-example scenario, where the goal of the FSL-based neural network classifier is to correctly classify the query set into N classes that were not seen during training but are known using the M support samples.

[0035] In real-world scenarios where KWS can be utilized, the input audio signal may naturally include utterances, words, phrases, etc. of unexpected new classes for which no supporting examples are given. An unexpected new class may be a class that does not include supporting examples in the initial training set or support set. These unexpected samples and / or classes may be referred to as "open sets" (e.g., in contrast to the closed sets of classes from the initial training set and support set).

[0036] A neural network classifier may exhibit overconfidence and incorrectly classify unseen open set samples into one of the FSL classes. In some examples, the accuracy and / or performance of the KWS system may be improved by rejecting open set utterances instead of classifying them into one of the (e.g., closed set) predetermined keyword classes. For example, open set recognition (OSR) may be used to detect unseen open set classes, and then process and / or reject query samples belonging to the detected open set classes.

[0037] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") for performing small sample open set recognition (FSOSR) and / or small sample open set keyword detection (FSOS-KWS) using a machine learning classifier. In some examples, the machine learning classifier may be a neural network classifier. The small sample neural network classifier may be used to perform improved keyword detection (KWS) using OSR by predicting episode-specific (e.g., episode-known) dummy classes based on supporting examples in each episode. In some examples, the small sample neural network classifier may classify the predicted dummy classes as open sets for rejection from downstream or subsequent KWS processes, as will be described in more depth below. In some aspects, the neural network described herein may be referred to as and / or included in a dummy prototype network (D-ProtoNet).

[0038] In some examples, these systems and techniques can generate a set of prototype representations based on a support set provided as input during inference. The support set can include multiple support examples associated with known keyword classes. For example, the support set can include example utterances labeled with the corresponding keyword class of the utterance. The machine learning encoder can generate embedded support examples from multiple support examples included in the support set. The machine learning encoder can be a neural network encoder. In some examples, the embedded support examples for each known class (e.g., each class seen in the labeled support set) can be used to generate or otherwise determine a prototype representation for each class. In some cases, the prototype representation can be determined as the average of the embedded support examples for each class.

[0039] In some examples, a machine learning (e.g., neural network) encoder may also receive a query set as input, wherein the query set includes a combination of query samples belonging to known classes and query samples belonging to unknown classes. The neural network encoder may generate embeddings for the query samples (e.g., may generate embedded query samples), which may then be analyzed against the prototype representation to perform classification.

[0040] In some examples, one or more dummy prototypes (e.g., dummy prototype representations) can be generated based on a set of prototype representations of each known class in the known classes. The dummy prototypes can be used to classify an open set query sample as not belonging to any of the known classes in the known classes. For example, the open set query sample can be rejected or removed as an outlier. In some examples, the dummy prototypes can be used to perform small sample open set keyword detection (FSOS-KWS), as described in more depth below.

[0041] An illustrative example of a fictitious protocol network for small-shot open-set keyword detection is described in Appendix A attached hereto.

[0042] Various aspects of the disclosure will be described with respect to the figures. Figure 1 An example implementation of a system on a chip (SoC) 100 is illustrated, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed among multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from the memory block 118.

[0043] The SoC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 (which may, for example, detect and recognize gestures, voice, and / or other interactive user actions or inputs). In one specific implementation, the NPU is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SoC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120 (which may include a global positioning system). In some examples, the sensor processor 114 may be associated with or connected to one or more sensors for providing sensor input to the sensor processor 114. For example, the one or more sensors and the sensor processor 114 may be disposed in, coupled to, or otherwise associated with the same computing device.

[0044] In some examples, the one or more sensors may include one or more microphones for receiving sound (e.g., audio input), including sound or audio input that can be used to perform keyword detection (KWS) as described herein. In some cases, the sound or audio input received by the one or more microphones (and / or other sensors) may be digitized into data packets for analysis and / or transmission. The audio input may include ambient sound near a computing device associated with the SoC 100 and / or may include speech from a user of a computing device associated with the SoC 100. In some cases, additionally or alternatively, a computing device associated with the SoC 100 may be communicatively coupled to one or more peripheral devices (not shown) and / or configured to communicate with one or more remote computing devices or external resources, for example using a wireless transceiver and a communication network such as a cellular communication network.

[0045] SoC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into CPU 102 may include code for searching a lookup table (LUT) for a stored multiplication result corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into CPU 102 may also include code for disabling a multiplier during a multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. In addition, the instructions loaded into CPU 102 may include code for storing a calculated multiplication product of an input value and a filter weight when a lookup table miss of the multiplication product is detected.

[0046] SoC 100 and / or its components may be configured to perform audio signal processing according to aspects of the present disclosure and using one or more machine learning networks and / or machine learning techniques as described herein. For example, SoC 100 and / or its components may be configured to perform keyword detection according to aspects of the present disclosure. In some examples, aspects of the present disclosure may improve the accuracy and / or efficiency of keyword detection (KWS) by using one or more neural network classifiers to perform small sample learning (FSL) and / or open set recognition (OSR).

[0047] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems may include algorithms and statistical models that a computer system can use to perform various tasks by relying on patterns and inferences without the use of explicit instructions. An example of an ML system is a neural network (also called an artificial neural network), which may include a set of interconnected artificial neurons (e.g., a neuron model). Neural networks can be used in various applications and / or devices, such as speech analysis, audio signal analysis, image and / or video decoding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, etc.

[0048] Individual nodes in a neural network can simulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​constrain the way the input data is related to the output data. For example, the input data of each node can be multiplied by the corresponding weight value, and the products can be added. The sum of the products can be adjusted by an optional bias, and an activation function can be applied to the result, thereby producing an output signal or "output activation" (sometimes called a feature map or activation map) of the node. The weight values ​​can initially be determined by an iterative flow of training data through the network (for example, the weight values ​​are established during a training phase in which the network learns how to identify a particular class by the typical input data characteristics of the particular class).

[0049] There are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multi-layer perceptron (MLP) neural networks, transformer neural networks, and the like. For example, a convolutional neural network (CNN) is a feed-forward artificial neural network. A convolutional neural network may include a collection of artificial neurons that each have a receptive field (e.g., a spatially local region of an input space) and collectively tile the input space. RNNs work on the principle of saving the output of a layer and feeding that output back to the input to help predict the results of that layer. GANs are generative neural networks that learn patterns in input data so that a neural network model can generate new synthetic outputs that are reasonably likely to come from the original data set. GANs may include two neural networks operating together, including a generative neural network that generates a synthetic output and a discriminative neural network that evaluates the authenticity of the output. In an MLP neural network, data may be fed to an input layer, and one or more hidden layers provide a level of abstraction for the data. The output layer may then be predicted based on the abstract data.

[0050] Deep learning (DL) is an example of a machine learning technique and can be considered a subset of ML. Many DL methods are based on neural networks, such as RNN or CNN, and utilize multiple layers. The use of multiple layers in a deep neural network allows for the gradual extraction of higher-level features from a given raw data input. For example, the output of the first layer of artificial neurons becomes the input of the second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input of the third layer of artificial neurons, and so on. The layer located between the input and output of the entire deep neural network is generally referred to as a hidden layer. Hidden layer learning (e.g., being trained) transforms the intermediate input from the previous layer into a slightly more abstract and complex representation that can be provided to subsequent layers until the final or desired representation is obtained as the final output of the deep neural network.

[0051] As mentioned above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and output is generated by the output nodes of the output layer. A deep learning network typically includes multiple hidden layers. Each layer of the neural network may include a feature map or activation map, which may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. A node may include one or more weights for indicating the importance of a node in one or more layers in the layer. In some cases, a deep learning network may have a series of many hidden layers, where early layers are used to determine simple and low-level characteristics of the input, and later layers establish a hierarchical structure of more complex and abstract characteristics.

[0052] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or sound combinations for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases. Deep learning architectures can perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher layers to recognize cars, trucks, and airplanes.

[0053] Neural networks can be designed to have a variety of connectivity patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0054] The connections between neural network layers can be fully connected or partially connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In a fully connected neural network 202, a neuron in a first layer may communicate the output of that neuron to each neuron in a second layer, such that each neuron in the second layer will receive input from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, a locally connected layer of the locally connected neural network 204 may be configured such that each neuron in the layer will have the same or similar connectivity pattern, but may be configured with connection strengths that may have different values ​​(e.g., 210, 212, 214, and 216). The locally connected connectivity patterns may produce spatially distinct receptive fields in higher layers because higher layer neurons in a given region may receive input that is tuned through training to properties of a limited portion of the total input to the network.

[0055] Figure 3 is a diagram illustrating an example of a few-shot learning (FSL) scenario 300. In some examples, FSL and FSL-based methods can be used to perform few-shot classification, where a classifier (e.g., a neural network or other machine learning classifier) ​​can generalize and extend inference to include new classes not seen in the training set, provided that each new class has only a small number of examples. For example, as previously described, a FSL-based neural network classifier can be used to perform few-shot KWS for detecting keywords in an audio stream or audio data.

[0056] As shown in the figure, Figure 3An example of a 3-class 5-example FSL scenario is depicted, where a neural network classifier (not shown) receives as input a support set comprising a total of 15 labeled support examples, which is partitioned into three previously unseen classes (e.g., 310, 320, and 330) with five support examples per class. In an illustrative example, the 3-class 5-example FSL scenario 300 can be associated with a prototype network that learns a metric space in which classification can be performed by computing distances to prototype representations of each class.

[0057] Using prototypical networks for FSL can be based on the idea that there exists an embedding where points for each class are clustered around a single prototypical representation. Figure 3 As illustrated, it can be seen that the five supporting examples included in class 310 are clustered around prototype representation 315; it can be seen that the five supporting examples included in class 320 are clustered around prototype representation 325; it can be seen that the five supporting examples included in class 330 are clustered around prototype representation 335; and so on. The prototype network may include one or more neural networks that learn (e.g., during training) a nonlinear mapping of inputs in an embedding space. Using the learned embedding space, the prototype network can take the prototype of each previously unseen class as the average of the support set of the prototype network in the embedding space. For example, prototype representation 315 can be determined as the average of the five embedded supporting examples of class 310; prototype representation 325 can be determined as the average of the five embedded supporting examples of class 320; prototype representation 335 can be determined as the average of the five embedded supporting examples of class 330; and so on.

[0058] Classification may then be performed on the embedded query point by determining the class prototype that is closest to the query point. For example, embedded query point 370 may be classified into class 320 based on determining that the distance from embedded query point 370 to prototype representation 325 is less than the distance from embedded query point 370 to any of the remaining prototype representations 315 and 335. In some examples, embedded query points (e.g., embedded query point 370) may be classified based on the Euclidean distance between the embedded query point and each of the prototype representations, but it should be noted that various other distance metrics and / or determinations may also be utilized without departing from the scope of the present disclosure.

[0059] Figure 4A This is an example of using the above Figure 34 is a diagram of another example of an FSL scenario 400a for a prototypical network (ProtoNet) described above. As previously described, in an N-class M-example scenario (e.g., a round of inference for a support set containing N*M support examples), FSL can be performed with the goal of correctly classifying one or more queries of a query set into N classes that were unseen during training but are known based on receiving M support samples for each previously unseen class. In some examples, a purely FSL-based approach does not consider open-set classes among the N classes, meaning that a purely FSL-based approach (such as the purely FSL-based approach of FSL scenario 400a) will only classify a query into one of the N closed-set classes.

[0060] In an illustrative example, the FSL scenario 400a is a 3-class 5-example FSL scenario, which means that classification is performed on three classes 410, 420, and 430 in an evaluation (e.g., inference) setting, and means that corresponding prototype representations 415, 425, and 435 are generated using five embedded support examples for each of the three classes. In the context of FSL-based KWS, the three classes 410, 420, and 430 may each represent a keyword or a keyword classification. For example, class 410 may represent the keyword "one", class 420 may represent the keyword "two", and class 430 may represent the keyword "three". Because the FSL scenario 400a is a 3-class 5-example FSL scenario, the support set S (not shown) may include a total of 15 support examples (e.g., five example utterances of "one" marked with the keyword classification 410, five example utterances of "two" marked with the keyword classification 420, and five example utterances of "three" marked with the keyword classification 430).

[0061] like Figure 4A As illustrated, the query set Q of the test utterances may include a combination of closed set queries and open set queries. The closed set queries may represent test utterances from known classes (e.g., test utterances of "one", "two", or "three" corresponding to the support set classes 410, 420, 430, respectively). Figure 4AAs illustrated, the query set includes three closed-set test utterances for each class 410, 420, 430. An open-set query may represent a test utterance from an unknown class, and is depicted as an open circle containing a "?" inside. For example, an open-set query may include test utterances such as "four," "five," "six," etc., but it should be noted that open-set queries / test utterances may take various other forms with varying degrees of similarity to closed-set test utterances (e.g., because an open set may be defined to include any test utterance that does not match a known class). The test utterances of the open-set query may be associated with one or more unseen and unsupported classes that were not seen in an initial training dataset (e.g., used to initially train a neural network classifier associated with the FSL scenario 400a) or a support set S (e.g., provided to a neural network classifier at inference time and used to perform FSL).

[0062] Because the open set class is unseen, unknown, or otherwise unsupported by the neural network classifier associated with the FSL scenario 400a, each of the open set queries will be misclassified into one of the three known (e.g., closed set) classes 410, 420, 430. For example, Figure 4A The decision boundary depicted as a dashed line dividing the embedding space in can represent a (wrong) closed-set classification that can be determined for an open-set query. In some examples, the misclassification of queries from open-set classes can occur due to the design of existing FSL methods (e.g., such as FSL scenario 400a), which are limited to known closed-set classes and cannot consider any open-set classes regardless of the distance between the open-set query and the nearest prototype representation determined for one of the closed-set classes.

[0063] In some real-world scenarios and / or KWS implementations, unseen and unsupported classes such as "four", "five", and "six" may be frequently encountered and are consistently misclassified by existing FSL methods as belonging to known or seen classes from the closed set of FSL classes seen in training or included in the support set S. It is desirable to handle or reject any unseen and unsupported classes during KWS classification so that new open-set classes are not misclassified into the seen closed-set classes.

[0064] Figure 4B A first example episode 400b of a few-shot open set recognition (FSOSR) method according to one or more aspects of the present disclosure is illustrated. Figure 4C A second example scenario 400c of the FSOSR method according to one or more aspects of the present disclosure is illustrated. In an illustrative example, the first FSOSR scenario 400b and the second FSOSR scenario 400c may represent or be applied to the same method as described above with respect to Figure 4AThe illustrated example FSL scenario 400a describes the same basic sample set S and query set Q.

[0065] The systems and techniques described herein can perform FSOSR, including FSOSR for KWS (e.g., FSOS-KWS), to distinguish unknown open sets from known closed classes while still performing FSL. For example, these systems and techniques can receive as input a query set that includes a combination of closed-set test utterances (e.g., which should be correctly classified as belonging to one of the known classes) and open-set test utterances (e.g., which belong to one or more unknown classes and should not be classified as belonging to any of the known classes). As will be described in more depth below, these systems and techniques can perform FSOSR and / or FSOS-KWS using metric learning-based episode-known fictitious prototypes to detect and reject open sets. In some examples, episode-specific (e.g., episode-known) fictitious classes are predicted based on support examples included in a support set provided for each episode, and open sets are classified as fictitious classes and rejected for inclusion in a subsequent KWS.

[0066] In one illustrative example, the open set of test utterances with unknown or unseen classes can vary from episode to episode based on the selection of N classes in the support set S provided as inference input for each episode. Figure 4B A first episode 400b is illustrated, which has a first query open set with unseen classes, Figure 4C A second episode 400c is illustrated, which has a second query open set with unseen classes. The first episode 400b and the second episode 400c are built on the same base query set (e.g., Figure 4B and Figure 4C 400c). Based on the selection of N classes identified or included in the support set provided for each of episode 400b and episode 400c, the composition of the open set changes from episode 400b to episode 400c. The composition of the closed set classification also changes from episode 400b to episode 400c. For example, the first classification 410b includes a different set of query / test utterances than the first classification 410c; the second classification 420b includes a different set of query / test utterances than the second classification 420c; the third classification 430b includes a different set of query / test utterances than the third classification 430c; and so on.

[0067] As will be described in more depth below, the systems and techniques described herein can perform FSOS-KWS to distinguish unknown open sets from known closed sets in an episode-wise manner (e.g., for any given selection of N classes associated with a support set S provided to a given episode, the unknown open sets can be uniquely distinguished from the known closed sets). These systems and techniques predict episode-specific (e.g., episode-known) hypotheses based on the support examples and / or support set S provided as input to each episode. The predicted episode-specific hypotheses are then classified as episode-specific open sets, which can then be rejected or otherwise excluded from the KWS classification. In an illustrative example, these systems and techniques can utilize a hypothetical prototype network (D-ProtoNet), as will be described in more depth below.

[0068] Figure 5 is a diagram illustrating an example of a dummy prototype network (D-ProtoNet) 500 for performing FSOSR and / or FSOS-KWS with episode-specific open set recognition and / or rejection. In an illustrative example, the dummy prototype network 500 can perform FSOS-KWS and adapt to varying open sets between episodes. In some aspects, the dummy prototype network 500 can perform FSOS-KWS by using metric-based learning to learn a metric space in which a distance metric can classify samples.

[0069] In the context of the following discussion, the FSOSR setting may include visible training data D that does not overlap with the class train and no evaluation data D eval For example, it can be seen that the training data D train Can include multiple labeled samples where x i is the input feature, and y i is its corresponding label. In the example of FSOS-KWS, the input feature x i It can be speech or other speech and / or audio data. The corresponding label y i It can be the input feature x i Keyword class or category.

[0070] In one illustrative example, training can be performed in which a machine learning model (e.g., a neural network model) associated with the fictitious prototype network 500 learns from a plurality of N-class, M-example pseudo FSOSR episodes. For example, each of the pseudo FSOSR episodes may include N known classes, each class having M supporting examples (e.g., such that each pseudo FSOSR episode includes a total of N*M supporting examples) and one or more pseudo unknown (e.g., pseudo open set) classes without any supporting examples.

[0071] The pseudo episodes used in training can be designed to simulate the FSOSR inference task by subsampling the classes and data points. Regarding the FSOSR inference task, inference can be performed on episodes associated with the support set S and the query set Q, respectively. For example, Figure 5 A support set S (indicated as 512) is illustrated, which includes a closed set K (indicated as 513) of known classes, providing supporting examples for the closed set.

[0072] Figure 5 Also illustrated is a query set Q (indicated as 516), which includes both a closed set K of known classes (indicated as set 517) and an open set U of unknown classes (indicated as set 519), but without supporting examples. It should be noted that although the closed set sample set 513 and the closed set of samples 517 correspond to the same underlying known class, the constituent samples in 513 are not necessarily the same as the constituent samples in the closed set 517.

[0073] In one illustrative example, the support set S (e.g., 512) may include M samples for each of the N classes, as described above. The query set Q (e.g., 516) may include one or more queries from the N known classes (e.g., set 517), and may further include N known classes (e.g., set 519). U One or more queries of unknown classes, where:

[0074] |S|=N·M

[0075] |Q|=(N+N U )·M Q

[0076] Here, M Q It can represent the number of queries for each class. During inference, the evaluation dataset D eval All classes in may not be seen by the virtual prototype network 500. In some examples, the evaluation dataset D eval It can be the same as the query set 516. It can be used to include N known classes with support samples (e.g., set 517) and N classes without support samples. U We perform inference on the episode with unknown open set classes (e.g., set 519). Note that in the inference episode, the evaluation dataset D eval (e.g., set 517) does not necessarily include M support examples for each of the N classes, as is the case with support set S (e.g., 512).

[0077] As previously described, the virtual prototype network 500 can perform FSOS-KWS based at least in part on one or more metric-based learning methods. For example, in N-class M example scenarios, the virtual prototype network 500 can use the average value n of the embedded support samples of each class to determine or otherwise obtain N prototypes. to compute the corresponding prototype for each class, where:

[0078]

[0079] Here, S n is a subset of the support set S (e.g., 512), whose label is n; |S n |=M; and f φ is an encoder with φ parameters. Figure 5 In the example, the prototype c n It can be obtained based on determining the average value of the embedded support samples of each class at the averaging engine 530. In an illustrative example, using c n symbols to represent the prototypes, while C symbols (for example, in Figure 5 As indicated at block 532 in FIG. 1 , f may represent a set of N prototypes. φ Depicted as an encoder 520 that receives support set 512 and query set 516 as input for each episode. In one illustrative example, encoder 520 can be a neural network or other machine learning network (e.g., prototype c ) that generates an embedding based on support set 512 and / or query set 516. n This can be done by using the encoder f φ 520 is determined by averaging the embeddings generated for each class N).

[0080] In one illustrative example, the encoder f φ 520 may include or implement a machine learning backbone. For example, the encoder f φ 520 may include or implement one or more of Conv4-64, ResNet-12, and / or BCResNet-8. φ In the example of using Conv4-64 architecture, the output dimension can be 768. φ In the example of using the ResNet-12 architecture, the output dimension can be 512. φ In the example of 520 using the BCResNet-8 architecture, the output dimension may be 256. In some cases, a Conv4-64 architecture may be associated with 768 dimensions (e.g., resulting in 768 dimensions that are greater than 64, the number of input channels of the architecture) based at least in part on the absence of global average pooling at the output of the architecture.

[0081] Based on the prototype n , the virtual prototype network 500 can determine or otherwise obtain the probability distribution over N known classes:

[0082]

[0083] Here, d(·) is a distance metric (e.g., Figure 5 In one illustrative example, the distance metric d(·) may be determined based on the Euclidean distance:

[0084] d(z,z')=||z-z'|| 2 Equation (3)

[0085] In some aspects, the distance metric d(·) can be determined or otherwise implemented using a distance metric other than the Euclidean distance (e.g., the Euclidean distance of equation (3)). In some examples, based on equation (2) and / or equation (3), the systems and techniques described herein can use the virtual prototype network 500 to minimize the negative log probability, which can be expressed as the negative log probability of the true class n

[0086] In an illustrative example, the probability distribution of equation (2) can be used to determine the distance between the query example x and the prototype representation c of each class by (e.g., using a distance metric d(·)) n d(·) (e.g., a Euclidean distance metric such as that given in equation (3)).

[0087] In one illustrative example, the systems and techniques described herein can further determine one or more dummy prototypes for which the scenario is known in order to handle (e.g., reject from KWS) an open set of variations on multiple FSOS-KWS scenarios. For example, the dummy prototype network 500 can also include a dummy generator 550, for determining or otherwise generating a fictitious prototype for a known scenario. 550 may be based on receiving a set of original prototypes C for a given episode (e.g., by using equation (1) to encode the encoder f φ 520 output embedding average generated prototype C set) as input to generate the plot known fictitious prototype c d As will be explained in more depth below, the dummy generator 550 generated dummy prototype c d It can be enhanced to the original prototype C = {c1,…,c N} to generate an enhanced (e.g., virtual enhanced) prototype set {c1,…,cN ,c d}.

[0088] In some examples, a set of N prototypes may be used in each inference episode of the virtual prototype network 500. where c∈R 1xD , the output dimension is f φ D. In some cases, for the virtual prototype network 500, the prototypes may be arranged invariantly with respect to each other. In one illustrative example, the virtual generator 550 can have parameters based on DeepSets In some aspects, the dummy generator 550 may be permutation-invariant. In one illustrative example, the dummy generator 550 can use or otherwise provide N prototypes C = [c1; c2; . . . ; c N ]∈R NxD As input to generate a dummy c d 552.

[0089] In an illustrative example, based on a fictitious c d 552 is generated using a prototype set C determined for each class of the episode (e.g., uniquely determined based on a support set and / or supporting examples provided for a given episode), the generated virtual set c d can be scenario-specific and scenario-known. As mentioned above, changes in the selection and / or composition of N can change the open set (e.g., as respectively Figure 4B and Figure 4C In some aspects, based on the fictitious c d 552 is generated based on a uniquely determined prototype set C for each class, assuming that c d 552 is not static. For example, the virtual c d 552 can vary with the open set of changes caused by changes in N and / or the support set S.

[0090] In an illustrative example, let c d 552 can be generated by a dummy generator 550 is generated as:

[0091]

[0092] Here, g1 consists of a fully connected (FC) layer with nonlinearity and g1(C)∈R with hidden dimension H NxH For example, g1 can be two linear layers with a nonlinear level between the two linear layers. In some examples, the dummy generator 550 may use g1 with FC-ReLU-FC with hidden D=32.

[0093] After g1, max pooling can be applied. For example, the max pooling of Eq. (4) operates on N (e.g., the number of prototypes / classes) and is multiplied by R 1xH Output features. The feature output from the maximum pooling can be combined with W g Perform matrix multiplication, which can be implemented as a learnable HxD matrix. The result of this matrix multiplication can be the generated virtual c d 552, of which c d ∈R 1xD .

[0094] As mentioned above, the enhanced prototype set {c1, ···, c N ,c d} can be achieved by adding a dummy prototype c d 552 (e.g., by a dummy generator 550) is attached to the original prototype set C (e.g., generated by the function generator f φ 520 and the averaging engine 530 generate, where the averaging engine 530 applies equation (1)) to obtain.

[0095] In one illustrative example, the label of the open set query can be set to correspond to the fictitious c d The N+1th label y of 552 d . Use the virtual enhanced prototype {c1,···,c N ,c d}, equation (2) can be modified to determine or otherwise obtain the probability distribution over the N+1 prototypes as follows:

[0096]

[0097] In some examples, the probability distribution over the N+1 prototypes (e.g., provided above in equation (5)) can be determined in the same or similar manner as the probability distribution over the N prototypes (e.g., described previously with respect to equation (2)). For example, the probability distribution over the N+1 prototypes can be determined using softmax (e.g., Figure 5 The illustrated N+1 softmax 560) is determined based on the same distance metric d(·) 540. In some examples, the distance metric 540 can be based on the Euclidean distance, as previously described (e.g., with respect to equation (3)). The N+1 softmax 560 can be followed by an N+1 class classification 570, which can generate an N class classification 580. In some aspects, the N+1 softmax 560 and the N+1 class classification 570 can be a combination of the N classes and the newly generated scenario-specific dummy class c.d In one example, N-class classification 580 may be based on an argmax function (such as Figure 5 The argmax shown (n∈{1,…,N}) p(y=n|x)).

[0098] In the example of equation (5), θ may represent or otherwise include the encoder 520 parameter φ and the dummy generator 550 parameter In some cases, τ n is the softmax temperature. For example, τ n may be a softmax temperature associated with or implemented by the N+1 softmax 560. In some examples, the softmax temperature may be kept constant or otherwise the same across different classes N.

[0099] In one illustrative example, N+1 softmax 560 may be used to set the label of the open set query to the N+1th label y d (e.g., as described above). In some aspects, with other τ n ≠N+1, a larger τ can be used N+1 (e.g., can be used in or by N+1 softmax 560), so that the dummy c d 552 affects the original N-class classification to a lesser extent (e.g., τ N+1 =γ·τ n ≠N+1, where γ>1). In some examples, during training, for the loss function softmax temperature τ n≠N+1 can be fixed to 1, where γ = 3 (e.g., τ N+1 =3), which is described below with respect to equation (7).

[0100] In some aspects, the probability distribution of equation (5) can be determined using the N+1 softmax 560 output or otherwise. During training, the cross entropy loss for known classes and the cross entropy loss for known classes can be determined and used to generate a combined loss or loss function that drives the training process. In an illustrative example, one or more cross entropy losses can be used to learn (e.g., during training) the N+1 class classification 570 as follows:

[0101]

[0102] Here, Q K and Q U are the known queries and unknown queries of the query set Q 516, respectively. Kmay be a known query from the known query set 517, and Q U It can be an unknown query from the unknown query set 519. Figure 5 is illustrated as being included in query set Q 516.

[0103] In some cases, the two losses given in Equations (6) and (7) can be balanced by a hyperparameter λ, where the total loss function (e.g., again the cross entropy loss) is expressed as:

[0104]

[0105] In some examples, the hyperparameter λ can be set equal to 0.1, but other hyperparameter values ​​can also be used. Using the cross entropy loss of equation (8), it can be generated using the fictitious 550 generated dummy prototype c d and using the function generator f φ The virtual prototype network described in this article (for example, the virtual prototype network 500) is trained simultaneously on the original prototype set C generated at 520.

[0106] During inference, the cross entropy loss (e.g., of equations (6) to (8)) is no longer needed or utilized. In an illustrative example, the N+1 classes can be identified based on or using the N+1 softmax output. For example, the output of N+1 softmax 560 can be used to detect and reject open set queries. Figure 5 As illustrated, the fictitious prototype network 500 may include an outlier detection engine 590 that may be used to detect open set queries for a given FSOS-KWS episode.

[0107] In one illustrative example, the outlier detection engine 590 may be based on the virtual class c d For example, the outlier detection engine 590 can determine the probability of a given x being an open set query based on the following: i Is it an open set (i.e., and therefore an outlier and / or should be rejected):

[0108] p θ (y i =y d |x i ) Equation (9)

[0109] In some examples, the probability p given by equation (9) θ It can be expressed as a given query x i Belongs to the fictitious category c d The probability of (for example, where y i Represents the query x iThe label or classification of y d Indicates that it is assigned to the dummy class c d 552 labels or categories; and the label of the open set query can be set to the N+1th label y d ).

[0110] In one illustrative example, the outlier detection engine 590 may be based on the above probability (e.g., p θ (y i =y d |x i )) is compared or analyzed with a threshold δ to detect or otherwise determine that a given query x i Belongs to the open set.

[0111] like Figure 5 As illustrated, the classification output of the N+1 classifier 570 with open set rejection applied by the outlier detection engine 590 is obtained by arg max n∈{1,…N} p θ (y=n|x i ) for N-category classification.

[0112] In some examples, Figure 5 The fictitious prototype network can be tested by classifying N known classes as follows:

[0113]

[0114] To verify that a given x i Whether it is or whether it belongs to an open set in some other way, the probability p given by equation (9) can be θ (y i =y d |x i ) is compared or analyzed with a threshold value δ (e.g., the test may be performed the same or similar to the above inference).

[0115] In one illustrative example, the systems and techniques described herein can utilize multiple virtual L. For example, Figure 5 The illustrated example prototype network 500 can be based on the learnable matrix W (e.g., of Equation (4)) g is set to a matrix of dimension H×(L·D) to include or generate multiple virtual sets L. Note that in the above single virtual set example, the learnable matrix W is provided g As a matrix of dimension H x D (eg, where L = 1).

[0116] In some cases, the learning method is based, at least in part, on using a learnable matrix W of dimension H×(L·D) g , the example prototype network 500 can be used for a given input x in the following wayi Naturally choose the most likely hypothesis among the L hypotheses: argmax l (-d(x i ,c l )).

[0117] In an illustrative example, Gumbel softmax can be used to replace indistinguishable samples with distinguishable samples (e.g., argmax). For example, using Gumbel softmax, the probability of selecting a virtual set l from a set of multiple virtual sets L can be expressed as:

[0118]

[0119] Here,∈1,…,∈ L is an independent and identically distributed (IID) sample drawn from a standard Gumbel distribution with μ = 0 and β = 1. L is a temporary dummy label among the L dummy labels. In some examples, training can be performed to produce:

[0120] c d =Σ l p(y L =l|x)·c l Equation (12)

[0121] In one illustrative example, when inferring, a single virtual set y from among multiple virtual sets L may be selected. L For example, a single virtual set y can be selected by the following formula: L :

[0122] arg max l p(y L =l|x) Equation (13)

[0123] For example, the selected fictitious y L It can be used to classify and reject open set queries as dummies in the same or similar manner as described above with respect to the example in which dummies generator 550 generates a single dummies.

[0124] In one illustrative example, training and / or inference may be performed with the τ term in the Gumbel softmax (e.g., in Equation (11) above) annealed from 2 to 0.5. In some examples, when multiple dummy L are utilized, the number of dummy L may be L=3, but it should be noted that a greater or lesser number of dummy L may also be utilized without departing from the scope of the present disclosure.

[0125] In some examples, one or more voice command data sets can be used to train, verify and / or test systems and technologies described herein. In an illustrative example, the voice command data set may include a total of 35 keywords and may be used to support 12 class classifications. For example, the 12 classes may include 10 keywords (e.g., "yes," "no," "up," "down," "left," "right," "open," "close," "stop," and "advance") and two additional classes. The two additional classes may include an "unknown word" class that includes 25 remaining keywords (e.g., a total of 35 keywords in the voice command data set minus the above-mentioned 10 keywords), and a "silent" class. In some cases, the "silent" class may include only background noise.

[0126] In some examples, the voice command dataset can be split by class label. For example, the split can be performed to establish 15 keywords for the training set, 10 keywords for the validation set, and 10 keywords for the test set:

[0127] · Training Keywords : "happy", "house", "bird", "bed", "back",

[0128] "Sheila", "Marvin", "wow", "tree", "follow", "dog", "vision", "forward", "learn", and "cat"

[0129] · Verify keywords (e.g., numbers): "zero", "one", "two", "three",

[0130] Four, Five, Six, Seven, Eight, and Nine

[0131] · Test keywords (10 keyword categories used in the conventional 12-category KWS): "Yes",

[0132] No, Up, Down, Left, Right, Open, Close, Stop, and Go.

[0133] In some cases, using a fixed keyword split can prevent performance differences that may be introduced by changes in the split during trials. In addition to the split, a specific class "silent" can be added, which can be included in the open set only as a background noise class. For example, in the 5 classes 5 example scenario, five known classes without "silent" can be randomly selected, and the same number of open set classes can also be selected from the remaining classes including "silent".

[0134] In some examples, these settings can be referred to as a split speech command dataset.

[0135] In some examples, the base voice command dataset (e.g., before applying the above split) can be the Google Voice Command (GSC) dataset ver2, which contains 105,829 utterances from 2,618 speakers. As described above, using the official split (e.g., using a hash function on the name of each utterance file), the dataset can first be split into a training set, a validation set, and a test set with 84,843, 9,981, and 11,005 utterances, respectively.

[0136] Subsequently, samples can be selected based on the above split voice command dataset method to obtain 22,916, 3,643, and 4,074 samples for training, validation, and testing, respectively. "Silent" samples can be added to each split by the average number of utterances of each class for each split, resulting in a final split voice command dataset, including 24,444, 4,007, and 4,482 utterances for training, validation, and testing, respectively. In some examples, training can be performed using the minimal data augmentation typically used in the GSC task, adding the official background noise provided by GSC with a probability of 0.8.

[0137] In some aspects, the encoder f φ 520 may include or implement a machine learning backbone (e.g., as described above). For example, the encoder f φ 520 may include or implement one or more of Conv4-64, ResNet-12, and / or BCResNet-8. φ In the example of using Conv4-64 architecture, the output dimension can be 768. φ In the example of using the ResNet-12 architecture, the output dimension can be 512. φ In the example of 520 using the BCResNet-8 architecture, the output dimension may be 256. In some cases, a Conv4-64 architecture may be associated with 768 dimensions (e.g., resulting in 768 dimensions that are greater than 64, the number of input channels of the architecture) based at least in part on the absence of global average pooling at the output of the architecture.

[0138] In some aspects, each utterance in GSC is 1 second long and the sampling rate is 16kHz. During training (e.g., using the above-mentioned final split voice command dataset), input features of 40-dimensional log Mel spectrograms with frameshifts and window lengths of 10 milliseconds (ms) and 30 ms, respectively, can be used. In some examples, the Adam optimizer with an initial learning rate of 0.001 can be used to train the fictitious prototype network described herein (e.g., Figure 5The illustrated fictitious prototype network 500) is trained for 100 epochs. The learning rate may be gradually decayed by multiplying by 0.5 every 20 epochs. Each epoch may include 100 episodes, wherein each episode includes 5 known classes (e.g., 5 classes) and 5 open set classes. 5 supporting examples (e.g., 5 examples) may be used or provided for each episode. During training, five queries may be provided or otherwise utilized for each class. During testing, 15 queries may be provided or otherwise utilized for each class. In some examples, early stopping may be used by validating accuracy with a small sample, and the trained model may be evaluated with 1,000 episodes.

[0139] Figure 6 is a flow chart illustrating an example of a process 600 for processing one or more data samples (e.g., one or more data samples of audio data). At block 602, the process 600 includes determining one or more prototype representations based on a plurality of supporting samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes. For example, the one or more prototype representations may be a class using Figure 5 The example virtual prototype network 500 is determined. In some examples, the one or more prototype representations can be generated by a machine learning and / or neural network encoder (such as Figure 5 The one or more prototype representations may be determined based on receiving as input the plurality of support samples, where each support sample is associated with a known class (e.g., labeled with a known class). In some examples, an encoder (e.g., encoder 520) may generate multiple embeddings for the plurality of support samples associated with each keyword class. A prototype representation may then be determined for each keyword class as the average of the embedded support samples for that class. For example, the average of the embedded support samples for each class may be calculated using Figure 5 The illustrated averaging engine 530 determines.

[0140] In some examples, the one or more classes may be keyword classes (e.g., of a keyword detection (KWS) system). In some cases, each prototype representation may be determined using a plurality of supporting samples associated with one of the keyword classes. For example, each prototype representation may be determined for one of the plurality of keyword classes, where the prototype representation of the keyword class is determined based on supporting samples associated with the same keyword class.

[0141] At block 604, process 600 includes determining a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space. For example, the fictitious prototype representation may be a representation of Figure 5The virtual prototype representation is determined by the illustrated virtual generator 550. In some examples, the virtual prototype representation can be determined using the prototype representation determined for each known class (e.g., as described above with respect to block 602) as input. In some examples, the one or more prototype representations and the virtual prototype representation can be determined in the same learned embedding space of the neural network encoder (e.g., the virtual prototype representation can be determined in the same embedding space associated with the encoder 520).

[0142] In some examples, a dummy generator network (e.g., dummy generator 550) may be used to determine the dummy prototype representation. For example, the dummy generator network may be a machine learning network including one or more linear layers and a max pooling layer. The one or more linear layers may receive the one or more prototype representations as input (e.g., from encoder 520 and averaging engine 530, both in Figure 5 ). The maximum pooling layer may receive the output of the one or more linear layers as input.

[0143] In some examples, determining the dummy prototype representation also includes generating multiple dummy prototypes using a dummy generator network (e.g., dummy generator 550). The dummy prototype representation can be a single dummy prototype selected from multiple dummy prototypes. For example, the dummy prototype representation can be a single dummy prototype selected from multiple dummy prototypes based on Gumbel softmax.

[0144] At block 606, process 600 includes determining one or more distance metrics for each of the one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation. For example, the one or more distance metrics may be determined using distance metric 540, such as Figure 5 In some examples, the distance metric may be the Euclidean distance between each query sample and each of the prototype representation and the fictitious prototype representation.

[0145] At block 608, process 600 includes classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the fictitious prototype representation. For example, the query sample may be a class that is identified using a distance metric such as Figure 5 The softmax classification is performed by the illustrated N+1softmax 560. In some examples, the softmax classification can be based on or otherwise utilize the distance metric described above with respect to block 606. For example, the softmax classification can classify a given query sample into one of the classes associated with the prototype representation or into an open set class associated with the fictitious prototype representation.

[0146] In some examples, classifying each query sample based on the one or more distance metrics includes determining a probability distribution over the one or more classes and the open set class, wherein the probability distribution is determined at least in part based on a Euclidean distance metric (such as the Euclidean distance metric described above). For example, as the Euclidean distance (or distance metric) determined between the query sample and a class (or open set class) in the class decreases, the probability that a given query sample is classified into a given one of the classes or the open set class in the class can be relatively high. In some examples, based on the probability distribution, each query sample can be classified into one of the one or more classes associated with the prototype representation or into the open set class associated with the fictitious prototype representation. In some examples, open set rejection (OSR) can be performed based on a set of classified query samples classified into the open set class associated with the fictitious prototype representation.

[0147] In some examples, the processes described herein (e.g., process 600 and / or any other process described herein) can be performed by a computing device, apparatus, or system. In one example, process 600 can be performed by a computer having Figure 7 The computing device, apparatus or system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a laptop, a smart TV, a camera and / or any other computing device with resource capabilities to perform the processes described herein (including process 600 and / or any other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0148] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0149] Process 600 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0150] In addition, process 600 and / or any other process described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware, or a combination thereof. As described above, the code may, for example, be stored on a computer-readable or machine-readable storage medium in the form of a computer program that includes multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0151] Figure 7An example computing device architecture 700 of an example computing device that can implement the various techniques described herein is illustrated. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other device. The components of the computing device architecture 700 are shown as being in electrical communication with each other using a connector 705 such as a bus. The example computing device architecture 700 includes a processing unit (CPU or processor) 710 and a computing device connector 705 that couples various computing device components including a computing device memory 715 (such as a read-only memory (ROM) 720 and a random access memory (RAM) 725) to the processor 710.

[0152] The computing device architecture 700 may include a cache of high-speed memory directly connected to the processor 710, very close to the processor, or integrated as part of the processor. The computing device architecture 700 may copy data from the memory 715 and / or the storage device 730 to the cache 712 for fast access by the processor 710. In this way, the cache can provide a performance improvement that avoids the processor 710 from delaying when waiting for data. These engines and other engines can control or be configured to control the processor 710 to perform various actions. Other computing device memories 715 are also available. The memory 715 may include a variety of different types of memory with different performance characteristics. The processor 710 may include any general-purpose processor and hardware or software services (such as service 1 732, service 2 734, and service 3 736 stored in the storage device 730) configured to control the processor 710, as well as a dedicated processor in which software instructions are incorporated into the processor design. The processor 710 may be a self-contained system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0153] To enable user interaction with the computing device architecture 700, the input device 745 may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 735 may also be one or more of a plurality of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multi-mode computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 700. The communication interface 740 may generally control and manage user input and computing device output. There is no restriction on operating on any particular hardware arrangement, and thus the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0154] The storage device 730 is a non-volatile memory and may be a hard disk or other type of computer readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 725, a read-only memory (ROM) 720, and hybrids thereof. The storage device 730 may include services 732, 734, 736 for controlling the processor 710. Other hardware or software modules or engines are contemplated. The storage device 730 may be connected to the computing device connector 705. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer readable medium connected to the necessary hardware components (such as the processor 710, the connector 705, the output device 735, etc.) to perform the function.

[0155] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0156] The term "device" is not limited to one or a specific number of physical objects (such as a smart phone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. In addition, the term "system" is not limited to multiple components or specific aspects. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0157] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology can be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those components shown in the accompanying drawings and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details to avoid confusing various aspects.

[0158] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a process, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0159] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed over a network. Computer executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0160] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as flash memory), memory or memory devices, disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, compact discs (CDs) or digital versatile discs (DVDs), any suitable combination thereof, etc. Computer-readable media may store thereon code and / or machine-executable instructions that may represent any combination of a process, function, subroutine, program, routine, subroutine, module, engine, software package, class, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0161] In some aspects, computer readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0162] The equipment implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By way of further example, such functionality may also be implemented on circuit boards among different chips or different processes executed in a single device.

[0163] Instructions, media for conveying such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0164] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be appreciated that, in alternative aspects, each method can be performed in an order different from the described order.

[0165] One of ordinary skill in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0166] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0167] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0168] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0169] Various exemplary logic blocks, modules, engines, circuits and algorithmic steps described in conjunction with aspects disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Technicians can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as making it depart from the scope of the present application.

[0170] The technology described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices, which have multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components can be implemented together in an integrated logic device or separately implemented as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable data storage medium including program codes, which include instructions that execute one or more of the above methods when executed. Computer-readable data storage media can form a part of a computer program product, which can include packaging materials. Computer-readable media may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0171] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0172] Illustrative aspects of the present disclosure include:

[0173] Aspect 1: A method for processing one or more data samples, the method comprising: determining one or more prototype representations based on multiple support samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determining a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representations; and classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the fictitious prototype representation.

[0174] Aspect 2: A method according to Aspect 1, wherein classifying each query sample based on the one or more distance metrics includes: determining a probability distribution over the one or more classes and the open set class, wherein the probability distribution is determined at least in part based on the Euclidean distance metric; classifying each query sample into one of the one or more classes or into the open set class based on the probability distribution; and performing open set rejection (OSR) based on a set of classified query samples classified into the open set class associated with the fictitious prototype representation.

[0175] Aspect 3: The method according to any one of aspects 1 to 2, wherein the one or more classes are keyword classes, and each prototype representation is determined using a plurality of supporting samples associated with the keyword class.

[0176] Aspect 4: According to the method of Aspect 3, the method also includes: using a neural network encoder to generate multiple embeddings for the multiple support samples associated with the keyword class, the multiple embeddings include an embedding for each support sample in the multiple support samples; and determining each prototype representation based on an average of the multiple embeddings generated for the multiple support samples.

[0177] Aspect 5: The method according to Aspect 4, wherein the one or more prototype representations and the virtual prototype representation are determined in the same learned embedding space of the neural network encoder.

[0178] Aspect 6: According to the method described in any one of Aspects 1 to 5, the method further includes using a virtual generator network to determine the virtual prototype representation, wherein: the virtual generator network includes one or more linear layers and a maximum pooling layer; the one or more linear layers receive the one or more prototype representations as input; and the maximum pooling layer receives the output of the one or more linear layers as input.

[0179] Aspect 7: The method according to Aspect 6, wherein determining the virtual prototype representation further comprises: generating a plurality of virtual prototypes using the virtual generator network; and determining the virtual prototype representation by selecting one of the plurality of virtual prototypes based on Gumbel softmax.

[0180] Aspect 8: The method according to any one of aspects 1 to 3, wherein the keyword class comprises a closed set of known classes.

[0181] Aspect 9: A method according to any one of Aspects 1 to 8, wherein: the one or more query samples include speech audio data; and the multiple support samples include speech audio data labeled with a corresponding one of the one or more classes for each support sample in the multiple support samples.

[0182] Aspect 10: A method according to any one of Aspects 1 to 9, wherein the plurality of support samples are obtained for a single few-shot learning (FSL) episode, and the fictitious prototype representation is a fictitious prototype representation known to the episode.

[0183] Aspect 11: A device for processing one or more data samples, the device comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: determine one or more prototype representations based on a plurality of support samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determine a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determine one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; and classify each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the fictitious prototype representation.

[0184] Aspect 12: An apparatus according to Aspect 11, wherein in order to classify each query sample based on the one or more distance metrics, the at least one processor is configured to: determine a probability distribution over the one or more classes and the open set class, wherein the probability distribution is determined at least in part based on the Euclidean distance metric; classify each query sample into one of the one or more classes or into the open set class based on the probability distribution; and perform open set rejection (OSR) based on a set of classified query samples classified into the open set class associated with the fictitious prototype representation.

[0185] Aspect 13: The apparatus according to any one of aspects 11 to 12, wherein the one or more classes are keyword classes, and the at least one processor is configured to determine each prototype representation using a plurality of supporting samples associated with the keyword class.

[0186] Aspect 14: An apparatus according to Aspect 13, wherein the at least one processor is further configured to: generate multiple embeddings for the multiple support samples associated with the keyword class using a neural network encoder, the multiple embeddings including an embedding for each support sample in the multiple support samples; and determine each prototype representation based on an average of the multiple embeddings generated for the multiple support samples.

[0187] Aspect 15: The apparatus according to Aspect 14, wherein the at least one processor is configured to determine the one or more prototype representations and the virtual prototype representation in the same learned embedding space of the neural network encoder.

[0188] Aspect 16: An apparatus according to any one of Aspects 11 to 15, wherein the at least one processor is further configured to determine the virtual prototype representation using a virtual generator network, wherein: the virtual generator network comprises one or more linear layers and a maximum pooling layer; the one or more linear layers receive the one or more prototype representations as input; and the maximum pooling layer receives the output of the one or more linear layers as input.

[0189] Aspect 17: An apparatus according to Aspect 16, wherein, in order to determine the virtual prototype representation, the at least one processor is configured to: generate a plurality of virtual prototypes using the virtual generator network; and determine the virtual prototype representation by selecting one of the plurality of virtual prototypes based on Gumbel softmax.

[0190] Aspect 18: The apparatus according to any one of Aspects 11 to 13, wherein the keyword class comprises a closed set of known classes.

[0191] Aspect 19: An apparatus according to any one of Aspects 11 to 18, wherein: the one or more query samples include speech audio data; and the multiple support samples include speech audio data labeled with a corresponding one of the one or more classes for each support sample in the multiple support samples.

[0192] Aspect 20: An apparatus according to any one of Aspects 11 to 19, wherein the plurality of support samples are obtained for a single few-shot learning (FSL) episode, and the fictitious prototype representation is an episode-known fictitious prototype representation.

[0193] Aspect 21: A non-transitory computer-readable storage medium, comprising instructions stored thereon, which, when executed by at least one processor, causes the at least one processor to: determine one or more prototype representations based on a plurality of support samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determine a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determine one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; and classify each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the fictitious prototype representation.

[0194] Aspect 22: A non-transitory computer-readable storage medium according to Aspect 21, wherein in order to classify each query sample based on the one or more distance metrics, the instructions cause the at least one processor to: determine a probability distribution over the one or more classes and the open set class, wherein the probability distribution is determined at least in part based on the Euclidean distance metric; classify each query sample into one of the one or more classes or into the open set class based on the probability distribution; and perform open set rejection (OSR) based on a set of classified query samples classified into the open set class associated with the fictitious prototype representation.

[0195] Aspect 23: The non-transitory computer-readable storage medium according to any one of aspects 21 to 22, wherein the one or more classes are keyword classes, and the at least one processor is configured to determine each prototype representation using a plurality of supporting samples associated with the keyword class.

[0196] Aspect 24: A non-transitory computer-readable storage medium according to Aspect 23, wherein the instructions further cause the at least one processor to: generate multiple embeddings for the multiple support samples associated with the keyword class using a neural network encoder, the multiple embeddings including an embedding for each support sample in the multiple support samples; and determine each prototype representation based on an average of the multiple embeddings generated for the multiple support samples.

[0197] Aspect 25: A non-transitory computer-readable storage medium according to Aspect 24, wherein the instructions cause the at least one processor to determine the one or more prototype representations and the virtual prototype representation in the same learned embedding space of the neural network encoder.

[0198] Aspect 26: A non-transitory computer-readable storage medium according to any one of Aspects 21 to 25, wherein the instructions further cause the at least one processor to determine the virtual prototype representation using a virtual generator network, wherein: the virtual generator network includes one or more linear layers and a maximum pooling layer; the one or more linear layers receive the one or more prototype representations as input; and the maximum pooling layer receives the output of the one or more linear layers as input.

[0199] Aspect 27: A non-transitory computer-readable storage medium according to Aspect 26, wherein, in order to determine the virtual prototype representation, the instructions cause the at least one processor to: generate a plurality of virtual prototypes using the virtual generator network; and determine the virtual prototype representation by selecting one of the plurality of virtual prototypes based on Gumbel softmax.

[0200] Aspect 28: The non-transitory computer-readable storage medium according to any one of Aspects 23 to 27, wherein the keyword class comprises a closed set of known classes.

[0201] Aspect 29: A non-transitory computer-readable storage medium according to any one of Aspects 21 to 28, wherein: the one or more query samples include speech audio data; and the multiple support samples include speech audio data labeled with a corresponding one of the one or more classes for each support sample in the multiple support samples.

[0202] Aspect 30: A non-transitory computer-readable storage medium according to any one of Aspects 21 to 29, wherein the plurality of support samples are obtained for a single few-shot learning (FSL) episode, and the fictitious prototype representation is a fictitious prototype representation known to the episode.

[0203] Aspect 31: A computer-readable storage medium storing instructions, which, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of aspects 1 to 30.

[0204] Aspect 32: An apparatus for processing one or more data samples, the apparatus comprising one or more means for performing the operations according to any one of aspects 1 to 30.

Claims

1. A method for processing one or more data samples, the method comprising: determining one or more prototype representations based on a plurality of supporting exemplars associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determining a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample of the one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; as well as Each query sample is classified based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the fictitious prototype representation.

2. The method according to claim 1, wherein classifying each query sample based on the one or more distance metrics comprises: determining a probability distribution over the one or more classes and the open set class, wherein the probability distribution is determined based at least in part on a Euclidean distance metric; Classifying each query sample into one of the one or more classes or into the open set class based on the probability distribution; as well as Open set rejection (OSR) is performed based on a set of classified query samples classified into the open set classes associated with the fictitious prototype representation. 3 . The method of claim 1 , wherein the one or more classes are keyword classes, and each prototype representation is determined using a plurality of supporting samples associated with the keyword class.

4. The method according to claim 3, further comprising: generating, using a neural network encoder, a plurality of embeddings for the plurality of support samples associated with the keyword class, the plurality of embeddings comprising an embedding for each support sample in the plurality of support samples; as well as Each prototype representation is determined based on an average of the plurality of embeddings generated for the plurality of support samples.

5. The method of claim 4, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learned embedding space of the neural network encoder.

6. The method of claim 1 , further comprising determining the fictitious prototype representation using a fictitious generator network, wherein: The virtual generator network includes one or more linear layers and a maximum pooling layer; The one or more linear layers receive as input the one or more prototype representations; and The maximum pooling layer receives as input the output of the one or more linear layers.

7. The method of claim 6, wherein determining the fictitious prototype representation further comprises: Generate a plurality of dummy prototypes using the dummy generator network; as well as The fictitious prototype representation is determined by selecting one of the plurality of fictitious prototypes based on Gumbel softmax. The method of claim 3 , wherein the keyword classes comprise a closed set of known classes.

9. The method according to claim 1, wherein: The one or more query samples include speech audio data; and The plurality of support samples include speech audio data labeled with a corresponding one of the one or more classes for each support sample in the plurality of support samples.

10. The method of claim 1, wherein the plurality of support samples are obtained for a single few-shot learning (FSL) episode, and the fictitious prototype representation is an episode-known fictitious prototype representation.

11. An apparatus for processing one or more data samples, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: Determining one or more prototype representations based on a plurality of support samples associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determining a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample of the one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; as well as Each query sample is classified based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the fictitious prototype representation.

12. The apparatus according to claim 11, wherein in order to classify each query sample based on the one or more distance metrics, the at least one processor is configured to: determining a probability distribution over the one or more classes and the open set class, wherein the probability distribution is determined based at least in part on a Euclidean distance metric; Classifying each query sample into one of the one or more classes or into the open set class based on the probability distribution; as well as Open set rejection (OSR) is performed based on a set of classified query samples classified into the open set classes associated with the fictitious prototype representation.

13. The apparatus of claim 11, wherein the one or more classes are keyword classes, and the at least one processor is configured to determine each prototype representation using a plurality of supporting samples associated with the keyword class.

14. The apparatus of claim 13, wherein the at least one processor is further configured to: generating, using a neural network encoder, a plurality of embeddings for the plurality of support samples associated with the keyword class, the plurality of embeddings comprising an embedding for each support sample in the plurality of support samples; and Each prototype representation is determined based on an average of the plurality of embeddings generated for the plurality of support samples.

15. The apparatus of claim 14, wherein the at least one processor is configured to determine the one or more prototype representations and the fictitious prototype representation in a same learned embedding space of the neural network encoder.

16. The apparatus of claim 11, wherein the at least one processor is further configured to determine the fictitious prototype representation using a fictitious generator network, wherein: The virtual generator network includes one or more linear layers and a maximum pooling layer; The one or more linear layers receive as input the one or more prototype representations; and The maximum pooling layer receives as input the output of the one or more linear layers.

17. The apparatus of claim 16, wherein to determine the fictitious prototype representation, the at least one processor is configured to: generating a plurality of dummy prototypes using the dummy generator network; and The fictitious prototype representation is determined by selecting one of the plurality of fictitious prototypes based on Gumbel softmax.

18. The apparatus of claim 13, wherein the keyword classes comprise a closed set of known classes.

19. The device according to claim 11, wherein: The one or more query samples include speech audio data; and The plurality of support samples include speech audio data labeled with a corresponding one of the one or more classes for each support sample in the plurality of support samples.

20. The apparatus of claim 11, wherein the plurality of support samples are obtained for a single few-shot learning (FSL) episode, and the fictitious prototype representation is an episode-known fictitious prototype representation.

21. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, cause the at least one processor to: determining one or more prototype representations based on a plurality of supporting exemplars associated with one or more classes, wherein each prototype representation is associated with one of the one or more classes; determining a fictitious prototype representation, wherein the one or more prototype representations and the fictitious prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample of the one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the fictitious prototype representation; as well as Each query sample is classified based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the fictitious prototype representation.

22. The non-transitory computer-readable storage medium of claim 21, wherein to classify each query sample based on the one or more distance metrics, the instructions cause the at least one processor to: determining a probability distribution over the one or more classes and the open set class, wherein the probability distribution is determined based at least in part on a Euclidean distance metric; Classifying each query sample into one of the one or more classes or into the open set class based on the probability distribution; as well as Open set rejection (OSR) is performed based on a set of classified query samples classified into the open set classes associated with the fictitious prototype representation.

23. The non-transitory computer-readable storage medium of claim 21, wherein the one or more classes are keyword classes, and the at least one processor is configured to determine each prototype representation using a plurality of supporting samples associated with the keyword class.

24. The non-transitory computer readable storage medium of claim 23, wherein the instructions further cause the at least one processor to: generating, using a neural network encoder, a plurality of embeddings for the plurality of support samples associated with the keyword class, the plurality of embeddings comprising an embedding for each support sample in the plurality of support samples; and Each prototype representation is determined based on an average of the plurality of embeddings generated for the plurality of support samples.

25. The non-transitory computer-readable storage medium of claim 24, wherein the instructions cause the at least one processor to determine the one or more prototype representations and the fictitious prototype representation in a same learned embedding space of the neural network encoder.

26. The non-transitory computer-readable storage medium of claim 21, wherein the instructions further cause the at least one processor to determine the virtual prototype representation using a virtual generator network, wherein: The virtual generator network includes one or more linear layers and a maximum pooling layer; The one or more linear layers receive as input the one or more prototype representations; and The maximum pooling layer receives as input the output of the one or more linear layers.

27. The non-transitory computer-readable storage medium of claim 26, wherein to determine the virtual prototype representation, the instructions cause the at least one processor to: generating a plurality of dummy prototypes using the dummy generator network; and The fictitious prototype representation is determined by selecting one of the plurality of fictitious prototypes based on Gumbel softmax.

28. The non-transitory computer-readable storage medium of claim 23, wherein the keyword class comprises a closed set of known classes.

29. The non-transitory computer readable storage medium of claim 21, wherein: The one or more query samples include speech audio data; and The plurality of support samples include speech audio data labeled with a corresponding one of the one or more classes for each support sample in the plurality of support samples.

30. The non-transitory computer-readable storage medium of claim 21, wherein the plurality of support samples are obtained for a single few-shot learning (FSL) episode, and the fictitious prototype representation is an episode-known fictitious prototype representation.