Task agnostic open set prototype for small sample open set identification

Abstract

Systems and techniques are provided for processing one or more data samples. For example, a neural network classifier can be trained to perform small sample open set recognition (FSOSR) based on a task agnostic open set prototype. The process can include determining one or more prototype representations for each class included in the plurality of support samples. A task-agnostic open set prototype representation can be determined in the same learning metric space as the one or more prototype representations. Based on the one or more prototype representations and the task-agnostic open set prototype representation, one or more distance metrics can be determined for each of the one or more query samples. Based on the one or more distance metrics, each query sample can be classified into one of the classes associated with the one or more prototype representations or into an open-set class associated with the task-agnostic open-set prototype representation.

Description

Technical Field

[0001] The present disclosure generally relates to machine learning classification. For example, aspects of the present disclosure relate to systems and techniques for performing small sample open set recognition (FSOSR) using one or more machine learning systems (e.g., including one or more neural network-based models). Background Art

[0002] Artificial neural networks attempt to use computer technology to replicate the logical reasoning performed by biological neural networks that make up animal brains. Deep neural networks, such as convolutional neural networks, are widely used in many applications, such as object detection, object classification, object tracking, big data analysis, etc. For example, a convolutional neural network is able to extract high-level features (such as facial shape) from an input image, and use these high-level features to output, for example, the probability that the input image includes a specific object. Summary of the invention

[0003] In some examples, systems and techniques for performing small sample open set recognition (FSOSR) using task-agnostic open set prototypes are described. According to at least one example, a method (e.g., a processor-implemented method) for processing one or more data samples is provided. The method includes: determining one or more prototype representations based on a plurality of support samples associated with one or more classes of the data sample, wherein each prototype representation is associated with one of the one or more classes; determining a task-agnostic open set prototype representation, wherein the one or more prototype representations and the task-agnostic open set prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open set prototype representation; and classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the task-agnostic open set prototype representation.

[0004] In another example, a device for processing one or more data samples is provided, the device comprising: a memory (e.g., configured to store data); and one or more processors (e.g., implemented in a circuit), the one or more processors coupled to the memory. The one or more processors are configured and can: determine one or more prototype representations based on a plurality of support samples associated with one or more classes of the data sample, wherein each prototype representation is associated with one of the one or more classes; determine a task-agnostic open set prototype representation, wherein the one or more prototype representations and the task-agnostic open set prototype representation are determined in the same learning metric space; determine one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open set prototype representation; and classify each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the task-agnostic open set prototype representation.

[0005] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which instructions, when executed by one or more processors, cause the one or more processors to: determine one or more prototype representations based on a plurality of support samples associated with one or more classes of data samples, wherein each prototype representation is associated with one of the one or more classes; determine a task-agnostic open-set prototype representation, wherein the one or more prototype representations and the task-agnostic open-set prototype representation are determined in the same learning metric space; determine one or more distance metrics for each of one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open-set prototype representation; and classify each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the task-agnostic open-set prototype representation.

[0006] In another example, a device for processing one or more data samples is provided. The device includes: a component for determining one or more prototype representations based on a plurality of support samples associated with one or more classes of the data sample, wherein each prototype representation is associated with one of the one or more classes; a component for determining a task-agnostic open-set prototype representation, wherein the one or more prototype representations and the task-agnostic open-set prototype representation are determined in the same learning metric space; a component for determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open-set prototype representation; and a component for classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the task-agnostic open-set prototype representation.

[0007] In some aspects, one or more of the devices described herein are, are part of, and / or include a mobile device (e.g., a mobile phone or other mobile device), a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a connected device, a head mounted device (HMD) device, a wireless communication device, a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof. An electronic device (e.g., a mobile phone, etc.) is configured with hardware components that enable the electronic device to perform or execute a specific context or application. In some aspects, the device includes a camera or multiple cameras for capturing one or more images or video frames of a scene including various items (such as a person, an animal, and / or any object). In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the above-mentioned device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors). In some cases, a machine learning model (e.g., one or more neural networks or other machine learning models) may be used to process the sensor data and generate classifications related to the sensor data.

[0008] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0009] The foregoing, as well as other objectives and advantages associated with the various aspects disclosed herein, will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are presented to aid in describing various aspects of the present disclosure and are provided solely for illustration and not limitation of the various aspects:

[0011] Figure 1 illustrates example implementations of a system on a chip (SoC) according to some examples;

[0012] Figure 2A and Figure 2B An example of a fully connected neural network according to some examples is illustrated;

[0013] Figure 3 is a diagram illustrating an example of a few-shot learning (FSL) scenario according to some examples;

[0014] Figure 4A is a diagram illustrating another example of a few-shot learning (FSL) scenario according to some examples;

[0015] Figure 4B is a diagram illustrating a first example episode in a few-shot open set recognition (FSOSR) episode according to some examples;

[0016] Figure 4C is a diagram illustrating a second example episode in a few-shot open set recognition (FSOSR) episode according to some examples;

[0017] Figure 5A is a diagram illustrating an example task-agnostic open set prototype associated with a first task having a first selection of open sets and closed sets according to some examples;

[0018] Figure 5B is a diagram illustrating an example task-agnostic open set prototype associated with a second task having a second selection of open sets and closed sets according to some examples;

[0019] Figure 6 is a diagram illustrating an example architecture of a task-agnostic open-set prototypical network according to some examples;

[0020] Figure 7 is a flow chart illustrating an example of a process for processing one or more data samples to perform FSOSR based on a task-agnostic open set prototype according to some examples; and

[0021] Figure 8 is a block diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0022] Some aspects and examples of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples can be applied independently, and some of them can be applied in combination. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects and examples of the application. However, it is apparent that various aspects and examples can be practiced without these specific details. Each drawing and description are not intended to be restrictive.

[0023] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of exemplary aspects will provide an enabling description of examples and aspects for implementing the present disclosure to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0024] Deep learning can be used to perform tasks such as recognition and / or classification for various types of input data. For example, deep learning can be used to perform tasks such as image recognition, image classification, object classification, speech recognition, keyword spotting (KWS), etc. In some cases, different types of input data may also be referred to as "input domains" or "domains" (e.g., image data may be associated with an image domain, audio data may be associated with an audio domain, etc.).

[0025] Deep learning can be performed using one or more neural networks. For example, a neural network can be trained to perform a recognition or classification task based on one or more training data sets that are provided as input to the neural network during the training process. The training data can be labeled, unlabeled, or a combination of both. The training (e.g., learning) process can be supervised, unsupervised, or a combination of both. Although there are different neural network architectures, deep learning methods, and training processes, the need for large amounts of training data can often be a common requirement.

[0026] For example, tens of thousands or hundreds of thousands of individual training data samples may be used to train a neural network to perform a recognition or classification task to a given level of accuracy. In some cases, compiling a representative training data set for a given machine learning task may be a challenging and time-consuming task. Additionally, training a neural network using a relatively large training data set may be a computationally intensive task that is also time-consuming. In some examples, the performance of various neural networks and / or machine learning networks may be reduced when the available training data set is small or otherwise limited.

[0027] Many neural networks rely on learning from large amounts of data for training, and may not generalize quickly beyond the few examples seen in the training data. For example, an image classification neural network may be trained to classify the type of plant present in an image based on a large training dataset with thousands of training data pairs, each training data pair comprising an image of the plant and a ground-truth classification label. However, such an image classification network may be unlikely to subsequently be used to classify the type of aircraft present in an image given that there are only a few labeled examples of aircraft classification (e.g., a neural network trained to classify a set of predefined data samples may achieve accurate performance when evaluated on a closed set of classes seen in training, but is typically limited to detecting only those classes seen in training).

[0028] Few-shot learning (FSL) is a machine learning method that can be used to provide a neural network (e.g., a neural network classifier, etc.) with improved generalization to new tasks that involve only a few samples with supervised information. For example, a neural network classifier based on FSL can attempt to correctly classify one or more classes that have not been seen previously (e.g., not seen during training) but are known based on a set of labeled support samples (e.g., provided during inference). In some cases, a neural network classifier based on FSL can classify a given query (e.g., inference input) into one or more classes in a closed set of predefined classes seen in training, or into previously unseen classes identified during an FSL episode (e.g., based on support samples).

[0029] For example, a FSL-based neural network classifier can be used to detect data samples (e.g., images, audio data, video data, etc.) or classify these data samples into one or more classes that were not initially seen during training. In an example FSL-based classification method, a neural network classifier is trained on a training set that includes labeled samples representing various predefined classes or categories of data samples. During training, the neural network classifier absorbs knowledge from the labeled samples of the training set. For example, the neural network classifier can learn an embedding or embedding space that relates the training samples to their corresponding labels. Subsequently, the trained neural network classifier can be adapted or generalized (e.g., under inference) to the evaluation task of new classes (e.g., unseen classes) that were not seen in the training set.

[0030] For example, assuming that there are only a small number of support examples for each new class, an FSL-based neural network classifier can use an attention mechanism on the learned embedding of a set of labeled support examples (e.g., a support set) to predict the class of an unlabeled point (e.g., a query set). In the process of example-by-example querying, a trained FSL-based neural network classifier can receive a support set including M support examples for each of N unseen classes, and a query set including one or more query samples. The neural network classifier can determine a prototype representation for each unseen class N (e.g., using the M support examples associated with each unseen class N). Each unlabeled query sample can be classified into one of the previously unseen classes N based on the calculated distance between the query sample and each prototype representation. In some cases, this inference process can be referred to as an N-class M-sample scenario, where the goal of the FSL-based neural network classifier is to correctly classify the query set into N classes that were not seen during training but are known using the M support samples.

[0031] In real-world scenarios and / or datasets where FSL-based neural network classifiers can be utilized, input queries (e.g., inference inputs) may naturally include data samples of unexpected novel classes for which no supporting examples are given. Unexpected novel classes may be classes for which supporting examples are not included in the initial training set or support set. These unexpected samples and / or classes may be referred to as "open sets" (e.g., in contrast to closed sets of classes from the initial training set and support set).

[0032] In some cases, a FSL-based neural network classifier may exhibit overconfidence and incorrectly classify unseen open-set examples into one of the FSL classes. In some approaches to FSL, a FSL-based neural network classifier may classify each input query into one of the known (e.g., closed) classes even if the input query belongs to an unknown (e.g., open-set) class. In this approach, each input query is treated as a closed-set example, and the classification is not extended to the possibility that the input query is an open-set example. In some examples, the accuracy and / or performance of a FSL-based neural network classifier may be improved by rejecting open-set examples instead of classifying them into one of the predetermined closed-set classes (e.g., closed-set). For example, open set recognition (OSR) may be used to detect unseen open-set classes, and query samples belonging to the detected open-set classes may be subsequently processed and / or rejected.

[0033] OSR can be used to distinguish unknown novel categories (e.g., classes) from fixed closed sets. For example, an input query that is not classified into a closed set class (e.g., because the input query is not close enough to any of the closed set classes) can be classified into an open set class and / or can be classified as an open set example. OSR can be performed together with FSL, which can also be referred to as Few-Shot Open Set Recognition (FSOSR). FSOSR can be used to extend FSL-based classification methods by (e.g., in addition to classifying closed set examples into closed set classes) also attempting to correctly classify open set examples into open set classes.

[0034] In FSOSR-based methods, for small-shot tasks (e.g., small-shot classification tasks), a neural network may be trained to correctly identify closed-set examples whose classes are described by a few labeled support examples, while also rejecting open-set examples without class hints. In some cases, open-set recognition may be more challenging in FSOSR-based methods than in conventional OSR-based methods, for example, based at least in part on the relatively small number of support examples utilized in each FSL episode and / or the different closed and open sets across multiple FSL episodes. For example, a closed-set class in one FSL episode may be selected as an open-set class in a different FSL episode, and vice versa, as will be described in more depth below. Therefore, there is a need for improved FSOSR performance that can correctly separate closed-set examples and open-set examples into two parts while maintaining classification capabilities for closed-set examples, and further, the improved FSOSR performance can be performed in a task-agnostic manner.

[0035] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") for performing few-shot open set recognition (FSOSR) using a machine learning classifier based on learned task-agnostic open set prototype classes. In some examples, the machine learning classifier may be a neural network classifier. The few-shot neural network classifier may be used to perform improved classification tasks on various data samples (e.g., in various data domains) using OSR by determining (e.g., learning) a task-agnostic (e.g., plot-agnostic) open set prototype representation that may be used to classify an open set query sample as belonging to an open set class and / or to reject an open set query sample as belonging to an open set class. In some examples, a classifier based on the FSOSR task-agnostic open set prototype may classify a predicted task-agnostic open set class as an open set that will be rejected from downstream or subsequent classification, detection, or other machine learning processes, and the like.

[0036] In some examples, these systems and techniques may generate a task-agnostic open set prototype to which each query sample is mapped if the true class (e.g., real data class) of the query sample is not available (e.g., because the query sample is included as an open set sample in a given FSOSR episode). In some aspects, a task-agnostic open set prototype may be generated (e.g., learned) such that the task-agnostic open set prototype is a global second-best class for any given query sample across multiple different FSO SR tasks or episodes. Based on the learned task-agnostic open set prototype as a global second-best classification, the open set samples can be accurately classified into the task-agnostic open set class in all tasks. In some examples, one or more scalar-valued scaling factors may be learned along with the task-agnostic open set prototype. The scaling factors may also be task-agnostic. In some cases, the learned scaling factors may be learned to refine and implement the task-agnostic open set prototype described herein as a global second-best classification across multiple classification tasks for both open set and closed set query examples, as will be described in more depth below.

[0037] An illustrative example of a neural network for generating task-agnostic open-set prototypes for few-shot open-set recognition (FSOSR) is described in Appendix A attached hereto.

[0038] Various aspects of the disclosure will be described with respect to the figures. Figure 1 An example implementation of a system on a chip (SoC) 100 is illustrated, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed among multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from the memory block 118.

[0039] The SoC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures, speech, and / or other interactive user actions or inputs. In one specific implementation, the NPU is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SoC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120 (which may include a global positioning system). In some examples, the sensor processor 114 may be associated with or connected to one or more sensors for providing sensor input to the sensor processor 114. For example, the one or more sensors and the sensor processor 114 may be provided in, coupled to, or otherwise associated with the same computing device.

[0040] In some examples, one or more sensors may include one or more microphones for receiving sound (e.g., audio input), including sound or audio input that can be used to perform tasks such as keyword spotting (KWS). In some cases, the sound or audio input received by one or more microphones (and / or other sensors) may be digitized into data packets for analysis and / or transmission. The audio input may include ambient sound near a computing device associated with SoC 100 and / or may include speech from a user of a computing device associated with SoC 100. In some cases, a computing device associated with SoC 100 may additionally or alternatively be communicatively coupled to one or more peripheral devices (not shown) and / or be configured to communicate with one or more remote computing devices or external resources, for example using a wireless transceiver and a communication network such as a cellular communication network.

[0041] SoC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into CPU 102 may include code for searching a lookup table (LUT) for a stored multiplication result corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into CPU 102 may also include code for disabling a multiplier during a multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. In addition, the instructions loaded into CPU 102 may include code for storing a calculated multiplication product of an input value and a filter weight when a lookup table miss of the multiplication product is detected.

[0042] SoC 100 and / or its components may be configured to perform audio signal processing using one or more machine learning networks and / or machine learning techniques according to aspects of the present disclosure and as described herein. For example, SoC 100 and / or its components may be configured to perform keyword spotting according to aspects of the present disclosure. In some examples, aspects of the present disclosure may increase the accuracy and / or efficiency of keyword spotting (KWS) by using one or more neural network classifiers to perform small sample hint learning (FSL) and / or open set recognition (OSR).

[0043] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems may include algorithms and statistical models that a computer system can use to perform various tasks by relying on patterns and inferences without the use of explicit instructions. An example of an ML system is a neural network (also called an artificial neural network), which may include a group of interconnected artificial neurons (e.g., a neuron model). Neural networks can be used in various applications and / or devices, such as speech analysis, audio signal analysis, image and / or video decoding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, etc.

[0044] Each node in a neural network can simulate a biological neuron by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​restrict the way in which the input data is related to the output data. For example, the input data of each node can be multiplied by the corresponding weight value, and the products can be added. The sum of the products can be adjusted by an optional bias, and an activation function can be applied to the result to produce an output signal or "output activation" (sometimes called a feature map or activation map) of the node. The weight values ​​can initially be determined by an iterative flow of training data through the network (for example, the weight values ​​are established during the training phase when the network learns how to identify a specific class by its typical input data characteristics).

[0045] There are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multi-layer perceptron (MLP) neural networks, transformer neural networks, and the like. For example, a convolutional neural network (CNN) is a feed-forward artificial neural network. A convolutional neural network may include a collection of artificial neurons that each have a receptive field (e.g., a spatially local region of an input space) and collectively tile an input space. RNNs work on the principle of saving the output of a layer and feeding that output back to the input to help predict the results of that layer. GANs are generative neural networks that learn patterns in input data so that a neural network model can generate new synthetic outputs that are reasonably likely to come from the original data set. GANs may include two neural networks operating together, including a generative neural network that generates a synthetic output and a discriminative neural network that evaluates the authenticity of the output. In an MLP neural network, data may be fed to an input layer, and one or more hidden layers provide a level of abstraction for the data. The output layer may then be predicted based on the abstract data.

[0046] Deep learning (DL) is an example of a machine learning technique and can be considered a subset of ML. Many DL methods are based on neural networks, such as RNN or CNN, and utilize multiple layers. The use of multiple layers in a deep neural network allows for the gradual extraction of higher-level features from a given raw data input. For example, the output of the first layer of artificial neurons becomes the input of the second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input of the third layer of artificial neurons, and so on. The layer located between the input and output of the entire deep neural network is generally referred to as a hidden layer. Hidden layer learning (e.g., being trained) transforms the intermediate input from the previous layer into a slightly more abstract and complex representation that can be provided to subsequent layers until the final or desired representation is obtained as the final output of the deep neural network.

[0047] As mentioned above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and output is generated by the output nodes of the output layer. A deep learning network typically includes multiple hidden layers. Each layer of the neural network may include a feature map or activation map, which may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. A node may include one or more weights for indicating the importance of a node in one or more layers in the layer. In some cases, a deep learning network may have a series of many hidden layers, where early layers are used to determine simple and low-level characteristics of the input, and later layers establish a hierarchical structure of more complex and abstract characteristics.

[0048] Deep learning architectures can learn hierarchies of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases. Deep learning architectures can perform particularly well when applied to problems with a natural hierarchy. For example, the classification of motorized vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher layers to recognize cars, trucks, and airplanes.

[0049] Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0050] The connections between neural network layers can be fully connected or partially connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In a fully connected neural network 202, a neuron in a first layer may communicate its output to every neuron in a second layer, such that every neuron in the second layer will receive input from every neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strengths may have different values ​​(e.g., 210, 212, 214, and 216). The locally connected connectivity patterns may produce spatially distinct receptive fields in higher layers because higher layer neurons in a given region may receive input that is tuned through training to properties of a limited portion of the total input to the network.

[0051] Figure 3 is a diagram illustrating an example of a few-shot learning (FSL) scenario 300. In some examples, FSL and / or FSL-based methods can be used to perform few-shot classification, where a classifier (e.g., a neural network or other machine learning classifier) ​​can generalize and extend inference to include new classes not seen in the training set, assuming that each new class has only a small number of examples. For example, an FSL-based neural network classifier can be trained to correctly classify unseen test classes based on a relatively small number of support examples provided at inference time, as will be described in more depth below.

[0052] As shown in the figure, Figure 3 An example of a 3-class 5-sample FSL scenario is depicted, where a neural network classifier (not shown) receives as input a support set comprising a total of 15 labeled support examples, the support set being partitioned into three previously unseen classes (e.g., a first class 310, a second class 320, and a third class 330), each with five support examples. In an illustrative example, the 3-class 5-sample FSL scenario 300 can be associated with a prototype network that learns a metric space in which classification can be performed by computing distances to prototype representations of each class.

[0053] Using prototypical networks for FSL can be based on the idea that there exists an embedding where points for each class are clustered around a single prototypical representation. Figure 3 As illustrated, it can be seen that the five supporting examples included in class 310 are clustered around prototype representation 315; it can be seen that the five supporting examples included in class 320 are clustered around prototype representation 325; it can be seen that the five supporting examples included in class 330 are clustered around prototype representation 335; and so on. The prototype network may include one or more neural networks that learn (e.g., during training) a nonlinear mapping of inputs in an embedding space. Using the learned embedding space, the prototype network can take the prototype of each previously unseen class as the average of the support set of the prototype network in the embedding space. For example, prototype representation 315 can be determined as the average of the five embedded supporting examples of class 310; prototype representation 325 can be determined as the average of the five embedded supporting examples of class 320; prototype representation 335 can be determined as the average of the five embedded supporting examples of class 330; and so on.

[0054] Classification may then be performed on the embedded query point by determining the class prototype that is closest to the query point. For example, embedded query point 370 may be classified into class 320 based on determining that the distance from embedded query point 370 to prototype representation 325 is less than the distance from embedded query point 370 to either of the remaining prototype representations 315 and 335. In some examples, embedded query points (e.g., embedded query point 370) may be classified based on the Euclidean distance between the embedded query point and each of the prototype representations, but it should be noted that various other distance metrics and / or determinations may also be utilized without departing from the scope of the present disclosure.

[0055] Figure 4A 400a. In some examples, the FSL scenario 400a can be used with one or more prototype networks (e.g., such as those described above with respect to Figure 3 As previously described, in an N-class, M-sample FSL scenario (e.g., a round of inference on a support set containing N*M support examples), FSL can be performed with the goal of correctly classifying one or more queries in the query set into N classes that were not seen during training but are known based on receiving M support samples for each previously unseen class. In some examples, a purely FSL-based approach does not consider open-set classes among the N classes, which means that a purely FSL-based approach (such as the purely FSL-based approach of FSL scenario 400a) will only classify a query into one of the N closed-set classes.

[0056] In an illustrative example, the FSL scene 400a is a 3-class 5-sample FSL scene, which means that classification is performed on three classes 410, 420, and 430 in an evaluation (e.g., inference) setting, and that five embedded support examples for each of the three classes are used to generate corresponding prototype representations 415, 425, and 435. In the context of FSL-based image classification, the three classes 410, 420, and 430 may each represent an image classification (e.g., landscape, selfie, indoor, etc.) and / or an object classification (e.g., person A, dog, face, etc.). For example, class 410 may represent an image classification "landscape", class 420 may represent an image classification "selfie", and class 430 may represent an image classification "indoor". Because FSL scene 400a is a 3-category, 5-sample FSL scene, the support set S (not shown) may include a total of 15 support examples (e.g., five example landscape images labeled with the "landscape" classification 410, five example selfie images labeled with the "selfie" classification 420, and five example indoor images labeled with the "indoor" classification 430).

[0057] like Figure 4AAs shown, the query set Q may include a combination of closed set queries and open set queries. The closed set queries may include test samples from known classes (e.g., landscape, selfie, or indoor test images corresponding to the support set classes 410, 420, 430, respectively). Figure 4A As shown, the query set includes three closed-set test samples for each class 410, 420, 430. An open-set query may represent a test sample from an unknown class and is depicted as an open circle containing a "?" inside. For example, an open-set query may include test samples that do not belong to the landscape, selfie, or indoor classifications (e.g., an open-set query may include test images of boats, sporting events, urban landscapes, etc.). In some aspects, it should be noted that the open-set query / test samples may take various other forms with varying degrees of similarity to the closed-set test samples (e.g., because an open set may be defined to include any test sample that does not match a known class). The test samples of the open-set query may be associated with one or more unseen and unsupported classes that were not seen in the initial training dataset (e.g., used to initially train a neural network classifier associated with the FSL scene 400a) or the support set S (e.g., provided to the neural network classifier at inference time and used to perform FSL).

[0058] Because the open set class is unseen, unknown, or otherwise unsupported by the neural network classifier associated with the FSL scenario 400a, each of the open set queries will be misclassified into one of the three known (e.g., closed set) classes 410, 420, 430. For example, Figure 4A The decision boundary depicted as a dashed line dividing the embedding space in can represent a (wrong) closed-set classification that can be determined for an open-set query. In some examples, the misclassification of queries from open-set classes can occur due to the design of existing FSL methods (e.g., such as FSL scenario 400a), which are limited to known closed-set classes and cannot consider any open-set classes regardless of the distance between the open-set query and the nearest prototype representation determined for one of the closed-set classes.

[0059] In some real-world scenarios, unseen and unsupported open-set classes may be frequently encountered and are consistently misclassified by existing FSL methods as belonging to known classes from the closed set of FSL classes seen in training or included in the support set S. It is necessary to handle or reject any unseen and unsupported classes during FSL-based classification so that novel open-set classes are not misclassified into seen closed-set classes.

[0060] Figure 4B A first example episode 400b of a few-shot open set recognition (FSOSR) method according to one or more aspects of the present disclosure is illustrated. Figure 4CA second example scenario 400c of the FSOSR method according to one or more aspects of the present disclosure is illustrated. In an illustrative example, the first FSOSR scenario 400b and the second FSOSR scenario 400c may represent or be applied to the same method as described above with respect to Figure 4A The illustrated example FSL scenario 400a describes the same basic sample set S and query set Q.

[0061] The systems and techniques described herein can perform FSOSR to distinguish unknown open sets from known closed set classes, while still performing FSL to correctly classify closed set samples into one of the known closed set classes. For example, these systems and techniques can receive as input a query set that includes a combination of closed set test samples (e.g., which should be correctly classified as belonging to one of the known classes) and open set test samples (e.g., which belong to one or more unknown classes and should not be classified as belonging to any of the known classes).

[0062] As will be explained in more depth below, these systems and techniques can be trained to learn task-agnostic open set prototypes (e.g., task-agnostic open set classes) to which each sample is mapped if its true class is not available. The same task-agnostic open set prototypes can be used to detect and reject open sets associated with multiple (e.g., different) episodes by classifying the open sets as task-agnostic open set prototypes and rejecting their inclusion in subsequent closed set classifications.

[0063] In one illustrative example, the open set of test samples with unknown or unseen classes may vary from episode to episode based on the selection of N classes in the support set S provided as inference input for each episode. Figure 4B A first episode 400b is illustrated having a first query open set with unseen classes, and Figure 4C A second episode 400c is illustrated having a second query open set with unseen classes. The first episode 400b and the second episode 400c are built on the same base query set (e.g., Figure 4B and Figure 4C). Based on the selection of N classes identified or included in the support set provided for each of episode 400b and episode 400c, the composition of the open set changes from episode 400b to episode 400c. The composition of the closed set classification also changes from episode 400b to episode 400c. For example, the first classification 410b (e.g., of the first episode 400b) includes a different set of query / test samples from the first classification 410c (e.g., of the second episode 400c); the second classification 420b (e.g., of the first episode 400b) includes a different set of query / test samples from the second classification 420c (e.g., of the second episode 400c); and the third classification 430b (e.g., of the first episode 400b) includes a different set of query / test samples from the third classification 430c (e.g., of the second episode 400c); etc.

[0064] Figure 5A and Figure 5B 2 is a diagram illustrating example FSOSR tasks 500a and 500b, respectively, wherein the classes for open sets and closed sets vary between the two tasks. As used herein, the terms "task" and "episode" may be used interchangeably (e.g., a FSOSR task may also be referred to as a FSOSR episode). As shown, FSOSR task 500a and FSOSR task 500b may be performed based on the same basic query and / or query set, wherein the name of the query set varies between an open set and a closed set across the two episodes. For example, in the context of FSOSR task 500a, classes "A" and "B" are used as closed sets 510a and 520a, respectively, and classes "C" and "D" are used as open sets 530a and 540a, respectively.

[0065] In the context of FSOSR task 500b, class "A" is no longer considered a closed set, but is used as an open set 510b. Similarly, class "C" is no longer considered an open set in the context of FSOSR task 500b, but is used as a closed set 530b. Classes "B" and "D" may be treated in the same manner for both FSOSR task 500a and FSOSR task 500b (e.g., closed set 520a and closed set 520b may be the same, and open set 540a and open set 540b may be the same). Because open set classes and closed set classes are determined or selected based on the same base query set and variations between FSOSR task 500a and FSOSR task 500b, some FSOSR methods may determine an open set prototype that correctly classifies open set queries in one of the two FSOSR tasks 500a, 500b, but not both. For example, some FSOSR methods may determine task-specific (e.g., episode-specific) open set prototypes that depend on the specific arrangement of open and closed set classes included in a given FSOSR task or episode, and therefore may not generalize to other FSOSR episodes.

[0066] According to various implementations, systems and techniques can be used to perform FSOSR using open set classification and / or rejection that is task agnostic (e.g., executable across different episodes). Figure 5A and Figure 5B As shown, FSOSR tasks 500a and 500b include example task agnostic open set prototypes c ag 572, which is the global second-best class for all samples in the embedding space. As will be described in more depth below, the example task-agnostic open set prototype c ag Open set examples that can be used to capture (e.g., classify and / or reject) both tasks.

[0067] Figure 6 is a diagram illustrating an example of a task-agnostic open set prototype network 600 for performing FSOSR using task-agnostic (e.g., episode-agnostic or non-episode-specific) open set identification and / or rejection. In one illustrative example, the task-agnostic open set prototype network 600 can perform FSOSR to adapt to varying open sets and / or varying open set selection between episodes (e.g., between tasks). In some aspects, the task-agnostic open set prototype network 600 can perform FSOSR by using learned task-agnostic open set prototypes c ag FSOSR is performed by classifying and rejecting open set queries. The remaining closed set queries can be classified into one or more closed set classes based on using metric-based learning to learn a metric space in which a distance metric can classify closed set queries or other input samples.

[0068] In the context of the following discussion, a FSOSR task (e.g., also referred to as a FSOSR episode) may be given by:

[0069]

[0070] Here, T represents the FSOSR task or episode; S n is a closed set sample C S The set of support provided; is an open set (e.g., samples with novel or unknown classes for which no supporting examples are given); Q is the query set (e.g., inference input) provided for the FSOSR task T; and N represents the set of samples included in the support set S. n The query set Q may include the number of classes in C S and Sampled known queries Q S and unknown queries

[0071] In some examples, the closed set C S and open set Sampling is performed to configure each FSOSR episode (e.g., training task T), where C S ={1, ..., N}, |C S |=N, and As described previously, a closed class in a first episode may later be selected as an open class in a subsequent episode (e.g., because C S and are randomly selected from the same class label space).

[0072] Only the closed set C S Provide support set S n ,in Here, M is the number of support samples provided for the nth closed class, and (For example, support samples The label of ) is n.

[0073] In one illustrative example, training may be performed in which a machine learning model (e.g., a neural network model) associated with a task-agnostic open set prototype network 600 learns from a plurality of N-class, M-sample pseudo FSOSR episodes (e.g., a plurality of N-class, M-sample pseudo FSOSR tasks T, as given by equation (1) and described above). For example, each of the pseudo FSOSR episodes may include N known classes, each class having M supporting examples (e.g., such that each pseudo FSOSR episode includes a total of N*M supporting examples) and one or more pseudo unknown (e.g., pseudo open set) classes without any supporting examples. The pseudo episodes used in training may be designed to simulate the FSOSR inference task by subsampling the classes and data points.

[0074] Regarding the FSOSR inference task, inference can be performed on episodes each associated with a support set S and a query set Q. For example, Figure 6 A support set S (indicated as 612) is illustrated, which includes one or more closed sets (indicated as 613a, 613b, ..., 613n, and each marked with (K) to indicate a closed set) of a known class, providing support examples for the known class. For example, the support set 612 may correspond to the support set S described above. n (For example, where In some aspects, the support set 612 may include: a first closed set S 1 , which includes M support samples for the first class; the second closed set S 2 , which includes M support samples for the second class; ...; and the nth closed set S N , which includes M support samples for the nth class.

[0075] Figure 6 Also illustrated is a query set Q (indicated as 616), which includes a closed query set Q belonging to a known class K. S (e.g., indicated as 617) and the set of open queries belonging to unknown class U with no supporting examples (e.g., indicated as 619.) It should be noted that although the closed sample sets 613a, 613b, ..., 613n (e.g., of the support set 612) and the closed sample set 617 (e.g., of the query set 616) may correspond to the same underlying known class, it is not necessarily the case that the constituent samples within 613a, 613b, ..., 613n are the same as the constituent samples within 617.

[0076] In one illustrative example, the support set S (e.g., 612) may include M samples for each of the N classes, as described above. The query set Q (e.g., 616) may include one or more queries from the N known classes (e.g., 617), and may also include queries from the N known classes. U One or more queries of unknown class (e.g., 619), where:

[0077] |S|=N·M

[0078] |Q|=(N+N U )·M Q

[0079] Here, M Q It can represent the number of queries for each class. In some cases, M Q can be equal to M, but M Q It may also be greater or less than M. At inference time, the example task-agnostic open set prototype network 600 may not have seen all the classes in the evaluation dataset. In some examples, the evaluation dataset may be the same as the query set 616. A dataset including N known classes with support samples (e.g., set 617) and N classes without support samples may be used. U Inference is performed in an episode of unknown open set classes (e.g., set 619). Note that in the inference episode, the evaluation dataset (e.g., 617) does not necessarily include M support examples for each of the N classes, as is the case with the support set S (e.g., 612).

[0080] As previously described, the task-agnostic open set prototype network 600 may perform FSOSR based at least in part on one or more metric-based learning methods. For example, in an N-class M sample scenario, the task-agnostic open set prototype network 600 may be based on a given query x and a closed set class C S = the distance between {1, ..., N} to determine the class of the query x. Class n∈C S The representative features (e.g., prototypes) of S nThe features of the support samples in are averaged to generate.

[0081] For example, the task-agnostic open-set prototype network 600 can calculate the corresponding prototype of each class by using the average value of the embedded support samples of each class n To determine or otherwise obtain the closed set C S N prototypes, where:

[0082]

[0083] Here, f is an encoder (e.g., a feature embedding network) and exist Figure 6 In the example of , an encoder or feature embedding network 620 may be used to generate an embedding for each support sample included in the support set 612. In some examples, each sub-portion of the support set 612 (e.g., S 1 ,...,S N ) may be provided to separate instances of the feature embedding network 620. In some examples, a sub-portion of the support set 612 may be provided to a single instance of the feature embedding network 620 in order to generate a corresponding embedding for the support set 612.

[0084] As shown, the output of the feature embedding network 620 (e.g., the embedding determined for each support sample included in the support set 612) can be used to generate N closed set prototypes c based on equation (2): 1 、c 2 , ..., c N The resulting closed prototype 632 can be a task-related prototype, because each closed prototype c n is generated depending on the particular choice of closed set within the support set 612.

[0085] The feature embedding network 620 may be further configured to generate one or more embeddings based on receiving as input some or all query samples included in the query set 616. For example, the feature embedding network 620 may be configured to determine a task-related closed set prototype c n 632, and can also be used to determine an embedding for each query sample of the query set 616. Subsequently, the embedding c can be determined based at least in part on the reference prototype embedding c n 632 Analyze the embedding determined for each query (e.g., in query set 616) to identify the closed set of queries Q S 617 and unknown / open set queries Metric-based classification is performed as will be described in more depth below.

[0086] In some aspects, the feature embedding network 620 may receive the support set 612 and the query set 616 as input (e.g., for each episode). In an illustrative example, the feature embedding network 620 may be a neural network or other machine learning network (e.g., a prototype c ) that generates an embedding based on the support set 612 and / or the query set 616. n 632 may be determined by averaging the embeddings generated by the feature embedding network f620 for each class N).

[0087] Based on the task-related closed set prototype c n , the task-agnostic open set prototype network 600 can determine or otherwise obtain the probability distribution over N known classes, so that the classification probability of a given query x on each class n is given by:

[0088]

[0089] In some aspects, the classification probability given in equation (3) can be compared with the distance metric d(·,·) (e.g., in Figure 6 In one illustrative example, the distance metric 640 may be proportional to the negative value of the distance metric 640. 2 To determine the distance metric 640, various other distance metrics may be utilized without departing from the scope of the present disclosure. Based on equation (3), the systems and techniques described herein can use the task-agnostic open set prototype network 600 to minimize the negative log probability of the true class.

[0090] In an illustrative example, the probability distribution of equation (3) can be used to determine the prototype representation c of the query example x and each class by n 632 (e.g., using distance metric 640 / Equation (3)) to classify a given input query x. For example, if the input query example x is closest to class number three (e.g., among the N classes), a relatively high probability can be determined for class three, and a relatively low probability can be determined for the remaining N-1 classes. For example, these probabilities can be determined based on equation (3), which itself can be determined based on distance metric 640.

[0091] In one illustrative example, the systems and techniques described herein may further determine one or more task-agnostic open set prototypes c ag , the one or more task-agnostic open set prototypes are Figure 6619 as a task-agnostic open set prototype 652. As will be described in more depth below, the task-agnostic open set prototype 652 can be a learnable prototype that can be used to classify and / or reject open set queries across different tasks or scenarios. In some aspects, the task-agnostic open set prototype 652 can be generated and used to extend the metric-based classification previously described to open set queries 619 as well as closed set queries 617. In some examples, the task-agnostic open set prototype 652 is consistent with the overall task (e.g., based on task agnosticism).

[0092] Given in the task Based on the given input query x, the closed set Q S 617 or unknown / open set For the membership of , we can consider the following two order relations for the input query x:

[0093] x∈Q S :d(f(x),c y )<min(d(f(x),c ag ), d(f(x), c n )) Equation (4)

[0094] where y is the true data class of x and n ≠ y, and:

[0095]

[0096] Combining equations (4) and (5) provides a general relationship that is independent of the membership of the input query x:

[0097] d(f(x),c y )<d(f(x),c ag )<d(f(x),c n ) Equation (6)

[0098] In an illustrative example, equation (6) can be applied to all n, where n≠y. Based on equation (6), the task-agnostic open set prototype c ag 652 can be viewed as a prototype c corresponding to the real data class of the query. n In other words, when the input query x is selected as an open set sample (e.g., belonging to the open set 619), the task-agnostic open set prototype c ag 652 can be reliably used to reject open set samples x. As will be described in more depth below, by learning the open set prototype c in a task-agnostic (e.g., episode-agnostic) manner ag 652, the performance of the example network 600 may be improved for a metric-based FSOSR classification task.

[0099] As mentioned before, we can learn task-agnostic open-set prototypes c ag 652, so that if any given query sample of the query set 616 is selected as an open set sample for which the true class is not available, then the task-agnostic open set prototype represents the closest classification match for the given query sample. In an illustrative example, the learning task-agnostic open set prototype c ag 652, so that it satisfies the global second-best classification criterion for any given query sample (e.g., for a given query sample, the only classification with a large probability based on equation (3) is the actual, true base class of the query sample when the query sample belongs to the known closed set 617; if the query sample belongs to the unknown open set 619, the true base class is not available to the FSOSR network 600, and the task-agnostic open set prototype c ag 652 is the best or most likely class for the open set query sample within the context of the current episode).

[0100] Task agnostic open set prototype c ag 652 can be implemented as a D-dimensional learnable feature associated with FSOSR network 600, as previously described. In some aspects, one or more learnable scaling factors can be used to better satisfy the order relationship of equation (6) so that a single c ag So that equation (6) is satisfied in a task-agnostic manner across multiple different tasks or episodes.

[0101] like Figure 6 As shown, the example task-agnostic open set prototype network 600 may also include a learnable scaling factor θ w and / or θ b In some examples, θ w and / or θ b Can be a scalar value. In one illustrative example, a scaling factor θ can be learned w and / or θ b A distance metric 642 may be used to determine scaling, which is Figure 6 The distance metric θ is depicted as a scaled w ·d ag +θ b Here, the distance value d ag can be determined as f(x), the embedding of the query sample x, and the task-agnostic open set prototype c ag 652 distance metric between ag =d(f(x),c ag )). In some aspects, the distance metric 640 may be used to determine the distance value d ag , this distance metric converts the distance value d ag A scaled distance metric 642 is provided as input.

[0102] In one illustrative example, the learnable scaling factor θ may be learned in conjunction or combination with the learnable task-agnostic open set prototype 652. w and / or θ b In some aspects, the example FSOSR network 600 may embed features into the network f620, the task-agnostic open set prototype c ag 652. Scaling factor θ w , and a scaling factor θ b One or more (or all) of are implemented as learnable components or learnable modules that are learned during training (e.g., as will be described in more depth below).

[0103] In an illustrative example, the task-agnostic open set prototype c ag 652 may be used to compute the softmax probability output 660 for a given input query x for N+1 classes as follows (e.g., representing N closed set classes + c ag ):

[0104]

[0105]

[0106] Here, n=1, ..., N. In some aspects, the task-agnostic open set prototype c based on learning ag 652 and the learned scaling factor θ w and θ b , a given input query sample x may be classified as an open set sample or a closed set sample using classification block 690. For example, a given input query sample x may be classified as an open set sample or a closed set sample based on determining p ag (x)>δ and is classified as an open set sample at classification block 690, where δ is a predefined threshold for classifying open set query samples. Figure 6 As shown, if the classification block 690 determines that p ag (x)>δ is false (e.g., if classification block 690 instead determines p ag <δ), then the input query sample x can be classified into the nearest closed (e.g., known) class in other ways. Among them The argmax-based classification of closed-set input query samples is Figure 6 Depicted as a closed set classification box 680.

[0107] As mentioned above, the probability distribution p of equations (7) and (8) is n (x) and p ag (x) can be used respectively by feature embedding network f620 and task agnostic open set prototype c ag 652. Scaling factor θ w, and a scaling factor θ b In some examples, feature embedding network f620, task-agnostic open set prototype c ag 652. Scaling factor θ w , and a scaling factor θ b To help d(f(x),c ag ) so that the order relationship of equation (6) is obtained or otherwise realized.

[0108] In one illustrative example, the example FSOSR network 600 can be trained to optimize its feature extractor (e.g., feature embedding network f620), task-agnostic open set prototypes c by minimizing the total loss function given by ag 652 and scaling factor θ w ,θ b :

[0109]

[0110] Here, In some aspects, based on the task-agnostic open set prototype c ag 652, the training process may consider (N+I)-class classification, where 1, ..., N represent N classes in the closed set of the support set 613 and the closed query set 617, and (N+1) represents the number of classes in the closed set of the prototype c ag 652 represents the task-agnostic open set class. In some examples, the true data class of the open set samples can be set to N+1 during training, and p (N+1) =p ag Based on the above, the open set loss Can be determined as:

[0111]

[0112] Here, is a hyperparameter used to balance between open and closed sets. In some examples, when yi=N+1, otherwise

[0113] In some aspects, the example FSOSR network 600 may be explicitly forced (e.g., during training) to satisfy the order relationship of equation (6) by implementing a closed set of suboptimal losses as follows:

[0114]

[0115] Here, represents the second best loss, where x i is a closed set sample, and p N+1 (xi ) \{yi} represents the softmax probability of the task-agnostic open set class, which is calculated without considering its true data class y.

[0116] In some aspects, the example FSOSR network 600 can additionally be trained to adjust the task-agnostic open set prototypes c ag 652 to prototype c n 632, and adjust the distance between the prototypes at the same time. In an illustrative example, the cosine entropy of each pair of prototypes can be maximized by the following formula:

[0117]

[0118] Here, represents the pairwise prototype entropy loss, where And N pair =N(N-1) / 2 is the number of prototype pairs.

[0119] Finally, as mentioned above, the feature extractor (e.g., feature embedding network f620), the task-agnostic open-set prototype c ag 652 and scaling factor θ w ,θ b It can be optimized during training of the example FSOSR network 600 by minimizing the total loss function given by equation (9).

[0120] Figure 7 is a flow chart illustrating an example of a process 700 for performing few-shot open set recognition (FSOSR) using task-agnostic open set prototypes. At block 702, the process 700 includes determining one or more prototype representations based on a plurality of support samples associated with one or more classes of a data sample.

[0121] Each stereotype representation can be associated with one of one or more classes. For example, you can use Figure 6 The one or more prototype representations may be determined by the example task-agnostic open set prototype network 600 illustrated in FIG. 1 . In some examples, the one or more prototype representations may be determined by machine learning and / or a neural network encoder (such as Figure 6 The one or more prototype representations may be determined based on receiving a plurality of support samples as input, wherein each support sample is associated with a known class (e.g., labeled with a known class). For example, the one or more prototype representations may be determined based on the feature embedding network 620 illustrated in FIG. Figure 6The one or more prototype representations may be determined based on a plurality of support samples in the support set 612 illustrated in FIG. In some cases, the plurality of support samples may include one or more sub-portions, wherein each sub-portion includes support samples associated with a particular class in the one or more classes of the data sample. For example, the plurality of support samples may include Figure 6 In some examples, the encoder (e.g., feature embedding network 620) can generate multiple embeddings for multiple support samples associated with each class. Then, a prototype representation can be determined for each class as the average of the embedded support samples of the class. For example, the average of the embedded support samples of each class can be determined based on a Euclidean distance metric to generate a set of task-related closed set prototypes, such as Figure 6 The prototype 632 illustrated in .

[0122] At block 704, process 700 includes determining a task-agnostic open set prototype representation. For example, the task-agnostic open set prototype representation may include Figure 6 In some examples, one or more prototype representations and the task-agnostic open set prototype representation can be determined in the same learning metric space. For example, in a neural network encoder (such as Figure 6 One or more prototype representations and task-agnostic open-set prototype representations are determined in the same learned embedding space of the feature embedding network 620 illustrated in (e.g., the task-agnostic open-set prototype representations can be determined in the same embedding space associated with the feature embedding network 620, which is used to generate embeddings associated with the one or more prototype representations).

[0123] In some examples, multiple supporting samples (e.g., Figure 6 The support set 612 illustrated in ) may be obtained for or otherwise associated with a single few-shot learning (FSL) episode. In some cases, the task-agnostic open-set prototype representation may be an episode-agnostic open-set prototype representation.

[0124] At block 706, process 700 includes determining one or more distance metrics for each of the one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open set prototype representation. For example, the one or more distance metrics may be determined using distance function 640, such as Figure 6 In some examples, the distance metric may be the Euclidean distance between each query sample and each of the prototype representation and the task-agnostic open set prototype representation. In some cases, one or more query samples may be included in a query set (such as Figure 6 The one or more query samples may include an open set query (eg, included in the open query set 619) and a closed set query (eg, included in the closed query set 617).

[0125] In some examples, determining one or more distance metrics further includes scaling the Euclidean distance metric between a given query sample and a task-agnostic open set prototype representation using one or more learned scaling factors. For example, one or more learned scaling factors (such as Figure 6 In some cases, determining one or more distance metrics may additionally or alternatively include scaling the Euclidean distance metric between a given query sample and each of the one or more prototype representations using one or more learned scaling factors.

[0126] In some cases, the one or more learned scaling factors may be determined as a first scalar value and a second scalar value. For example, the first scalar value and the second scalar value may be learned based on a loss function that implements the task-agnostic open set prototype representation as a task-agnostic global second-best classification for each query sample in the one or more query samples. In some examples, the one or more learned scaling factors may be learned as task-agnostic scaling factors. In some cases, the task-agnostic open set prototype representation may be a global second-best classification for multiple few-shot open sample recognition (FSOSR) episodes performed on the data sample.

[0127] At block 708 , process 700 includes classifying each query sample based on one or more distance metrics, wherein each query sample is classified into one of one or more classes associated with the one or more prototype representations or an open set class associated with the task-agnostic open set prototype representation.

[0128] For example, one or more softmax outputs such as Figure 6 ) to classify the query sample. In some examples, one or more softmax outputs may include one or more probability distributions determined based on a distance metric determined between a given query sample and each of the one or more closed set prototypes, and a probability distribution determined based on a distance metric (e.g., a scaled distance metric 642) determined between a given query sample and a task-agnostic open set prototype. In some examples, the softmax classification may be based on or otherwise utilize the distance metric described above with respect to block 706. For example, the softmax classification may classify a given query sample into one of the classes associated with the prototype representation or into an open set class associated with the task-agnostic open set prototype representation.

[0129] In some examples, classifying each query sample based on one or more distance metrics may include determining a probability distribution over one or more classes and open set classes. For example, the probability distribution may be determined based at least in part on a Euclidean distance metric determined between each query sample and a corresponding prototype associated with each of the one or more classes. Additionally or alternatively, the probability distribution may be determined based at least in part on a Euclidean distance metric determined between each query sample and a task-agnostic open set prototype representation. In some aspects, each query sample may be classified into one of the one or more closed set classes or into an open set class based on the probability distribution.

[0130] In some cases, open set rejection (OSR) may be performed based on a set of classified query samples that are classified into an open set class associated with a task-agnostic open set prototype representation. For example, performing OSR may include determining a probability that each query sample is included in an open set class associated with a task-agnostic open set prototype representation based on a probability distribution determined for the open set class (e.g., using a scaled distance metric between the task-agnostic open set prototype representation and / or a given query sample and the task-agnostic open set prototype representation), and comparing the determined probability with a predetermined threshold. In some cases, classifying each query sample based on one or more distance metrics may also include classifying the given query sample as being included in the open set class based on determining that the probability of the given query sample being included in the open set class is greater than a predetermined threshold, and classifying the given query sample as being included in the closed set class based on determining that the probability of the given query sample being included in the open set class is not greater than a predetermined threshold. In some cases, each query sample classified as being included in the closed set class may be further classified into a corresponding class in one or more classes. For example, each query sample may be classified based on maximizing a corresponding probability determined between each query sample and each corresponding class in one or more classes. In some examples, the corresponding probability may be an argmax probability.

[0131] In some examples, classifying each query sample may include providing each query sample to a trained small sample open sample recognition (FSOSR) neural network classifier, wherein the trained FSOSR neural network classifier includes at least a task-agnostic open set prototype representation and one or more distance scaling factors as learnable components. In some cases, the trained FSOSR neural network classifier may also include one or more feature embedding networks as learnable components. For example, the trained FSOSR neural network classifier may determine one or more distance metrics for each query sample based on determining an embedding for each query sample in one or more query samples using one or more feature embedding networks. For example, one or more prototype representations may be determined as an average embedding of a set of embeddings determined for multiple support samples associated with each class in one or more classes using embeddings. Each of the one or more distance metrics may then be generated based on a Euclidean distance metric between an embedding determined for a given query sample and an embedding determined for each of the one or more prototype representations associated with the one or more classes.

[0132] Additional aspects and examples related to the present disclosure are included in Appendix A attached hereto.

[0133] In some examples, the processes described herein (e.g., process 700 and / or any other process described herein) can be performed by a computing device, apparatus, or system. In one example, process 700 can be performed by a computer having Figure 8 The computing device or system of the computing device architecture 800 of the present invention may be executed. The computing device, apparatus or system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a laptop computer, a smart TV, a camera and / or any other computing device with resource capabilities to perform the processes described herein (including process 700 and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to convey and / or receive data, any combination thereof and / or other components. The network interface may be configured to convey and / or receive data based on an Internet Protocol (IP) or other types of data.

[0134] The components of the computing device may be implemented in circuit systems. For example, the components may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0135] Process 700 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0136] Additionally, process 700 and / or any other process described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0137] Figure 8An example computing device architecture 800 of an example computing device that can implement the various techniques described herein is illustrated. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other device. The components of the computing device architecture 800 are shown as using a connector 805 (such as a bus) to electrically communicate with each other. The example computing device architecture 800 includes a processing unit (CPU or processor) 810 and a computing device connector 805, which couples various computing device components including a computing device memory 815 (such as a read-only memory (ROM) 820 and a random access memory (RAM) 825) to the processor 810.

[0138] The computing device architecture 800 may include a cache of high-speed memory directly connected to the processor 810, very close to the processor, or integrated as part of the processor. The computing device architecture 800 may copy data from the memory 815 and / or the storage device 830 to the cache 812 for fast access by the processor 810. In this way, the cache can provide a performance improvement that avoids the processor 810 from delaying when waiting for data. These engines and other engines can control or be configured to control the processor 810 to perform various actions. Other computing device memories 815 can also be used. The memory 815 may include a variety of different types of memory with different performance characteristics. The processor 810 may include any general-purpose processor and hardware or software services (such as service 1832, service 2834, and service 3836 stored in the storage device 830) configured to control the processor 810, as well as a special-purpose processor in which software instructions are incorporated into the processor design. The processor 810 may be a self-contained system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0139] To enable user interaction with the computing device architecture 800, the input device 845 may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 835 may also be one or more of a plurality of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multi-mode computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 800. The communication interface 840 may generally command and manage user input and computing device output. There is no restriction on operating on any particular hardware arrangement, and therefore the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0140] The storage device 830 is a non-volatile memory and may be a hard disk or other type of computer readable medium that can store computer accessible data, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 825, a read-only memory (ROM) 820, and hybrids thereof. The storage device 830 may include services 832, 834, 836 for controlling the processor 810. Other hardware or software modules or engines are contemplated. The storage device 830 may be connected to the computing device connector 805. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer readable medium connected to the necessary hardware components (such as the processor 810, the connector 805, the output device 835, etc.) to perform the function.

[0141] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0142] The term "device" is not limited to one or a specific number of physical objects (such as a smart phone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. In addition, the term "system" is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0143] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in a combination of software or hardware and software. Additional components other than those components shown in the drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0144] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many of the operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0145] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0146] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or over a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as flash memory), memory or memory devices, disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, compact disks (CDs) or digital versatile disks (DVDs), any suitable combination thereof, and the like. Computer-readable media may have stored thereon code and / or machine-executable instructions that may represent a procedure, function, subroutine, program, routine, subroutine, module, engine, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, messaging, token passing, network sending, etc.

[0147] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0148] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0149] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0150] In the above description, the various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or jointly. In addition, the various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be appreciated that, in alternative aspects, each method can be performed in an order different from the described order.

[0151] Those of ordinary skill in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0152] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0153] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0154] Claim language or other language reciting "at least one of a set" and / or "one or more of a set" indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of a set" and / or "one or more of a set" does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0155] Various exemplary logic blocks, modules, engines, circuits and algorithmic steps described in conjunction with aspects disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0156] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0157] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0158] Exemplary aspects of the present disclosure include:

[0159] Aspect 1: A method for processing one or more data samples (e.g., a processor-implemented method), the method comprising: determining one or more prototype representations based on multiple support samples associated with one or more classes of the data sample, wherein each prototype representation is associated with one of the one or more classes; determining a task-agnostic open-set prototype representation, wherein the one or more prototype representations and the task-agnostic open-set prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open-set prototype representation; and classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the task-agnostic open-set prototype representation.

[0160] Aspect 2: A processor-implemented method according to Aspect 1, wherein determining the one or more distance metrics for each query sample further comprises: determining the Euclidean distance metric between a given query sample and each of the one or more prototype representations associated with the one or more classes; and determining the Euclidean distance metric between the given query sample and the task-agnostic open set prototype representation.

[0161] Aspect 3: The processor-implemented method according to Aspect 2 further includes: scaling the Euclidean distance metric between the given query sample and the task-agnostic open set prototype representation using one or more learned scaling factors.

[0162] Aspect 4: The processor-implemented method according to Aspect 3, further comprising: scaling the Euclidean distance metric between the given query sample and each of the one or more prototype representations using the one or more learned scaling factors.

[0163] Aspect 5: A method implemented by a processor according to any one of Aspects 3 to 4, wherein: the one or more learning scaling factors are determined as a first scalar value and a second scalar value; and the first scalar value and the second scalar value are learned based on a loss function, which implements the task-agnostic open set prototype representation as a task-agnostic global suboptimal classification for each query sample in the one or more query samples.

[0164] Aspect 6: A processor-implemented method according to Aspect 5, wherein: the one or more learned scaling factors are task-agnostic scaling factors; and the task-agnostic open set prototype representation is a global second-best classification for multiple small sample open sample recognition (FSOSR) episodes performed on the data sample.

[0165] Aspect 7: A method implemented by a processor according to any one of Aspects 1 to 6, wherein classifying each query sample based on the one or more distance metrics includes: determining a probability distribution over the one or more classes and the open set classes, wherein the probability distribution is determined at least in part based on a Euclidean distance metric determined between each query sample and a corresponding prototype associated with each of the one or more classes and a Euclidean distance metric determined between each query sample and the task-agnostic open set prototype representation; and classifying each query sample into one of the one or more classes or into the open set classes based on the probability distribution.

[0166] Aspect 8: The processor-implemented method according to Aspect 7, further comprising: performing open set rejection (OSR) based on a set of classified query samples classified into the open set class associated with the task-agnostic open set prototype representation.

[0167] Aspect 9: A method implemented by a processor according to any one of Aspects 7 to 8, wherein classifying each query sample based on the one or more distance metrics further includes: determining the probability of each query sample being included in the open set class associated with the task-agnostic open set prototype representation based on the probability distribution; and comparing the determined probability with a predetermined threshold.

[0168] Aspect 10: The processor-implemented method according to Aspect 9 further includes: classifying the given query sample as being included in the open set class based on determining that the probability of the given query sample being included in the open set class is greater than the predetermined threshold; and classifying the given query sample as being included in the closed set class based on determining that the probability of the given query sample being included in the open set class is not greater than the predetermined threshold.

[0169] Aspect 11: The processor-implemented method according to Aspect 10 further includes: classifying each query sample classified as being included in the closed set class into a corresponding class among the one or more classes, wherein each query sample is classified based on maximizing a corresponding probability determined between each query sample and each corresponding class among the one or more classes.

[0170] Aspect 12: The processor-implemented method of aspect 11, wherein the probability is an argmax probability.

[0171] Aspect 13: A processor-implemented method according to any one of Aspects 1 to 12, wherein classifying each query sample comprises: providing each query sample to a trained small sample open sample recognition (FSOSR) neural network classifier, wherein the trained FSOSR neural network classifier includes at least the task-agnostic open set prototype representation and one or more distance scaling factors as learnable components.

[0172] Aspect 14: The processor-implemented method of aspect 13, wherein the trained FSOSR neural network classifier further comprises one or more feature embedding networks as learnable components.

[0173] Aspect 15: A processor-implemented method according to Aspect 14, wherein determining the one or more distance metrics for each query sample further comprises: determining an embedding for each of the one or more query samples using the one or more feature embedding networks; determining the one or more prototype representations as an average embedding of a set of embeddings determined for the multiple support samples associated with each of the one or more classes; and determining each of the one or more distance metrics based on determining a Euclidean distance metric between the embedding determined for a given query sample and the embedding determined for each of the one or more prototype representations associated with the one or more classes.

[0174] Aspect 16: A processor-implemented method according to any one of Aspects 1 to 15, wherein the multiple support samples are obtained for a single few-shot learning (FSL) episode, and the task-agnostic open-set prototype representation is an episode-agnostic open-set prototype representation.

[0175] Aspect 17: A device for processing one or more data samples, comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: determine one or more prototype representations based on multiple support samples associated with one or more classes of the data sample, wherein each prototype representation is associated with one of the one or more classes; determine a task-agnostic open set prototype representation, wherein the one or more prototype representations and the task-agnostic open set prototype representation are determined in the same learning metric space; determine one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open set prototype representation; and classify each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or into an open set class associated with the task-agnostic open set prototype representation.

[0176] Aspect 18: An apparatus according to Aspect 17, wherein in order to determine the one or more distance metrics for each query sample, the at least one processor is further configured to: determine the Euclidean distance metric between a given query sample and each of the one or more prototype representations associated with the one or more classes; and determine the Euclidean distance metric between the given query sample and the task-agnostic open set prototype representation.

[0177] Aspect 19: The apparatus according to Aspect 18, wherein the at least one processor is further configured to: scale the Euclidean distance metric between the given query sample and the task-agnostic open set prototype representation using one or more learned scaling factors.

[0178] Aspect 20: The apparatus according to Aspect 19, wherein the at least one processor is further configured to: use the one or more learned scaling factors to scale the Euclidean distance metric between the given query sample and each of the one or more prototype representations.

[0179] Aspect 21: An apparatus according to any one of Aspects 19 to 20, wherein: the one or more learning scaling factors are determined as a first scalar value and a second scalar value; and the first scalar value and the second scalar value are learned based on a loss function, which implements the task-agnostic open set prototype representation as a task-agnostic global suboptimal classification for each query sample in the one or more query samples.

[0180] Aspect 22: An apparatus according to Aspect 21, wherein: the one or more learned scaling factors are task-agnostic scaling factors; and the task-agnostic open set prototype representation is a global suboptimal classification for multiple few-shot open sample recognition (FSOSR) episodes performed on the data sample.

[0181] Aspect 23: An apparatus according to any one of Aspects 17 to 22, wherein in order to classify each query sample based on the one or more distance metrics, the at least one processor is configured to: determine a probability distribution on the one or more classes and the open set classes, wherein the probability distribution is at least partially based on the Euclidean distance metric determined between each query sample and the corresponding prototype associated with each class in the one or more classes and the Euclidean distance metric determined between each query sample and the task-agnostic open set prototype representation; and classify each query sample into one of the one or more classes or into the open set classes based on the probability distribution.

[0182] Aspect 24: The apparatus according to Aspect 23, wherein the at least one processor is further configured to: perform open set rejection (OSR) based on a set of classified query samples classified into the open set class associated with the task-agnostic open set prototype representation.

[0183] Aspect 25: An apparatus according to any one of Aspects 23 to 24, wherein in order to classify each query sample based on the one or more distance metrics, the at least one processor is further configured to: determine the probability of each query sample being included in the open set class associated with the task-agnostic open set prototype representation based on the probability distribution; and compare the determined probability with a predetermined threshold.

[0184] Aspect 26: An apparatus according to Aspect 25, wherein the at least one processor is further configured to: classify the given query sample as being included in the open set class based on determining that the probability of the given query sample being included in the open set class is greater than the predetermined threshold; and classify the given query sample as being included in the closed set class based on determining that the probability of the given query sample being included in the open set class is not greater than the predetermined threshold.

[0185] Aspect 27: An apparatus according to Aspect 26, wherein the at least one processor is further configured to: classify each query sample classified as being included in the closed set class into a corresponding class among the one or more classes, wherein each query sample is classified based on maximizing a corresponding probability determined between each query sample and each corresponding class among the one or more classes.

[0186] Aspect 28: The apparatus according to Aspect 27, wherein the probability is an argmax probability.

[0187] Aspect 29: An apparatus according to any one of Aspects 17 to 28, wherein, in order to classify each query sample, the at least one processor is configured to: provide each query sample to a trained small sample open sample recognition (FSOSR) neural network classifier, wherein the trained FSOSR neural network classifier includes at least the task-agnostic open set prototype representation and one or more distance scaling factors as learnable components.

[0188] Aspect 30: The apparatus according to Aspect 29, wherein the trained FSOSR neural network classifier further comprises one or more feature embedding networks as learnable components.

[0189] Aspect 31: An apparatus according to Aspect 30, wherein in order to determine the one or more distance metrics for each query sample, the at least one processor is configured to: determine an embedding for each of the one or more query samples using the one or more feature embedding networks; determine the one or more prototype representations as the average embedding of a set of embeddings determined for the multiple support samples associated with each of the one or more classes; and determine each of the one or more distance metrics based on determining a Euclidean distance metric between the embedding determined for a given query sample and the embedding determined for each of the one or more prototype representations associated with the one or more classes.

[0190] Aspect 32: An apparatus according to any one of Aspects 17 to 31, wherein the multiple support samples are obtained for a single few-shot learning (FSL) episode, and the task-agnostic open-set prototype representation is an episode-agnostic open-set prototype representation.

[0191] Aspect 33: A computer-readable storage medium storing instructions, which, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of Aspects 1 to 32.

[0192] Aspect 34: An apparatus for processing one or more data samples, comprising one or more means for performing the operations according to any one of aspects 1 to 32.

Claims

1. A processor-implemented method for processing one or more data samples, comprising: determining one or more prototype representations based on a plurality of supporting samples associated with one or more classes of the data sample, wherein each prototype representation is associated with one of the one or more classes; Determining a task-agnostic open-set prototype representation, wherein the one or more prototype representations and the task-agnostic open-set prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open-set prototype representation; as well as Each query sample is classified based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the task-agnostic open set prototype representation.

2. The processor-implemented method of claim 1 , wherein determining the one or more distance metrics for each query sample further comprises: determining a Euclidean distance metric between a given query sample and each of the one or more prototype representations associated with the one or more classes; And determining a Euclidean distance metric between the given query sample and the task-agnostic open set prototype representation.

3. The processor-implemented method of claim 2 , further comprising: The Euclidean distance metric between the given query sample and the task-agnostic open-set prototype representation is scaled using one or more learned scaling factors.

4. The processor-implemented method of claim 3 , further comprising: The Euclidean distance metric between the given query sample and each of the one or more prototype representations is scaled using the one or more learned scaling factors.

5. The processor-implemented method of claim 3, wherein: The one or more learned scaling factors are determined as a first scalar value and a second scalar value; and The first scalar value and the second scalar value are learned based on a loss function that implements the task-agnostic open-set prototype representation as a task-agnostic global second-best classification for each of the one or more query samples.

6. The processor-implemented method of claim 5, wherein: The one or more learned scaling factors are task agnostic scaling factors; And the task-agnostic open-set prototype representation is a global second-best classification for multiple few-shot open-sample recognition (FSOSR) episodes performed on the data sample.

7. The processor-implemented method of claim 1 , wherein classifying each query sample based on the one or more distance metrics comprises: determining a probability distribution over the one or more classes and the open set of classes, wherein the probability distribution is determined based at least in part on a Euclidean distance metric determined between each query sample and a corresponding prototype associated with each of the one or more classes and a Euclidean distance metric determined between each query sample and the task-agnostic open set prototype representation; as well as Each query sample is classified into one of the one or more classes or into the open set class based on the probability distribution.

8. The processor-implemented method of claim 7, further comprising: Open set rejection (OSR) is performed based on a set of classified query samples classified into the open set class associated with the task-agnostic open set prototype representation.

9. The processor-implemented method of claim 7, wherein classifying each query sample based on the one or more distance metrics further comprises: determining, based on the probability distribution, a probability that each query sample is included in the open set class associated with the task-agnostic open set prototype representation; as well as The determined probability is compared to a predetermined threshold.

10. The processor-implemented method of claim 9, further comprising: classifying the given query sample as being included in the open set class based on determining that the probability that the given query sample is included in the open set class is greater than the predetermined threshold; and The given query sample is classified as being included in the closed set class based on determining that the probability that the given query sample is included in the open set class is not greater than the predetermined threshold.

11. The processor-implemented method of claim 10 , further comprising: Each query sample classified as being included in the closed set of classes is classified into a corresponding class among the one or more classes, wherein each query sample is classified based on maximizing a corresponding probability determined between each query sample and each corresponding class among the one or more classes.

12. The processor-implemented method of claim 11, wherein the probability is an argmax probability.

13. The processor-implemented method of claim 1 , wherein classifying each query sample comprises: Each query sample is provided to a trained few-shot open-sample recognition (FSOSR) neural network classifier, wherein the trained FSOSR neural network classifier includes at least the task-agnostic open-set prototype representation and one or more distance scaling factors as learnable components.

14. The processor-implemented method of claim 13, wherein the trained FSOSR neural network classifier further comprises one or more feature embedding networks as learnable components.

15. The processor-implemented method of claim 14, wherein determining the one or more distance metrics for each query sample further comprises: determining, using the one or more feature embedding networks, an embedding for each of the one or more query samples; determining the one or more prototype representations as an average embedding of a set of embeddings determined for the plurality of support samples associated with each of the one or more classes; as well as Each of the one or more distance metrics is determined based on determining a Euclidean distance metric between the embedding determined for a given query sample and the embedding determined for each of the one or more prototype representations associated with the one or more classes.

16. An apparatus for processing one or more data samples, comprising: at least one memory; as well as at least one processor coupled to the at least one memory and configured to: determining one or more prototype representations based on a plurality of supporting samples associated with one or more classes of the data sample, wherein each prototype representation is associated with one of the one or more classes; determining a task-agnostic open-set prototype representation, wherein the one or more prototype representations and the task-agnostic open-set prototype representation are determined in the same learning metric space; determining one or more distance metrics for each query sample in one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open set prototype representation; as well as Each query sample is classified based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open set class associated with the task-agnostic open set prototype representation.

17. The apparatus of claim 16, wherein to determine the one or more distance metrics for each query sample, the at least one processor is further configured to: Determining a Euclidean distance metric between a given query sample and each of the one or more prototype representations associated with the one or more classes; and determining a Euclidean distance metric between the given query sample and the task-agnostic open set prototype representation.

18. The apparatus of claim 17, wherein the at least one processor is further configured to: The Euclidean distance metric between the given query sample and the task-agnostic open-set prototype representation is scaled using one or more learned scaling factors.

19. The apparatus of claim 18, wherein the at least one processor is further configured to: The Euclidean distance metric between the given query sample and each of the one or more prototype representations is scaled using the one or more learned scaling factors.

20. The apparatus of claim 19, wherein: The one or more learned scaling factors are determined as a first scalar value and a second scalar value; and The first scalar value and the second scalar value are learned based on a loss function that implements the task-agnostic open-set prototype representation as a task-agnostic global second-best classification for each of the one or more query samples.

21. The apparatus of claim 20, wherein: The one or more learned scaling factors are task-agnostic scaling factors; and the task-agnostic open set prototype representation is a global second-best classification for a plurality of few-shot open sample recognition (FSOSR) episodes performed on the data samples.

22. The apparatus of claim 16, wherein to classify each query sample based on the one or more distance metrics, the at least one processor is configured to: determining a probability distribution over the one or more classes and the open set of classes, wherein the probability distribution is determined based at least in part on a Euclidean distance metric determined between each query sample and a corresponding prototype associated with each of the one or more classes and a Euclidean distance metric determined between each query sample and the task-agnostic open set prototype representation; and Each query sample is classified into one of the one or more classes or into the open set class based on the probability distribution.

23. The apparatus of claim 22, wherein the at least one processor is further configured to: Open set rejection (OSR) is performed based on a set of classified query samples classified into the open set class associated with the task-agnostic open set prototype representation.

24. The apparatus of claim 22, wherein in order to classify each query sample based on the one or more distance metrics, the at least one processor is further configured to: determining, based on the probability distribution, a probability that each query sample is included in the open set class associated with the task-agnostic open set prototype representation; and The determined probability is compared to a predetermined threshold.

25. The apparatus of claim 24, wherein the at least one processor is further configured to: classifying the given query sample as being included in the open set class based on determining that the probability that the given query sample is included in the open set class is greater than the predetermined threshold; and The given query sample is classified as being included in the closed set class based on determining that the probability that the given query sample is included in the open set class is not greater than the predetermined threshold.

26. The apparatus of claim 25, wherein the at least one processor is further configured to: Each query sample classified as being included in the closed set of classes is classified into a corresponding class among the one or more classes, wherein each query sample is classified based on maximizing a corresponding probability determined between each query sample and each corresponding class among the one or more classes.

27. The apparatus of claim 26, wherein the probability is an argmax probability.

28. The apparatus according to claim 16, wherein in order to classify each query sample, the at least one processor is configured to: Each query sample is provided to a trained few-shot open-sample recognition (FSOSR) neural network classifier, wherein the trained FSOSR neural network classifier includes at least the task-agnostic open-set prototype representation and one or more distance scaling factors as learnable components.

29. The apparatus of claim 28, wherein the trained FSOSR neural network classifier further comprises one or more feature embedding networks as learnable components.

30. The apparatus of claim 29, wherein to determine the one or more distance metrics for each query sample, the at least one processor is configured to: determining, using the one or more feature embedding networks, an embedding for each of the one or more query samples; determining the one or more prototype representations as an average embedding of a set of embeddings determined for the plurality of support samples associated with each of the one or more classes; as well as Each of the one or more distance metrics is determined based on determining a Euclidean distance metric between the embedding determined for a given query sample and the embedding determined for each of the one or more prototype representations associated with the one or more classes.

Images