System and method for adapting a base classifier to a novelty class

Abstract

Providing adapting a base classifier to novel classes. A system (100) for adapting a base classifier to one or more novel classes is disclosed. The base classifier classifies instances into base classes by extracting feature representations from the instances using a feature extractor and matching them to class representations of the base classes. The base classifier is adapted using training data for the novel classes. Class representations of the novel classes are determined based on feature representations of instances of the novel classes. The class representations of the novel classes and the base classes are then adapted, wherein at least one class representation of the novel classes is adapted based on class representations of the base classes and at least one class representation of the base classes is adapted based on class representations of the novel classes. The adapted class representations of the base classes and the novel classes are associated with the base classifier.

Description

TECHNICAL FIELD

[0001] The present invention relates to a system for adapting a base classifier to one or more novel classes and a corresponding computer-implemented method. The present invention further relates to a system for learning a set of transformation parameters for adapting a base classifier to one or more novel classes and a corresponding computer-implemented method. The present invention further relates to a computer-readable medium comprising instructions for performing the method and / or the set of transformation parameters. BACKGROUND

[0002] Modern machine learning techniques are capable of classifying images with impressive accuracy, and as such, image classification is increasingly used in real-life applications such as autonomous driving, medical imaging, etc. However, the quality of a machine learning model trained using conventional techniques largely depends on the amount of training data available. For example, a large number of examples per class can be required to reliably classify instances into that class. However, especially in settings such as autonomous driving, there can be a large number of different classes, e.g. different traffic signs, various other types of traffic participants, etc. Moreover, the characteristics of such classes can depend on where the machine learning model is deployed, e.g. traffic signs differ from country to country. Hence, a large amount of training data for many classes and many variants can be required, which makes it expensive and complex to effectively deploy such a machine learning model.

[0003] Few-shot learning techniques aim to provide machine learning models that can adapt to novel classes for which relatively few training examples are available. For example, a known setting for few-shot learning is to a lens class. In this case, the goal is to recognize a previously unseen novel class given a given number of labeled examples per class. This task can be approached from a meta-learning perspective. Unlike in classical transfer learning, the goal of meta-learning is not to adapt to a particular target task, but to adapt to various new tasks presented at test time and perform well across them. For example, in meta-learning, process parameters for determining a to lens classifier can be determined that generally provide the best results over several runs.

[0004] In S. Gidaris and N. Komodakis, "Dynamic Few-Shot Visual Learning without Forgetting," available at https: / / arxiv.org / abs / 1804.09458 and incorporated herein by reference, a few-shot visual learning system is proposed. Based on a large set of training data for a set of base classes, the system aims to provide an object recognition learning system that is able to recognize objects from these base classes as well as from novel classes for which only a few training examples are provided at test time. Object recognition is performed by a convolutional neural network based recognition model that compares a feature vector extracted from an input image with a classification weight vector for each object class. The classification weight vector for a novel class is inferred from the training examples of the novel class and the classification weight vectors of the base classes. The model is trained in two stages. In a first training stage, the parameters of the feature extractor and the classification weight vectors of the base classes are learned. In a second training stage, the parameters of the inference process are learned. SUMMARY

[0005] According to a first aspect of the invention, a system for adapting a base classifier to one or more novel classes is proposed. According to another aspect of the invention, a computer-implemented method of adapting a base classifier is proposed. According to a further aspect of the invention, a system for learning a set of transformation parameters is proposed, wherein the set of transformation parameters is used for adapting a base classifier to one or more novel classes. According to a further aspect of the invention, a computer-implemented method of learning a set of transformation parameters is proposed. According to an aspect of the invention, a computer readable medium is provided.

[0006] The above aspects of the invention involve the classification of instances into respective classes using a classifier. For example, various embodiments involve image classification, wherein the input instances comprise images, e.g., images of traffic situations, or, more generally, images of the environment of a control system. The classification task can be the recognition of objects in the images, e.g., specific traffic signs in the environment of an autonomous vehicle, specific organs in medical images, etc. However, the invention is by no means limited to images only, e.g., the input instances to be classified can comprise various types of sensor data obtained from sensors, e.g., video data, radar data, LiDAR data, ultrasound data, motion data, etc. Classification of such sensor data can be used for controlling various computer-controlled machines, e.g., robots, vehicles, household appliances, power tools, manufacturing machines, personal assistants, access control systems, etc.

[0007] The above aspects of the invention also relate to adaptation of such classifiers to novel classes. For example, a classifier trained on European traffic signs can be adapted to the Chinese market. In various embodiments, a classifier classifies instances into a set of classes based on class representations of these classes. A class representation can represent one or more attributes or characteristics of a class, e.g., obtained by training based on one or more class instances. A class representation is also referred to as a class prototype or a classification weight vector. A classifier can classify an instance by matching the instance to respective prototypes and selecting the class whose class representation most corresponds to the instance. A prototype network is an example of such a classifier.

[0008] To adapt a classifier to novel classes, a class representation of the novel classes can be determined and associated with the base classifier to obtain a joint classifier. Interestingly, such a joint classifier can be used to classify instances into both the base classes and the novel classes. In other words, the goal is to adapt a classifier to quickly incorporate novel classes with few shots while at the same time maintaining discriminative ability on all classes including both previously seen base classes and novel classes. For example, a classifier should accurately recognize both European and Chinese traffic signs even if few Chinese traffic signs are available. The problem of determining such a classifier that can classify both base classes and novel classes can be referred to as "generalized few-shot learning" (GFSL). Note that the GFSL problem of classifying into both base classes and novel classes is more challenging than the traditional FSL problem of classifying into novel classes only, e.g., the label space is much larger as it includes both base classes and novel classes. In particular, a model trained for FSL does not typically provide satisfactory performance in a GFSL setting.

[0009] Note that both the base classifier and the joint classifier can use the same process to classify instances based on class representations. Such a process can be referred to as a prototype classifier. Thus, the base classifier can include a prototype classifier and class representations of the base classes and can classify instances based on a base label space that includes the base class representations. The joint classifier can include the same prototype classifier and class prototypes for both the base classes and the novel classes and can classify instances based on a joint label space that includes both the base and novel class representations. Adapting the base classifier can include adapting the set of prototypes without affecting the prototype classifier, e.g., associating updated class prototypes with the base classifier.

[0010] To adapt the base classifiers to the joint classifier that can additionally classify into one or more novelty classes, typically training data for the novelty classes is obtained in the form of one or more instances of the respective novelty class. Advantageously, in various embodiments, relatively little training data can suffice, e.g. the number of instances per class can be one, two, at most five or at most ten. Based on these instances, a class representation of the one or more novelty classes can be determined using a feature extractor, e.g. by extracting a feature representation of the instances and determining a class representation of the novelty class therefrom for use by the joint classifier.

[0011] While such class representations can be used directly by the joint classifier, the inventors realized that this can not lead to an optimal solution. If the class representations are effectively determined as a function of their support set, e.g. the set of instances of the class, information from other classes can best have an indirect influence, e.g. through the learning parameters of the feature extractor. Moreover, this influence is only in one direction, e.g. base classes can influence novelty classes, but not the other way around. As the inventors realized, this can lead to class representations that are not as globally consistent and discriminative as they could be.

[0012] However, interestingly, the inventors designed to improve the joint classifier not only by determining class representations of the novelty classes, but additionally to adapt each other based on the class representations of the base classes and the class representations of the novelty classes. For example, at least one class representation of a novelty class can be adapted based on a class representation of a base class, and at least one class representation of a base class can be adapted based on a class representation of a novelty class. Moreover, at least one class representation of a novelty class can be adapted based on a class representation of another novelty class, and / or at least one class representation of a base class can be adapted based on a class representation of another base class. As discussed above, by using learned transformation parameters, the adaptation that is most relevant to the particular setup can be used.

[0013] Adapting both to the novel and base classes can help to obtain a globally consistent and / or discriminative set of class representations. For example, including information of similar base classes into the class representation of a novel class can increase its accuracy by effectively evening out the random influence due to the small number of novel class instances. However, doing so alone would not take into account that the class representation of a novel class can also be valuable for updating the class representations of the base classes. For example, if a novel class is relatively similar to a base class, then the optimal representation for the novel class can also be close to several base class representations given a fixed set of base class representations. Thus, the cluster of instances defined by the novel class representation can heavily overlap with the clusters of instances defined by the base class representations. In such a case, updating both the base class representations and the novel class representation, e.g., by moving the base class representations away from the novel class representation, can result in a more efficient distribution of the class representations. In effect, the joint update can introduce a repulsion mechanism by which overlapping base and / or novel classes are pushed apart to reduce confusion. Note that such an effect is difficult to achieve when only placing the novel class in the fixed space of the base classes.

[0014] More generally, updating the base class representations based on the representations of similar novel classes advantageously allows to incorporate information of the instances of the novel classes not only in terms of the general data sampling conditions, but also in terms of the classes themselves. For example, the base class representations can also be based on relatively few elements, so using additional instances that are expected to be broadly similar can help to reduce the random influence in the set of instances that was initially used to determine the base class representations.

[0015] In effect, the above measures can enable to define not only the relationships from the base classes to the novel classes, but also from the novel classes to the base classes and between the base and novel classes. For example, in addition to using the novel classes to update the base class representations and using the base classes to update the novel classes, the novel class representations can also be based on other novel classes and the base class representations can also be based on other base classes. In general, some or all of the class representations can be updated based on respective similar other classes, regardless of whether they are base or novel classes. This can be particularly effective to ensure global consistency of the class representations.

[0016] In various embodiments, a set of transformation parameters is used to perform adapting the base classifier to one or more novel classes. For example, when adapting the class representation of a base or novel class, the set of transformation parameters can indicate how to determine which other class representation to use and / or to what extent each other class representation influences the class representation. The set of transformation parameters can also indicate how to update the class representation, e.g., to what extent to consider which attributes, etc. Various examples are provided throughout. Using transformation parameters is not strictly necessary, but increases flexibility and adaptability of the adaptation.

[0017] In various aspects of the invention, the transformation parameters themselves are learned using a meta-learning approach, e.g., by repeatedly adapting the base classifier to determine which set of transformation parameters generally leads to the best joint classifier. Specifically, given training data for a plurality of classes, the transformation parameters can be learned by repeatedly selecting one or more base classes and one or more novel classes from the plurality of classes, constructing a base classifier for the base classes, selecting training data for the novel classes, and adapting the base classifier to the novel classes using a set of transformation parameters to obtain a joint classifier. The performance of the set of parameters can be measured in terms of the classification loss of the joint classifier when applied to instances of the selected base and novel classes. For example, the set of transformation parameters can be updated, e.g., using stochastic gradient descent or a similar optimization technique, to reduce the classification loss.

[0018] In various embodiments, the classifier classifies the instance based on its feature representation. For example, the feature representation can be matched against class representations of respective classes, e.g., to classify the instance into the class whose class representation best matches the feature representation of the instance. The classifier does not necessarily return a single class into which the instance is classified, e.g., the classifier can also be used to estimate the probability that the instance belongs to a given class, to determine the top-k classes into which the instance is most likely to be classified, etc.

[0019] The feature representation of an instance can be extracted from the instance by means of a feature extractor. Typically, the feature representation consists of fewer elements than the instance itself, e.g., the instance can be represented by a vector and the feature representation can be represented by a smaller vector. In this sense, the feature representation can provide a compressed representation of the instance. For example, the feature extractor can comprise a convolutional neural network (CNN) or other type of machine learning model. Interestingly, the feature extractor can be parameterized by a set of parameters. The set of parameters, e.g., the weights of a CNN or other model parameters, can be trained in order to obtain a feature representation as a compressed representation of an input instance that captures the information about the instance that is most relevant for the classification task at hand. When in use, e.g., when adapting a base classifier to a novel class, the parameters of the feature extractor are typically fixed, e.g., adapting the base classifier comprises updating the class representations used by the prototype classifier but not the parameters used by the feature extractor.

[0020] Optionally, when matching the feature representation of an instance against a respective class prototype, the cosine distance between the feature representation and the class prototype is computed. Although other distance measures such as the L2 distance are also possible, the inventors have found that the cosine distance is preferable as it works well both for the class representations of the base classes and for the class representations of the novel classes. The cosine distance can have a learnable temperature parameter.

[0021] ​Optionally, the processor subsystem is configured to determine an adapted class representation of a certain target class by computing a weighted sum of class representations of one or more base classes and one or more exotic classes, e.g. a weighted sum of the elements of the vectors, optionally in combination with other update operators, e.g. post-convolutional transformations and non-linear operators as discussed below. By using a weighted sum, the class representation can take into account information of other class representations depending on their relevance to the representation of the target class. Such a weighted sum of class representations can be seen as a convolution, and in that sense, the weighted sum can be referred to as a "convolution operator".

[0022] Optionally, the weights of the respective class representations in the weighted sum are indicative of a similarity between the respective class representation and the class representation of the target class. The weights can be determined from the respective class representation and the class representation of the target class. In other words, the weights can be computed at least partly based on the class representations themselves. Cosine similarity proves to work particularly well, but a range of other similarity measures are possible as well.

[0023] Optionally, the weights can be determined at least partly based on one of more side information values associated with the class representation. The one or more side information values of an exotic class can be independent of the one or more instances of the exotic class. In other words, the side information can be information related to the class itself rather than to a specific training instance, e.g. a semantic embedding such as text class information, or a class-specific attribute. Such side information can be obtained, e.g. from various external sources such as a Wikipedia page related to the class, a gathered medical dataset, etc. The side information values can be adapted along with their respective class representations, e.g. according to the same weighted sum that is used to adapt the class representations, while adapting the class representations. However, they are typically not part of the class representations of the joint classifier.

[0024] Including side information can particularly help adapting the class representations, as it allows to update a class representation based on other class representations that can not look similar based on the available training instances, but are still relevant as indicated by the side information. This can result in incorporating information into the class representation that would not be incorporated if the weights were based on the class representations and / or the underlying training instances only.

[0025] Note that the side information is typically not used when applying the joint classifier to a non-training instance. In fact, the side information is information about the class rather than about a specific instance, so it is typically not available for an instance that is still to be classified. Therefore, the joint classifier does not need to include the side information. However, the side information can still be included in the joint classifier, e.g. to further adapt the joint classifier to even more exotic classes.

[0026] Optionally, the similarity between the respective class representations and / or side information can be determined using a parametrized similarity measure, various examples of which are provided. Interestingly, the parameters of the similarity measure can be included in the set of transformation parameters, such that the similarity measure is optimally adapted to the task at hand, e.g., such that the similarity measure selects the other class representations that are most relevant for updating the target class representation. Interestingly, even if the number of parameters of the similarity measure is relatively large, e.g., even if the similarity measure is defined by a neural network, overfitting can still be avoided, since the same similarity measure can be applied to many pairs of class representations. In fact, an overall reduction in the number of parameters, and thus a more robust adaptation process, can be obtained compared to, e.g., determining a representation for each base class in the key space.

[0027] Optionally, the weights of the respective class representations in the weighted sum are indicative of the similarity between the class of the respective class representation and the target class independent of their class representations. Such weights can be accessed separately from the training data set, or at least not determined from instances of the class representations and / or classes. In this sense, such weights can be seen as relational side information about the various classes. This relational side information can be obtained from various external knowledge sources, e.g., as WordNet similarities or the like. As mentioned above, when applying a joint classifier, typically no relational side information is used, and thus it typically does not need to be stored together with or included in the joint classifier, unless the joint classifier needs to be further adapted. Since the relational side information allows for an update of the class representations in a way that is not suggested by the class representations themselves or the instances on which these class representations are based, a more relevant adaptation of the class representations can be achieved.

[0028] Optionally, normalization can be applied to the class representations and / or the computed weighted sum. For example, a softmax operator or the like can be applied to the class representations. The inventors realized that normalization greatly improves the accuracy of the obtained joint classifier, since it can prevent the updated class representation vector from becoming longer than the initial class representation, thereby ensuring that the updated class representation remains in the original space. When using relational side information, as discussed above, the normalization can include applying a softmax operator with a learnable temperature. This can allow the toe model to apply more effective contrast to the operator matrix, thereby effectively making the similarity structure more peaked or more flat.

[0029] Optionally, the adapted class representation of the target class can be determined by calculating a plurality of weighted sums using respective weights. The plurality of weighted sums can be combined into a single adapted class representation, e.g. as a sum or average. For example, some weighted sums can be based on similarities between the class representation and / or side information (e.g. calculated with different similarity measures), other weighted sums can be based on relational side information, etc. Thus, different ways of updating the class representation can be combined. The different ways can be parameterized by different respective parameters. By combining the weighted sums, information from other class representations can be particularly effectively used for updating the target class representation.

[0030] Optionally, the features of the class representation can be further adapted by adapting the features of the class representation by calculating a weighted sum of features and / or side information values of the class representation. In other words, the features of a class representation can be adapted based on other features and / or side information values of the same class representation. The weights can e.g. be included in the set of transformation parameters. The class representation whose feature values are adapted can be a weighted sum of the class representations to be adapted based on other class representations. In this sense, the adaptation can be seen as a "post-convolutional transformation". If a plurality of weighted sums is calculated, the plurality of weighted sums can be adapted separately, e.g. each weighted sum based on a respective weight in the set of transformation parameters. However, it is also possible to adapt the class representation not after, but before taking the weighted sum. In various embodiments, in fact, both possibilities can end up with the same result by correlation, e.g. if a linear transformation is used.

[0031] In any case, conceptually, adapting the features of a class representation can be seen as applying a fully connected layer to e.g. the class representation to be updated. Similar to their use in neural networks, the flexible transformation of the class representation can be captured, in particular in combination with the iterative process discussed below. In particular, such a fully connected layer can particularly effectively incorporate side information into the joint classifier; as noted, otherwise the side information values would typically not be included in the joint classifier. By performing the adaptation using a multiplication with a diagonal matrix, a reduction of the number of parameters, and thus a reduction of the risk of overfitting, can be achieved.

[0032] Optionally, the adapted class representations can be determined in an iterative process. That is, multiple iterations can be performed in which the class representations of the base classes and the novelty classes are updated. This allows to spread the class representation information through multiple hops. For example, a first class representation can be used to update a second class representation, e.g. the first and second classes can be similar. In the next iteration, the second class representation can be used to update a third class representation, e.g. the second and third classes can be similar. Thus, even without a direct cause for updating the third class representation based on the first class representation, information from the first class representation can be used to indirectly influence the third class representation, allowing for a better spreading of the knowledge captured by the class representations. As another example, in case multiple weighted sums are computed using respective weights, e.g. one based on relational side information and one based on class representation similarity, then updates between classes that are similar in relation in the first iteration can influence updates between classes that are similar in representation in the second iteration, and vice versa. Thus, the side information can be spread in a controlled manner across multiple types of class similarity.

[0033] When using the iterative process in combination with the weighted sum of class representations, optionally, a non-linearity can be applied to the computed weighted sum or to the average or sum of multiple such weighted sums to obtain the updated class representation. Similar to their use in neural networks, the non-linearity allows to capture more flexible transformations. For example, the non-linearity can be a ReLU, a leaky ReLU, a tanh function, etc. However, such a non-linearity is typically not applied in the last iteration in order not to limit the final value of the class representation.

[0034] Optionally, a joint classifier is used to classify the query instance. In order to classify the obtained query instance, a joint classifier can be used to determine a classification score of the query instance with respect to each base class and each novelty class. Using the classification score, a classification of the query instance into a base class or a novelty class can be determined. This way, the query instance can be classified into novelty classes and base classes in a way that effectively utilizes the training data for the novelty classes, e.g. not only determining the class representations of those novelty classes themselves but also influencing the class representations of at least one base class. Thus, a more accurate classification of the query instance is obtained.

[0035] With respect to learning the transformation parameters from the training data, optionally, such learning can be combined with learning class representations for the training data and / or the parameters. Learning class representations of the classes of a training dataset - as opposed to deriving them from the instances - is advantageous as it allows to combine a fast learning of novelty classes with a slower learning of the classes of the training dataset. For example, the training can reward global discrimination, e.g. selecting class representations that do not necessarily most accurately represent the training instances of their class but lead to better decision boundaries with respect to other classes, e.g. because in the training class representations can influence each other.

[0036] Optionally, the class representations of the plurality of classes and the parameters of the feature extractor are determined by learning the class representations and the parameters in a learning operation prior to learning the set of transformation parameters. Performing separate learning of the class representations and the parameters can be advantageous, for example, to reduce the number of parameters to be learned simultaneously.

[0037] Optionally, the class representations of the plurality of classes and / or the parameters of the feature extractor are learned together with the transformation parameters based on the determined classification loss of the adapted base classifier. For example, the set of transformation parameters, the class representations and / or the parameters of the feature extractor can be updated together with the transformation parameters to reduce the classification loss, for example, in a stochastic gradient descent iteration or the like. For example, the class representations of those classes that are selected as base classes in a particular run can be updated to reduce the loss. Thus, in effect, the class representations and the various parameters can be learned end-to-end. Such combined learning is more flexible because it allows the class representations and the feature extractor parameters to be updated based on feedback about their suitability and / or based on updates to the transformation parameters. For example, the feature extractor can learn to extract features that are useful for computing similarities between the computed class representations. If the transformation parameters are learned separately from the feature extractor parameters, such features would generally not be learned because this use only arises when adapting the classifier. The inventors have found that combined learning is particularly beneficial when using side information to update the class representations because it can avoid the model from getting stuck in a representation that is not consistent with the side information before the transformation parameters are determined and thus help ensure that the side information can benefit when updating the class representations. For example, the inventors have found that if a model is optimized for using particular features, it can continue to use such features even if additional side information becomes available because the use of such features can initially provide better improvement to the model.

[0038] The person skilled in the art will appreciate that two or more of the above-mentioned embodiments, implementations and / or optional aspects of the application can be combined in any manner deemed useful.

[0039] Any modifications and variants of the computer-implemented methods and / or of any computer-readable medium can be carried out by the person skilled in the art on the basis of the description, which correspond to the described modifications and variants of the corresponding system. BRIEF DESCRIPTION OF DRAWINGS

[0040] These and other aspects of the application will be apparent from and elucidated with reference to the embodiments described hereinafter, by way of example, and with reference to the accompanying drawings, in which:

[0041] Figure 1 A system for adapting a base classifier to one or more exotic classes to obtain a joint classifier is shown;

[0042] Figure 2 A system for learning a set of transformation parameters for adapting a base classifier to one or more novel classes is shown;

[0043] Figure 3 A detailed example of how a classifier with class representations can be used to classify instances is shown;

[0044] Figure 4 A detailed example of how a base classifier can be adapted to one or more novel classes to obtain a joint classifier is shown;

[0045] Figure 5 A detailed example of how a set of transformation parameters for adapting a base classifier to one or more novel classes can be learned is shown;

[0046] Figure 6 A detailed example of how a set of transformation parameters for adapting a base classifier to one or more novel classes can be learned is shown;

[0047] Figure 7 A computer-implemented method of adapting a base classifier to one or more novel classes is shown;

[0048] Figure 8 A computer-implemented method of learning a set of transformation parameters is shown;

[0049] Figure 9 A computer-readable medium comprising data is shown.

[0050] It should be noted that the various figures are purely illustrative and not drawn to scale. In the various figures, elements corresponding to elements already described can have the same reference number. DETAILED DESCRIPTION

[0051] Figure 1 A system 100 for adapting a base classifier to one or more novel classes is shown. The system 100 can include a data interface 120 and a processor subsystem 140, which can internally communicate via data communication 124. The data interface 120 can be used to access data 050 representing a base classifier. The data 050 can include class representations of one or more base classes of the base classifier. The data 050 can also include parameters of a feature extractor used to extract feature representations from instances. The base classifier can be configured to classify instances into the one or more base classes by matching feature representations extracted from instances by the feature extractor to the class representations of the one or more base classes.

[0052] The processor subsystem 140 can be configured to access the data 050 representing a base classifier during operation of the system 100 and using the data interface 120. For example, as inFigure 1 As shown in FIG. 1, data interface 120 can provide access 122 to external data storage 020, which can include the data 050. Alternatively, data 050 can be accessed from an internal data storage that is part of system 100. Alternatively, data 050 can be received from another entity via a network. In general, data interface 120 can take various forms, such as a network interface to a local or wide area network (e.g., the Internet), a storage interface to an internal or external data storage, etc. Data storage 020 can take any known and suitable form.

[0053] Processor subsystem 140 can be configured to adapt the base classifier during operation of system 100 and using data interface 120. Adapting the base classifier during system operation can include obtaining training data 035 for one or more novel classes. The training data for the novel classes includes one or more instances of the novel classes. For example, as shown in the image, data interface 120 can be used to access training data 035. Adapting the base classifier during system operation can further include determining class representations for the one or more novel classes. Determining the class representations for the novel classes can include extracting feature representations of the one or more instances of the novel classes using a feature extractor.

[0054] Adapting the base classifier during system operation can further include determining adapted class representations for the base classes and the novel classes, including adapting at least one class representation of the novel classes based on the class representations of the base classes, and adapting at least one class representation of the base classes based on the class representations of the novel classes. Adapting the base classifier during system operation can also include associating the adapted class representations with the base classifier to obtain a joint classifier for classifying instances into the one or more base classes and the one or more novel classes.

[0055] Optionally, processor subsystem 140 can be configured to, during operation of system 100, obtain a query instance (not shown); determine, using the joint classifier, classification scores of the query instance with respect to each base class and each novel class; and determine, from among the classification scores, a classification of the query instance into a base class or a novel class.

[0056] Optionally, the adapted class representations for the base classes and the novel classes can be determined using a set of transformation parameters (not shown). For example, data interface 120 can be used to access the set of transformation parameters. The set of transformation parameters can be obtained by learning the set of transformation parameters according to the methods described herein (e.g., by system 200 of FIG. 1). Figure 2 Optionally, the adapted class representations for the base classes and the novel classes can be determined using a set of transformation parameters (not shown). For example, data interface 120 can be used to access the set of transformation parameters. The set of transformation parameters can be obtained by learning the set of transformation parameters according to the methods described herein (e.g., by system 200 of FIG. 1).

[0057] As an optional component, the system 100 can comprise an image input interface 160 or any other type of input interface for obtaining sensor data from a sensor such as a camera 180. The sensor data can be part of instances of the novelty class to which the base classifier is to be adapted and / or in the query instance to be classified. For example, the camera can be configured to capture image data 162, the processor subsystem 140 being configured to obtain instances from the image data 162 obtained via the input interface 160.

[0058] As an optional component, the system 100 can comprise an actuator interface (not shown) for providing actuator data to an actuator, the actuator data causing the actuator to implement an action in the environment of the system 100. For example, the processor subsystem 140 can be configured to determine actuator data based at least in part on the classification score of the query instance determined using the joint classifier, and provide the actuator data to the actuator via the actuator interface.

[0059] Various details and aspects of the operation of the system 100, including optional aspects thereof, will be further elucidated with reference to the following description and accompanying drawings, in which Figures 3-6

[0060] Generally, the system 100 can be embodied as or in a single device or apparatus, such as a workstation (e.g., a laptop- or desktop-based workstation) or a server. The device or apparatus can comprise one or more microprocessors executing appropriate software. For example, the processor subsystem can be embodied by a single central processing unit (CPU), but also by a combination or system of such CPUs and / or other types of processing units. The software can have been downloaded and / or stored in a corresponding memory, e.g., a volatile memory such as a RAM, or a non-volatile memory such as a flash memory. Alternatively, functional units of the system, e.g., the data interface and the processor subsystem, can be implemented in the device or apparatus in the form of programmable logic, e.g., as a field-programmable gate array (FPGA) and / or a graphics processing unit (GPU). Generally, each functional unit of the system can be implemented in the form of circuitry. Note that the system 100 can also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as distributed servers, e.g., in the form of cloud computing.

[0061] ​As a specific example, the system 100 can be an automotive control system for controlling a vehicle. For example, the base classifier and the joint classifier can be image classifiers, and the instances can be images. The vehicle can be an autonomous or semi-autonomous vehicle, but the system 100 can also be a driver assistance system for a non-autonomous vehicle. For example, such a vehicle can incorporate the system 100 to control the vehicle based on images obtained from a camera 180, as discussed. In this particular example, the system 100 can be configured to use the joint classifier to determine a classification of a query image obtained from the camera 180 to detect an object of interest in an environment of the vehicle, e.g., a traffic sign. The system 100 in this example can further include an actuator interface (not shown separately) for providing actuator data to an actuator, the actuator data causing the actuator to implement an action for controlling the vehicle. The system 100 can be configured to determine the actuator data for controlling the vehicle based at least in part on the classification; and provide the actuator data to the actuator via the actuator interface. For example, the actuator can be caused to control steering and / or braking of the vehicle. For example, the control system can control an electric motor of the vehicle to perform (regenerative) braking based on a detected condition, e.g., a speed limit traffic sign.

[0062] Figure 2 A system 200 for learning a set of transformation parameters is shown. The set of transformation parameters can be used, e.g., according to the methods described herein, e.g., by Figure 1 The system 100 adapts the base classifier to one or more novel classes. The system 200 can include a data interface 220 and a processor subsystem 240, which can internally communicate via data communication 224. The data interface 220 can be used to access training data 030 for a plurality of classes. The training data for a class can include one or more instances of the class. The data interface 220 can be used to further access data 040, which represents class representations of the plurality of classes and parameters of a feature extractor for extracting feature representations from instances.

[0063] The processor subsystem 240 can be configured to access the data 030, 040 during operation of the system 200 and using the data interface 220. For example, as in Figure 2As shown in FIG. 2, the data interface 220 can provide access 222 to external data storage 022, which can include the data 030, 040. Alternatively, the data 030, 040 can be accessed from internal data storage that is part of the system 200. Alternatively, the data 030, 040 can be received from another entity via a network. In general, the data interface 220 can take various forms, such as a network interface to a local or wide area network (e.g., the Internet), a storage interface to internal or external data storage, etc. The data storage 022 can take any known and suitable form.

[0064] The processor subsystem 240 can be configured to learn the set of transformation parameters during system operation. To learn the set of transformation parameters, during operation, the processor subsystem 240 can perform a repeated process. The repeated process can include selecting one or more base classes and one or more novel classes from the plurality of classes. The repeated process can further include constructing a base classifier configured to classify instances into the one or more base classes based on class representations of the one or more base classes and parameters of the feature extractor. For example, as shown in the image, the data interface 220 can be used to access the base classifier. The repeated process can further include selecting training data for the one or more novel classes from the training data. The repeated process can also include adapting the base classifier to the one or more novel classes using the set of transformation parameters according to the methods described herein. For example, the base classifier can be adapted as described for the processor subsystem 140 of the system 100. The repeated process can also include determining a classification loss of the adapted base classifier for instances of the one or more base classes and the one or more novel classes.

[0065] As an optional component, the system 200 can include an image input interface (not shown) or any other type of input interface for obtaining sensor data from a sensor such as a camera (not shown). The processor subsystem 240 can be configured to obtain one or more instances of the training data 030 as sensor data obtained from the sensor.

[0066] Various details and aspects of the operation of the system 200, including optional aspects thereof, will be further clarified with reference to Figures 3-6

[0067] ​Generally, system 200 can be embodied as a single device or apparatus, such as a workstation (e.g., a laptop- or desktop-based workstation) or a server. The device or apparatus may include one or more microprocessors executing appropriate software. For example, a processor subsystem may be embodied by a single central processing unit (CPU), and also by a combination or system of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory like RAM, or non-volatile memory like flash memory. Alternatively, the functional units of the system, such as data interfaces and processor subsystems, may be implemented in the device or apparatus in the form of programmable logic (e.g., as field-programmable gate arrays (FPGAs) and / or graphics processing units (GPUs)). Generally, each functional unit of the system may be implemented in the form of circuits. Note that system 200 can also be implemented in a distributed manner, such as involving different devices or apparatuses, such as distributed servers in the form of cloud computing.

[0068] Generally, various embodiments involve adapting a base classifier used to classify instances into one or more base classes to obtain a joint classifier for classifying instances into one or more base classes and one or more novel classes. The problem of obtaining such a joint classifier can be called "Generalized Few-Shot Learning" or GFSL. GFSL can be considered a generalization of the "Few-Shot Learning" or FSL problem. FSL and GFSL will now be discussed from a mathematical perspective.

[0069] FSL missions and especially Towards Lens classification can be described as: within the so-called support set Each of the given classes has a relatively small number of... In the case of a labeled example, identify A novel type. In other words, FSL tasks can be performed by... It consists of several previously unseen novel classes, each novel class having An example of adding a label, such as More precisely, let The novelty category tag space, in which And make

[0070]

[0071] The so-called label Support set ,in Is class The A tagged example. For a new query image. FSL prediction can be represented as:

[0072] .

[0073] Note that traditional FSL only focuses on identifying novelty classes; for example, a query can be selected from one of the novelty classes. Therefore, for FSL, the above... Only for In other words, in FSL, the classes previously seen from the training set no longer play a role in measuring the model's performance. This setting may emphasize rapid adaptation to different new tasks, but on the other hand, it may not consider the model's ability to accumulate knowledge.

[0074] Interestingly, however, joint classifiers can also classify instances into the base classes of the base classifiers. In other words, joint classifiers can incorporate novel classes into the seen class label space while maintaining global discriminative power across all classes. This leads to joint classifiers being far more useful in practice. More precisely, in GFSL, joint classifiers can learn to discriminate not only novel label spaces... Furthermore, it can learn to distinguish between the joint label space of both novel classes and previously seen classes. For example, a joint classifier can predict the query instance. Class tags This query example It can come from any seen class and novel class. GFSL is generally more challenging than FSL because the joint classifier may also need to be able to be classified into the basic classes. Mathematically, GFSL can be formulated as a generalized... Towards The shot classification problem involves a model that must identify a joint label space consisting of all seen classes and novel classes from the training set, for example... For example, the training data on which the base classifier has already been trained can be labeled as:

[0075]

[0076] in It is the number of classes in the training data used for the base classifier, and Is for the first The number of labeled examples available for each training class. Typically, the number of base classes is greater than the number of novel classes, for example, at least two to five times the number of novel classes. Furthermore, the number of instances of base classes is typically greater than the number of instances of novel classes, for example, at least five to at least ten times the number of novel class instances. For query instances... A GFSL task can be described as deterministic:

[0077] .

[0078] Note that, unlike for FSL, Now consider Because may come from any base class and novelty class.

[0079] Figure 3 A detailed but non-limiting example is shown of how instances can be classified by a classifier, e.g. a base classifier or a joint classifier obtained by adapting a base classifier to one or more novelty classes. For example, a base classifier can be determined according to a method for learning a set of transformation parameters as described herein, e.g. as discussed with respect to Figures 5-6 A joint classifier can be determined according to a method for adapting a base classifier, e.g. as discussed with respect to Figure 4

[0080] Shown in the figure is a classifier C 350. The classifier C can comprise class representations CRi 351 until CRn 352 of one or more classes to which instances can be classified. For example, if the classifier C is a base classifier, the class representations can be class representations of base classes, while if the classifier C is a joint classifier, the class representations can be class representations of base classes and novelty classes. The class representations CRi are typically vectors. For example, a class representation can comprise at most or at least 64, at most or at least 128, or at most or at least 256 features. The class representations CRi can themselves be based on instances, e.g. via a feature extractor Fx; in the following explanations, it is assumed that they are given.

[0081] ​The classifier C can also comprise parameters FPAR of a feature extractor Fx 310 for extracting a feature representation from an instance. The feature extractor Fx is typically a machine learning model with trainable parameters FPAR, e.g. the feature extractor Fx can be a neural network with neural network weights comprised in the parameters FPAR, or any other type of model, e.g. a linear regression model, etc. As a specific example, the feature extractor Fx can comprise a neural network, e.g. a convolutional neural network. Feature extractors are known, e.g. from J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning” (available at https: / / arxiv.org / abs / 1703.05175 and incorporated herein by reference to the extent that it concerns the feature extractor) and S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting” (available at https: / / arxiv.org / abs / 1804.09458 and incorporated herein by reference to the extent that it concerns the feature extractor).

[0082] Also shown is an instance I 339 to be classified. The instance I can be represented as a vector, a matrix or any other data that can be input to the feature extractor Fx. For example, the instance can represent an image or other type of sensor data. In order to classify the instance I using the classifier C, a feature representation FR 329 can be extracted from the instance I by the feature extractor Fx. In a classification CI 370, the feature representation FR can be matched against the class representations CR1,..., CRn. For example, the instance I can be classified into the class whose class representation best matches the feature representation FR, e.g. a classification score for the instance I can be determined with respect to each class and the classification C 690 can be determined as the class with the highest classification score.

[0083] As a specific example, the classification score can be a conditional class probability I of the instance belonging to a particular class , which is computed from the cosine similarity between the feature representation of the instance and the class representation of the respective class CRi. The inventors have found that the cosine similarity is particularly effective, especially for joint classifiers. The cosine similarity can also optionally comprise a learnable temperature factor . For example, such a conditional class probability can be computed as:

[0084] .

[0085] As another example, the class conditional probability of an instance may be computed based on the distance (e.g. L2 distance) of the feature representation of the instance to the class representation of the class, e.g.:

[0086] .

[0087] The skilled person will envisage several variants of the above.

[0088] Figure 4 A detailed, but non-limiting example is shown which illustrates how a base classifier can be adapted to one or more novel classes to obtain a joint classifier.

[0089] In more detail, shown in the figure is a base classifier BC 450. The base classifier BC can be configured to classify instances into one or more base classes. The base classifier BC can comprise class representations CR1 451 until CRm 452 of the one or more base classes. The base classifier BC can further comprise parameters FPAR 455 of a feature extractor Fx 410 for extracting feature representations from instances. For example, the base classifier BC can be a classifier as discussed with respect to Figure 3 .

[0090] Also shown in the figure is training data TD 435 for one or more novel classes. In this example, the training data TD is for a single novel class, but in general any number of novel classes can be used, e.g. at most or at least two, at most or at least ten, etc. The training data for the novel class can comprise one or more instances of the novel class, e.g. instances II 431 until In 432 as shown in the figure. The number of training instances per novel class is typically small, e.g. one, two, at most five or ten, etc.

[0091] To adapt the base classifier BC to the one or more novel classes, first, a class representation of the one or more novel classes can be determined. In this example, a single class representation CR 453 of the novel class only of the training data TD is determined, but in general a class representation can be similarly determined for each novel class. The class representation CR can be determined based on the feature representations FI 421 until Fn 422 extracted from the respective training instances II,..., In using the feature extractor Fx. In a representation extractor Rx 440, the class representation CR can be obtained by combining the feature representations, e.g. as an average:

[0092]

[0093] where is the i-th instance of the class from the training data TD, and is the i-th instance of the class from the training data TD, and Indicates the normalized vector.

[0094] As shown in the diagram, the class representations CR1, ..., CRm of the basic classes and the class representation CR of the novel classes can together form a group of size ... matrix ,in It is the number of basic classes. It is the number of novel categories, and It is the dimension of the class representation.

[0095] Although the class representations CR of the novelty classes obtained so far can be associated with the base classifier BC to obtain a joint classifier for the base and novelty classes, it is interesting that in this example, the class representations of the base and novelty classes are first adapted in the adaptation operation Cad 470. As illustrated by the bidirectional arrows between the class representations CR, CRi, and the adaptation Cad, the adaptation can be bidirectional—in the sense that at least one class representation CR of the novelty class can be adapted based on the class representation CRi of the base class and at least one class representation CRj of the base class can be adapted based on the class representation of the (potentially different) novelty class CR. As explained earlier, performing bidirectional updates allows information to propagate between different classes, resulting in a better joint classifier. In particular, the joint classifier JC can be obtained by associating the adapted class representations with the base classifier BC; for example, the joint classifier JC 459 may include the feature extractor parameters FPAR, the updated base class representation CRi, and the updated novelty class representation CR. In this way, classifiers for the base and novelty classes are obtained (e.g., as per [reference to...]). Figure 3 (As discussed).

[0096] Interestingly, adapting the class representation can be performed based on a set of transformation parameters TPAR 460. For example, the transformation parameters can be learned to optimize how the class representation is adapted, thus resulting in higher quality class representations, such as those described above. Figure 5 and Figure 6 The following section discusses various examples of parameter transformations in more detail.

[0097] Now consider in more detail how the class representation of the target class (e.g., the base class representation CRi or the novel class representation CR) can be adapted. At least two possibilities are available. That is, the class representation can be adapted based on other class representations, and / or the features of the class representation can be adapted based on other features (or lateral information, as discussed later) of the class representation itself. In both cases, using linear transformations is a good design choice: to compute the adapted class representation, a weighted sum of the class representations can be computed, and / or to compute the adapted features, a weighted sum of the features and / or lateral information can be computed.

[0098] Mathematically, in the case of linear transformations, both types of updates can be expressed by the expression Capture, in It is a matrix in which row representation is used to represent the class; It is a convolution operator that transforms class representations based on other class representations; and It is a post-convolutional transformation that transforms features based on other features. It can contain relational information between classes. Operator, where It is the number of basic classes, and It is the number of novel categories. It can be viewed as a post-convolution transformation, although due to In this case, whether the convolution operator is applied first or later... And then apply post-convolution transformation. It doesn't matter whether it's the other way around or not.

[0099] For multiple operators Determine multiple weighted sums (e.g.) It is possible. As discussed later, there exists a way to determine the matrix weights. There are several ways to perform knowledge transfer between class transformations, each allowing knowledge transfer in a specific manner. Therefore, combining these multiple weighted sums allows for the combination of different knowledge transfer methods. In other words, relational information from different sources can be combined. For example, for each operator... The same or separate post-convolution transformations can be applied. To obtain the updated class representation This class represents They can be combined into a single adapted class representation.

[0100] As a concrete example, the updated class representation matrix can be determined by adjusting the CAD algorithm using the following formula. :

[0101]

[0102] in It is the corresponding convolution operator, which contains inter-class relational information. It is used for Post-convolution transformation, It is a scalar that can be learned from the transformation parameter TPAR to weigh the effects from different operators, and It is an optional nonlinearity, such as ReLU, leakyReLU, tanh, etc.

[0103] As another example, when computing the updated class representation matrix At times, intermediate normalization can be applied, e.g.,

[0104]

[0105] wherein denotes the normalized vector. This allows, for example, to exploit the cosine distance between the query and the prototypes in the classification by the joint classifier.

[0106] In various embodiments, in order to adjust the class representations, the Cad applies an iterative process. That is, for example, in two or more iterations, the class representations can be updated based on their initial values, updated again based on their updated values, and so on. For example, the above formula can be applied multiple times. Typically, the non-linear is applied in earlier iterations of the iterative process, but not in the final layer. In order to not restrict the values of the updated class representations, e.g., to ensure that negative entries can occur in the updated class representations. Depending on the type of the convolution operator, the matrix may be re-computed or kept constant in each iteration.

[0107] Now several possibilities to determine the weights (e.g., the entries of the convolution operator ) of the class representations are discussed. In essence, the weights encode relational information between the classes and / or their representations. The weights may denote the relevance of the th class representation for updating the th class representation.

[0108] In various embodiments, the weights indicate a similarity between the th class representation of the target class and the th class representation. In other words, the similarity can be computed based on the class representations themselves, e.g., wherein may be any standard similarity or distance measure, e.g., cosine distance, dot product, L2 distance, Mahalanobis distance, etc. In various embodiments, the similarity measure is a parametric similarity measure, whose parameters are included in the set of transformation parameters TPAR. For example, the similarity can be computed by first transforming the class representations into a transformation space, and then computing the similarity in the transformation space, e.g., wherein the transformation is given in the transformation parameters TPAR, e.g., as a linear transformation, a multi-layer perceptron, etc. As another example, the similarity can be determined by applying a neural network (e.g., a CNN or a multi-layer perceptron) to the element-wise absolute difference, e.g., where the neural network is given by the transformation parameters TPAR. Having computed the similarities, they can optionally be post-processed, e.g. by thresholding, selecting the top most similar entries, L2 / softmax normalization in the line aspect, etc. In various cases, the use of learned transformation parameters can allow to determine the similarity in a way that is most effective for the problem at hand.

[0109] Instead of or in addition to using class representations, the weights can also be based at least partly on one or more side information values associated with the class representation CRi(not shown). For example, the side information values can be included in a vector, the similarity of which is computed as discussed above. The side information values are typically independent of the instances of the class, e.g. they represent information about the class that is obtained separately from the class instances. Relevant side information values depend on the application, but can be obtained from, e.g., an ontology such as WordNet, a knowledge graph, annotated properties, semantic embeddings of class labels (e.g. using word2vec) or other semantic embeddings (e.g. embedding Wikipedia pages or other text about the class, or any other class-specific properties, e.g. typical dimensions of objects, etc.). For example, the weights can indicate the number of hops between two classes in an ontology. The weights can also be provided by expert annotations, for example.

[0110] Since the side information values do not specifically refer to any instance, they are typically also not included in the base classifier and / or the joint classifier, e.g. they are not used for the classification of instances. However, their availability can help to select which class representations to relate to the update target class representation when adapting the base classifier. It is possible that side information values are only available for a subset of the classes; in this case, side information values that are not available for a class can be left empty (e.g. zero) in order to ignore such classes for a particular convolution operator.

[0111] As another example of side information, relational side information can also be used, which indicates the similarity between the target class and a class that is independent of the class representation itself. Such relational side information can be used directly, or possibly after normalization, as a weight for the convolution operator. The relational side information can directly indicate the similarity for the update class The relevance of the representation of the correlation, e.g. the relational side information, can be obtained directly without the need to determine it by computing a similarity. As an example, the relational side information comprises the shortest path distance between two classes in WordNet, or any other side information that expresses the similarity of pairs. The relational side information can also be converted into class aspect side information, e.g. by using the normalized distance to other classes as a value. Moreover, the relational side information can remain empty if it is not available for a particular pair of classes.

[0112] Moreover, the weights for adapting the features of the class representations based on other features (e.g. in a post-convolutional transformation In embodiments, is a learnable quadratic weight matrix, which is parameterized by a transformation parameter TPAR, for example. In embodiments, is constrained to be diagonal, thereby reducing the number of transformation parameters TPAR to reduce the risk of overfitting.

[0113] Two detailed examples of how to learn a set of transformation parameters for adapting a base classifier to one or more exotic classes are now discussed with respect to Figure 5 and Figure 6 Generally, Figure 5 and Figure 6 the process for learning a set of transformation parameters can be seen as a meta-learning approach for the GFSL problem. While GFSL can aim at generalizing a base classifier to a specific set of unseen classes, this set of transformation parameters can be determined with the goal to adapt quickly and well across different tasks involving previously unseen classes. In Figure 5 and Figure 6 this meta-learning problem is approached by means of episodic sampling, e.g. in each so-called episode, a specific task, e.g. a specific class, can be sampled from the training data, to lens instances, then the current set of transformation parameters can be used to perform said specific task, e.g. to measure its quality in terms of classification loss.

[0114] In both cases, the set of transformation parameters can be learned based on training data for a plurality of classes. Such training data can include one or more instances per class. For example, the instances can represent images or other types of sensor data, as also discussed above. A relatively large number (e.g., at least 10, at least 50, at least 100, etc.) of classes can be available. The number of classes is typically much larger than the number of novel classes used when adapting the base classifier using the set of transformation parameters, e.g., at least 10 times larger, at least 100 times larger, etc. Moreover, the number of training instances per class can be relatively large, e.g., at least 10 instances can be available for each class or at least for a majority of the classes, and the number of training instances per class is typically at least twice or five times larger than the number of instances of novel classes used when adapting the base classifier using the transformation parameters.

[0115] Figure 5 A first detailed example is shown of how to learn a set of transformation parameters for adapting a base classifier to one or more novel classes.

[0116] As discussed above, shown in the figure is training data TD 530 for a plurality of classes. Also shown in the figure are class representations CRi 551 through CRm 552 for the plurality of classes of the training data TD, and parameters FPAR 555 for a feature extractor used to extract feature representations from instances. The class representations CRi and the feature extractor parameters FPAR can together form a classifier C 450 that can be used to classify instances into the plurality of classes, e.g., as discussed with respect to Figure 3 The classifier C can be used as a base classifier to be adapted to novel classes, e.g., as discussed with respect to Figure 4

[0117] As shown in Figure 5 In this example, the class representations CRi and the parameters FPAR are determined by learning the class representations CRi and the parameters FPAR in a learning operation LI 510 prior to learning the set of transformation parameters. In other words, starting from the training data TD, the training is conducted in two stages: a first stage in which a base classifier is learned, and a second stage in which the transformation parameters are learned. For example, the first stage of the training can be performed as disclosed in S. Gidaris and N. Komodakis, “Dynamic Few-Shot Visual Learning without Forgetting,” available at https: / / arxiv.org / abs / 1804.09458, and incorporated herein by reference to the extent it concerns the first stage of training.

[0118] ​In the second learning operation L2 520, a set of transformation parameters TPAR 460 is learned. The transformation parameters are learned through contextual training; in other words, by repeatedly constructing a base classifier and adapting it using the corresponding values ​​of the transformation parameter set. Therefore, the optimal transformation parameters TPAR that best fit the base classifier can be found. For example, using stochastic gradient descent or other optimization methods, the transformation parameters TPAR can be updated iteratively based on the classification loss. It is well known that such optimization methods can be heuristic and / or may reach local optima.

[0119] More specifically, a base classifier can be constructed by selecting one or more base classes and one or more novel classes from multiple classes in the training data TD. For example, for Towards Lens classification can, for example, randomly select a fixed number from multiple categories. A novel class can be selected. The remaining classes or subsets thereof can be chosen as base classes. A base classifier can be constructed, configured to classify instances into one or more base classes based on class representations of one or more base classes and parameters of a feature extractor. For example, a base classifier could include class representations CRi of the base classes and transformation parameters TPAR.

[0120] Then, the base classifier can be adapted to the novelty class. To do this, training data for one or more novelty classes can be selected from the training data TD. For example, for Towards Lens classification can, for example, randomly select a fixed number of lenses from the training data TD for each novelty class. An instance. The constructed base classifier can then be adapted to one or more selected novel classes, for example, as per [example class]. Figure 4 The base classifier BC 450, training data TD 435, and transformation parameter TPAR 460 discussed herein are described.

[0121] The classification loss of the adapted base classifier can then be determined for instances of one or more base classes and one or more novel classes. For example, a fixed number of query instances can be sampled randomly from the training data TD for each class. Various numbers of query instances can be used (e.g., at most or at least 5, at most or at least 10, at most or at least 50, etc.). Interestingly, both base class instances and novel class instances can be used as query instances, allowing the model to discriminate globally and not just be rewarded for its performance on the novel class. The classification loss can be standard cross-entropy. For example, the loss can also be computed separately for novel class queries and base class queries, and then combined by a weighted average. The classification loss can at least be used to measure the quality of the transformation parameter TPAR; the transformation parameter TPAR can also be updated to reduce the loss function when using optimization methods such as stochastic gradient descent, thus iteratively improving the set of transformation parameters.

[0122] Figure 6 A second detailed example is shown, illustrating how to learn a set of transformation parameters to adapt a base classifier to one or more novel classes. Interestingly, in this second example, instead of using... Figure 5 The two-stage strategy involves learning class representations and / or feature extractor parameters for multiple classes within the same learning process as the set of transform parameters. For example, as shown in the figure, transform parameters TPAR 660, feature extractor parameters FPAR 655, and class representations CRi 651, 652 can all be learned from training data TD 630 in a single learning process L1 612. Particularly when using lateral information, the inventors found it beneficial to combine the training of feature extractor parameters FPAR, class representations CRi, and transform parameters TPAR in a single stage, as it avoids getting bogged down in feature extractor parameters and class representations that are inconsistent with the lateral information available for the transform parameters before learning the transform parameters TPAR.

[0123] More specifically, the training data for multiple classes (TD training data) is shown. As discussed above, a relatively large number of classes / instances can be available.

[0124] The figure also shows the class representations CR1 651 to CRm 652 for multiple classes of the training data TD; the parameters FPAR 655 of the feature extractor used to extract feature representations from instances; and the set of transformation parameters TPAR 560. These class representations and parameters correspond to the... Figure 5 The ones shown, except in this case, involve class representations and feature extractor parameters that are learned together with the set of transformation parameters by repeatedly constructing a base classifier and adapting it to the novel class through contextual training.

[0125] Similar toFigure 5 In the so-called scenario, one or more base classes and one or more novelty classes can be selected from the plurality of classes of the training data TD, e.g. a fixed number of novelty classes are selected, e.g. randomly, and the remaining classes or a subset thereof are selected as base classes. A base classifier can then be constructed which is configured to classify instances into the one or more base classes based on the current class representations CRi of the one or more base classes and the current parameters FPAR of the feature extractor. Training data for the one or more novelty classes can be selected from the training data (e.g. for each class of the shot classification), and the base classifier can be adapted to the one or more novelty classes using the current transformation parameters TPAR, e.g. as discussed with respect to Figure 4 However, in the , the classification loss based on the adapted base classifier for the query instance is not only used to learn the transformation parameters: interestingly, the class representations CRi and / or the feature extractor parameters FPAR can also be learned based on the classification loss, e.g. the class representations and / or the feature extractor parameters can be updated using an optimization method such as stochastic gradient descent to reduce the loss function. Thus, a loss function such as cross-entropy can be backpropagated to update various learnable parts of the model, including the feature extractor parameters FPAR, the class prototypes CRi of the initial seen classes, and various transformation parameters TPAR such as trainable components of the operators, the post-convolutional transformation, and the classifier temperature. Interestingly, multi-stage training or other pre-training can not be required such as for image features. Still, a classifier C650 including the feature extractor parameters FPAR and the class representations CRi can be obtained, e.g. for use as a classifier according to

[0126] or as a base classifier according to Figure 6 ; and the transformation parameters TPAR for adapting the classifier C to novelty classes not in the training data TD. Figure 3 Figure 4 At this point, interesting differences can be observed between the base class representations CRi 651, 652 as trained in this example and the novelty class representations CR 453 as trained when adapting the base classifier. Here, the base class representations can be learnable model parameters learned e.g. by stochastic gradient descent or the like. In particular, they are not directly computed from the training instances of that class. On the other hand,

[0127] Figure 4 ​​​The novel class representations CR can at least initially be determined from few-shot learning instances, in other words, they are determined on-the-fly for each new task run. Thus, fast learning for novel few-shot classes can be combined with slow learning for classes of the training data set TD, e.g., information from the training data set TD can be incorporated into the classifier instead of being forgotten.

[0128] Figure 7 A block diagram of a computer-implemented method 700 of adapting a base classifier to one or more novel classes is shown. The method 700 can correspond to Figure 1 the operation of the system 100 of FIG. 1. However, this is not a limitation, as another system, apparatus, or device can be used to perform the method 700.

[0129] The method 700 can include, in an operation titled “Accessing a Base Classifier,” accessing 710 data representing a base classifier, the data including class representations of one or more base classes of the base classifier and parameters of a feature extractor for extracting feature representations from instances. The base classifier can be configured to classify instances into the one or more base classes by matching feature representations extracted from the instances by the feature extractor to the class representations of the one or more base classes. The method 700 can further include adapting the base classifier by performing one or more of the operations 720-750 discussed below.

[0130] The method 700 can include, in an operation titled “Obtaining Training Data,” obtaining 720 training data for one or more novel classes. The training data for the novel classes can include one or more instances of the novel classes. The method 700 can further include, in an operation titled “Determining Class Representations,” determining 730 class representations of the one or more novel classes. The method 700 can include, in an operation titled “Extracting Feature Representations” performed as part of the determining 730, extracting 735 feature representations of the one or more instances of the novel classes using the feature extractor.

[0131] The method 700 can further include, in an operation titled “Determining Adapted Class Representations,” determining 740 adapted class representations of the base classes and the novel classes. The method 700 can include, in an operation titled “Adapting Novel from Base” performed as part of the determining 740, adapting 746 at least one class representation of a base class based on a class representation of a novel class. The method 700 can include, in an operation titled “Adapting Novel from Base” performed as part of the determining 740, adapting 745 at least one class representation of a novel class based on a class representation of a base class.

[0132] The method 700 can further include, in an operation titled "obtaining adapted class representations," associating 750 the adapted class representations with the base classifier to obtain a joint classifier for classifying instances into one or more base classes and one or more novelty classes.

[0133] Figure 8 A block diagram of a computer-implemented method 800 of learning a set of transformation parameters is shown. The set of transformation parameters can be used to adapt a base classifier to one or more novelty classes. The method 800 can correspond to the operation of the system 200 of Figure 2 However, this is not a limitation, as another system, apparatus, or device can also be used to perform the method 800.

[0134] The method 800 can include, in an operation titled "accessing data," accessing 810 training data for a plurality of classes, the training data for a class including one or more instances of the class, and data representing class representations of the plurality of classes and parameters of a feature extractor for extracting feature representations from instances. The method 800 can further include learning a set of transformation parameters by repeatedly performing one or more of the operations 820-860 discussed below.

[0135] The method 800 can include, in an operation titled "selecting base, novelty classes," selecting 820 one or more base classes and one or more novelty classes from the plurality of classes. The method 800 can include, in an operation titled "constructing base classifier," constructing 830 a base classifier configured to classify instances into the one or more base classes based on the class representations of the one or more base classes and the parameters of the feature extractor. The method 800 can also include, in an operation titled "selecting novelty class training data," selecting 840 training data for the one or more novelty classes from the training data. The method 800 can further include, in an operation titled "adapting base classifier," adapting 850 the base classifier to the one or more novelty classes by a method as described herein. The adapted class representations of the one or more base classes and the one or more novelty classes can be determined using the set of classification parameters. The method 800 can also include, in an operation titled "determining classification loss," determining 860 a classification loss of the adapted base classifier for instances of the one or more base classes and the one or more novelty classes.

[0136] It will be appreciated that, in general, Figure 7 the operations of the method 700 and Figure 8 the method 800 can be performed in any suitable order (e.g., consecutively, simultaneously, or a combination thereof), subject to a particular order necessary, for example, by input / output relationships, where applicable.

[0137] The method(s) can be implemented on a computer as a computer- implemented method, special-purpose hardware, or a combination of both. Also as illustrated in Figure 9 instructions (e.g., executable code) for a computer can be stored on a computer-readable medium 900, e.g., in the form of a series 910 of machine-readable physical marks and / or as a series of elements having different electrical (e.g., magnetic) or optical properties or values. The executable code can be stored in a transitory or non-transitory manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 9 An optical disc 500 is shown. Alternatively, the computer-readable medium 900 can include transitory or non-transitory data 910 representing a set of transformation parameters for adapting a base classifier to one or more novelty classes, as described elsewhere in this specification.

[0138] No example, embodiment, or optional feature, whether or not it is

[0139] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in the claims. The use of the article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The recitation of "at least one of a, b or c" suggests that things mentioned jointly can be included cumulatively, i.e. one of a, one of b, and one of c, and not exclusively one thing selected from the three-mentioned things. The application can be implemented by means of both hardware and software, and any combination thereof. In a device claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A system (100) for adapting a base classifier used for image classification to one or more novel classes, the system comprising: - A data interface (120) for accessing data (050) representing a base classifier, the data including: - The class representation of one or more base classes of the base classifier; and - Parameters for a feature extractor used to extract feature representations from instances, where the instances are images; The base classifier is configured to classify instances into one or more base classes by matching feature representations extracted from instances by the feature extractor with class representations of the one or more base classes; and - Processor subsystem (140), which is configured to adapt the base classifier as follows: - Obtain training data (035) for one or more novel classes, wherein the training data for novel classes includes one or more instances of the novel class; - Determine the class representation of the one or more novel classes, wherein determining the class representation of the novel class includes extracting feature representations of one or more instances of the novel class using a feature extractor; - Determine adapted class representations of the base class and the novel class, which include adapting at least one class representation of the novel class to the class representation of the base class, and adapting at least one class representation of the base class to the class representation of the novel class. - The adapted class representation of the target class is determined by calculating a weighted sum of the class representations of the one or more base classes and the one or more novel classes; and The weights of the corresponding class representations in the weighted sum indicate the similarity between the corresponding class representation and the target class representation, and the weights are determined based on the corresponding class representation and the target class representation; and - Associate the adapted class representation with the base classifier to obtain a joint classifier for classifying instances into the one or more base classes and the one or more novel classes; The weights are determined at least in part based on one of the more relational side information values ​​associated with the class representation, with one or more relational side information values ​​for the novel class independent of one or more instances of the novel class.

2. The system (100) according to claim 1, wherein the weights of the corresponding class representations in the weighted sum indicate the similarity between the class represented by the corresponding class representation and the target class independent of its class representation, and the data interface (120) is used for further access to the weights.

3. The system (100) according to claim 1 or 2, wherein, The processor subsystem (140) is configured to determine the adapted class representation of the target class by calculating multiple weighted sums using appropriate weights and combining the multiple weighted sums into a single adapted class representation.

4. The system (100) according to claim 1 or 2, wherein, The processor subsystem (140) is configured to further adapt the features of the class representation by calculating a weighted sum of the feature and / or lateral information values ​​of the class representation, thereby determining the adapted class representation.

5. The system (100) according to claim 1 or 2, wherein, The processor subsystem (140) is configured to determine the adapted class representation during iterations of the iterative process and to further adapt the adapted class representation in one or more further iterations of the iterative process.

6. The system (100) according to claim 1 or 2, wherein, The processor subsystem (140) is further configured as follows: - Obtain a query instance; and - Use a joint classifier to determine the classification score of the query instance relative to each base class and each novelty class, and from there determine the classification of the query instance into the base class or the novelty class.

7. A computer-implemented method (700) for adapting a base classifier for image classification to one or more novel classes, the method comprising: - Access (710) represents the data of the base classifier, said data including: - The class representation of one or more base classes of the base classifier; and - Parameters for a feature extractor used to extract feature representations from instances, where the instances are images; The base classifier is configured to classify instances into one or more base classes by matching feature representations extracted from instances by the feature extractor with class representations of the one or more base classes; and - Adapt the base classifier using the following steps: - Obtain (720) training data for one or more novel classes, wherein the training data for novel classes includes one or more instances of the novel class; - Determine (730) the class representation of the one or more novel classes, wherein determining the class representation of the novel classes includes extracting (735) the feature representation of one or more instances of the novel classes using a feature extractor; - Determine (740) adapted class representations of the base class and the novel class, comprising adapting (745) at least one class representation of the novel class to the class representation of the base class, and adapting (746) at least one class representation of the base class to the class representation of the novel class; and - The adapted class representation of the target class is determined by calculating a weighted sum of the class representations of the one or more base classes and the one or more novel classes; and The weights of the corresponding class representations in the weighted sum indicate the similarity between the corresponding class representation and the target class representation, and the weights are determined based on the corresponding class representation and the target class representation; and - Associate the adapted class representation with the base classifier (750) to obtain a joint classifier for classifying instances into the one or more base classes and the one or more novel classes; The weights are determined at least in part based on one of the more relational side information values ​​associated with the class representation, with one or more relational side information values ​​for the novel class independent of one or more instances of the novel class.

8. A system (200) for learning a set of transformation parameters, the set of transformation parameters being adapted to one or more novel classes for use in image classification, the system comprising: - Data interface (220), which is used to access: - Training data for multiple classes (030), training data for one class includes one or more instances of said class, wherein said instance is an image; and - Data (040) representing the class representations of the plurality of classes and the parameters of the feature extractor used to extract feature representations from instances; and - A processor interface configured to learn a set of transformation parameters by repeatedly performing the following operations: - Select one or more base classes and one or more novel classes from the plurality of classes; - Construct a base classifier, which is configured to classify instances into the one or more base classes based on the class representations of the one or more base classes and the parameters of the feature extractor; - Select training data from the training data for the one or more novel classes; - Adapting the base classifier to the one or more novel classes using the method of claim 7, wherein the adapted class representations of the one or more base classes and the one or more novel classes are determined using a set of transformation parameters; and - Determine the classification loss of the adapted base classifier for instances of the one or more base classes and the one or more novel classes.

9. The system (200) according to claim 8, wherein, The processor interface is configured to determine the class representations of the plurality of classes and the parameters of the feature extractor by learning the class representations and the parameters in a learning operation prior to learning the set of transformation parameters.

10. The system (200) according to claim 8, wherein, The processor subsystem (240) is configured to additionally learn class representations of the multiple classes and / or parameters of the feature extractor based on the classification loss of the adapted base classifier.

11. A computer-implemented method (800) for learning a set of transformation parameters, said set of transformation parameters being used to adapt a base classifier for image classification to one or more novel classes, the method comprising: - Visits (810): - Training data for multiple classes, training data for one class including one or more instances of said class, wherein said instance is an image; and - Data representing the class representations of the plurality of classes and parameters of the feature extractor used to extract feature representations from instances; and - Learn the set of transformation parameters by repeatedly performing the following operations: - Select (820) one or more base classes and one or more novel classes from the plurality of classes; - Construct (830) a base classifier, which is configured to classify instances into the one or more base classes based on the class representations of the one or more base classes and the parameters of the feature extractor; - Select (840) training data from the training data for the one or more novel classes; - The base classifier is adapted (850) to the one or more novel classes by the method of claim 7, wherein the adapted class representations of the one or more base classes and the one or more novel classes are determined using a set of classification parameters; and - Determine the classification loss of the adapted base classifier for instances of the one or more base classes and the one or more novel classes.

12. A computer-readable medium (900) comprising temporary or non-temporary data (910) representing the following. - Instructions, which, when executed by the processor system, cause the processor system to perform the computer-implemented method according to claim 7 or 11; and / or - A set of transformation parameters used to adapt a base classifier to one or more novel classes using a computer-implemented method according to claim 7.

Images