Hybrid learning agent for small sample classification

Abstract

A computer system and method for training a machine learning system to perform a classification task by dividing input data into one of a plurality of classes. The system is configured to receive training data for each class from which a representation of each class can be derived, wherein each class is described by a plurality of representations; process the training data to form, for at least one class, a first proxy for a relatively global portion of training data items and a plurality of proxies for different relatively local portions of training data items, each proxy corresponding to a representation of data belonging to the class. For each training data item, the system is configured to: evaluate a match between the training data item and the proxies; estimate the class of the training data item from the match rating; adjust the proxies by updating a weighting matrix to reduce the distance between the training data item and the proxies of the estimated class. Defining a plurality of proxies in this way can result in richer and more stable representations of object classes.

Description

Technical Field

[0001] This invention relates to object classification, and more particularly to classifying objects in an image using few-sample classification. Background Technology

[0002] Deep neural networks for image classification can achieve superhuman performance when trained on large amounts of annotated data. However, these deep neural networks are susceptible to overfitting when training data is limited. Therefore, novel, rare classes, in which annotated data is difficult to obtain, can lead to low classification accuracy. In contrast, humans are able to identify novel classes from a small number of examples.

[0003] Few-Shot Learning (FSL) aims to teach models to mimic human behavior in order to identify and process novel classes previously unseen in systems with limited data. Previous research on FSL has generally fallen into two main categories: meta-gradient learning and metric learning.

[0004] Meta-gradient learning-based methods focus on teaching the model through a small number of regular gradient descent iterations to quickly adapt to new classes. Many recent meta-gradient methods use a learn-to-learn paradigm to train the meta-learner. A popular strategy in this paradigm involves finding the optimal network parameter initialization so that fine-tuning becomes fast, requiring only a few weight updates.

[0005] In metric learning-based techniques, a distance metric is learned between a query image and a set of labeled images, such that the query image is closest to the labeled images of the same class. The key idea of ​​metric learning is to learn deep embeddings of input samples to minimize a predefined distance metric between samples of the same class. These methods typically rely on class proxies, which are used to classify unlabeled images using a nearest neighbor strategy. A proxy can be defined as a global representation of a class computed from the embeddings of images supported by a set of annotations. The key to metric learning is learning a good global class representation for each class, which is used to classify unlabeled images at test time, typically using a nearest neighbor strategy. Common methods for defining representative class proxies include using the average feature representation of a set of labeled images. Metric learning methods constitute a very popular strategy for learning discriminative representations so that images containing different classes are well separated in the embedding space.

[0006] Despite significant improvements in metric learning methods, existing metric-based FSL methods may still have inherent limitations because they generally assume that a single agent can be used to summarize each category and then used as a reference for inferring class labels. By considering only a unimodal agent for each class, such methods fail to capture the complex multimodal class distributions often present in real-world problems, and also fail to capture the subtle differences between similar classes, such as... Figure 1 As shown. Figure 1 A visualization of the t-distributed random nearest neighbor embeddings of feature embeddings for supported and queried images in the miniImageNet test phase is shown in a 5-way 1-sample setting (CVPR 2018, Qi, H., Brown, M., and Lowe, DG, “Low-shot learning with imprinted weights”). This illustrates two drawbacks of metric learning methods using a single agent. First, the agent may lack representativeness and be unable to assign weights, as shown in 101. Second, a single agent cannot accurately capture the multimodal distribution of the class, as shown in 102 and 103.

[0007] Furthermore, the performance of these models can be perceived by the quality of the agent, and the discriminative power of the models may be limited.

[0008] The methods described by Allen, KR, Shelhamer, E., Shin, H., and Tenenbaum, JB in arXiv, 2019, "Infinite mixture prototypes for few-shot learning," and by Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., and Luo, J. in CVPR, 2019, "Revisiting localdescriptor based image-to-class measure for few-shot learning," consider relying on multiple agents. These methods propose multiple agent representations as clusters and local descriptors. However, these methods suffer from the limitation that the agents may not be optimized for diversity, thus limiting the benefits of multiple representations. Furthermore, due to the use of local inputs, the local descriptors are not regularized, resulting in potentially poorly representative agents.

[0009] Furthermore, compared to learning image-level representations, global image-based metrics can be too coarse and ineffective in few-shot scenarios where samples are scarce. Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., and Luo, J., proposed learning local descriptors for their image-to-class metrics in their CVPR 2019 paper, "Revisiting local descriptor-based image-to-class measure for few-shot learning." Allen, KR, Shelhamer, E., Shin, H., and Tenenbaum, JB, optionally used Infinite Mixture Prototypes (IMP) in their arXiv 2019 paper, "Infinite mixture prototypes for few-shot learning." The IMP method represents each class as a set of clusters (prototypes), each cluster consisting of class image representations. However, using the IMP clustering strategy for class representations may not provide any mechanism to account for prototype diversity.

[0010] The goal is to develop an improved few-sample classification method that can identify new, previously unseen object classes using only a limited number of training samples, thereby addressing these problems. Summary of the Invention

[0011] According to a first aspect, a computer system is provided for training a machine learning system to perform a classification task by classifying input data into one of a plurality of classes. The system is configured to: receive training data for each class, from which a representation of each class can be derived, wherein each class is described by a plurality of representations; process the training data to form, for at least one class, a first agent for a relatively global portion of a training data item and a plurality of agents for different relatively local portions of the training data item, each agent corresponding to a representation of data belonging to that class; for each training data item, evaluate a match between the training data item and the agents; estimate the class of the training data item based on a matching level; and adjust the agents by updating a weighting matrix to reduce the distance between the training data item and the agent whose class is estimated.

[0012] This reduces the inherent biases and limitations associated with using a single representation and allows for the learning of richer proxy representations that accurately capture the underlying data distribution and enhance model stability. Forming a combination of local and global descriptors allows for the computation of different class proxy sets that focus on different aspects of the image. This enables the model to be taught to handle new classes in data-constrained regimes, thereby simulating relevant human capabilities.

[0013] The agent can be defined by the weights of the model learned by the machine learning system. This allows for efficient learning of the agent.

[0014] The steps of processing the training data may further include: for at least one class, employing a self-supervised rotation prediction training task to enhance the representativeness of the agent. Using a self-supervised rotation loss task can normalize the learning process of local inputs and enhance the representativeness of local agents, producing stable and class-representative local agents.

[0015] The steps for processing training data may include: for at least one class, forming multiple agents through a process used to encourage differences between these agents. This maximizes the performance of the combination.

[0016] The system can be used to evaluate the match between training data items and the agent through a soft attention mechanism. This can improve the accuracy of the trained model.

[0017] The soft attention mechanism may include processing the matching degree between the training data items and each agent in the agents according to a soft attention algorithm, and the computer system is used to train the soft attention algorithm to improve the system's habit of correctly classifying input data. Soft attention gates may be trained to merge classification decisions associated with each local and global agent representation. Regularizing agents using an attention mechanism to merge agent classification decisions can effectively allow the ignoring of unreliable and non-discriminatory agents (and image regions).

[0018] Each training data item can be an image. This allows the trained model to be used for classifying images captured by image sensors in devices such as smartphones.

[0019] The computer system can be used to extract features from each image. This allows for the estimation of the proxy set through global and local pooling of the image feature extractor's output.

[0020] According to a second aspect, a computer system is provided, including a machine learning system for performing a classification task by classifying input data into one of a plurality of classes. The system is configured to: for each of the plurality of classes, store a plurality of proxies, wherein each proxy represents a feature of data belonging to that class; and classify the input data by evaluating a match between the input data and each of the proxies. The machine learning system is preferably trained by the aforementioned computer system. This allows images captured by image sensors in devices such as smartphones to be classified according to their content.

[0021] According to a third aspect, a method is provided for training a machine learning system to perform a classification task by classifying input data into one of a plurality of classes. The method includes: receiving training data for each class, from which a representation of each class can be derived, wherein each class is described by a plurality of representations; processing the training data to form, for at least one class, a first agent for a relatively global portion of a training data item and a plurality of agents for different relatively local portions of the training data item, each agent corresponding to a representation of data belonging to that class; for each training data item, evaluating a match between the training data item and the agents; estimating the class of the training data item based on a matching level; and adjusting the agents by updating a weighting matrix to reduce the distance between the training data item and the agent representing the estimated class.

[0022] Using this method reduces the inherent biases and limitations associated with using a single representation and allows for the learning of richer proxy representations that accurately capture the underlying data distribution and enhance model stability. Forming a combination of local and global descriptors allows for the computation of different class proxy sets that focus on different aspects of the image. The resulting trained model is able to handle new classes in data-limited regimes, thereby simulating relevant human capabilities.

[0023] The agent can be defined by the weights of the model learned by the machine learning system. This allows for efficient learning of the agent.

[0024] The steps for processing the training data may include: for at least one class, employing a self-supervised rotation prediction training task to enhance the representativeness of the agent. Using a self-supervised rotation loss task can normalize the learning process of local inputs and enhance the representativeness of local agents, producing stable and class-representative local agents.

[0025] The steps for processing training data may include: for at least one class, forming multiple agents through a process used to encourage differences between these agents. This maximizes the performance of the combination.

[0026] The matching between training data items and the agent can be evaluated using a soft attention mechanism. This can improve the training performance of the algorithm.

[0027] The soft attention mechanism may include processing the matching degree between the training data items and each agent in the agents according to the soft attention algorithm, and the method may include training the soft attention algorithm to improve the system's habit of correctly classifying input data.

[0028] Each training data item can be an image. This allows models trained using the method to classify images captured by image sensors in devices such as smartphones.

[0029] The method may also include extracting features from each image. This allows for the estimation of the proxy set using global and local pooling of the image feature extractor output.

[0030] The method can be executed by a computer system comprising one or more processors programmed with executable code stored non-transiently in one or more memories. This can help reduce the need for manual image classification. Attached Figure Description

[0031] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0032] Figure 1 A visualization of the t-distributed random nearest neighbor embeddings of feature embeddings for supported and queried images in a 5-way 1-shot setting during the miniImageNet test phase is shown.

[0033] Figure 2 This schematically illustrates a hybrid overview of the proxy model and the imprint weight implementation.

[0034] Figure 3 This is a flowchart of an exemplary method for training a machine learning system to perform a classification task by dividing input data into one of a number of classes.

[0035] Figure 4 An example of an imaging device for implementing the computational system and methods described herein is shown. Detailed Implementation

[0036] This paper describes a mixture of proxy (MP)-based metric learning method for few-shot classification. To address some limitations of single-proxy metric learning methods, the MP method learns multimodal class representations and can be integrated into existing metric-based methods. The method described in this paper focuses on learning high-quality proxies and maximizing the use of multiple class-specific representations.

[0037] A proxy can be defined as a global representation of a class. In the embodiments described below, class proxies are modeled as a set of feature representations designed to maximize both individual performance (high representativeness) and composite performance (high inter-proxy variance). This can be achieved by computing a set of local and global class proxies, which is advantageous for focusing on different regions and image attributes.

[0038] Figure 2 An overview of the machine learning system architecture 200 is illustrated schematically. In this paper, the hybrid agent method is combined with the imprinted weights (FSL) method described by Qi, H., Brown, M., and Lowe, DG at CVPR 2018, “Low-shot learning with imprinted weights,” as an example. This method can also be combined with other FSL methods.

[0039] The training phase is typically shown as 250. Consider a training set of images 201, which includes a large set of annotated images and B basic categories. The training set includes training data 201 for each class, from which a representation for each class can be derived, wherein each class is described by multiple representations. Using the training set 201, the model is first trained on the basic categories. The goal of the method is to learn to label a new set of unseen images, associating them with U new unseen categories.

[0040] The system is used to process training data 201 to form multiple agents W1-W for each class. N+1 Each agent corresponds to a representation of data belonging to that class. In this paper, the agents can be defined by the weights of a model learned by a machine learning system.

[0041] Trainable feature extractor 202 is used to extract features 203 from images in training set 201. Different feature representations (surrogates) W1-W are estimated through global 204 and local 205 pooling of the output of trainable image feature extractor 202. N+1 The set. Each representation is associated with a trainable classifier (shown as 2541-254 in the test phase). N+1 Using global pooling, as shown in 204, a single global agent W is computed for each training data item. N+1 Therefore, for a relatively global portion of the training data item, the first agent is preferably the entire training data item (e.g., an image). Using local pooling, as shown in 205, multiple local agents W1-W are computed. NDifferent relative local portions or regions of the training images (i.e., regions smaller than the larger global portion of the training data item used to determine the first agent) can be used to determine each local agent.

[0042] For each of the 201 training data items, the system is used to: evaluate the training data item against the agent W1-W N+1 The process involves matching each agent in the training data; estimating the class of the training data item based on the matching level; and adjusting the agents by updating the weighting matrix to reduce the distance between the training data item and the agent whose class is estimated.

[0043] In one embodiment, the normalized input embedding and classifier weight matrix W can be used as the basis. i The classification decision is made based on the proportional cosine distance between the columns, so that W i Each column constitutes a trainable class proxy.

[0044] As shown in 206, a soft attention gate can be trained to merge classification decisions associated with each local and global agent representation, outputting a classification loss 207. Therefore, the soft attention gate 206 can be used to regularize the local agent to merge classification decisions from each agent. This effectively allows for the ignoring of unreliable and non-discriminatory agents (image regions) and / or self-supervised tasks that regulate the learning process of local inputs, producing stable and class-representative local agents.

[0045] In some embodiments, the feature representation can be optimized using a self-supervised rotation loss associated with a rotation-specific embedding network, as shown in Figure 208. This is described in more detail below.

[0046] During the test period, such as Figure 2 As shown in Figure 260, the agent can be determined from the embeddings of a set of annotated supporting images. The global agent and multiple local agents of the new class can be computed by using the average representation calculated on the support set 253 using global 251 and local 252 pooling, and in the trained classifier 2541-254 N+1 This leaves an imprint, effectively allowing testing of new classes without retraining the model. As shown in Figure 255, soft attention gates can merge classification decisions associated with each local and global agent representation and give a classification output.

[0047] An example of the method will now be described in more detail.

[0048] Consider an annotated sample X b ={x1,…,x n} and its inclusion of C b The corresponding label Y for each basic category b ={y1,…,y nThe training dataset D base The test dataset D used in this paper novel Including D novel A new class is created, with each class associated with only a small number of labeled samples (e.g., less than or equal to 5 samples), while the remaining unlabeled samples are used for evaluation.

[0049] The purpose of small sample classification is to classify D base Learn a classifier that generalizes well to C++-based systems. n C of limited labeled samples in a new category n A new class. Specifically, these labeled samples constitute a class with K. n The support set S of the annotation samples n The unlabeled samples constitute the query set Q for evaluating the model. n This problem is also known as C. n -way K n -shot classification problem. Many FSL methods also use D base Support S in b Set and query Q b A subset of the set is used for set training and sampling to simulate query test scenarios.

[0050] By using local and global average pooling, a set of N local representations is used to enhance global image feature representations that focus on different regions. These representations, computed on the support set, constitute class proxies, which are then used to classify unlabeled examples using cosine distance. This allows for the utilization of high-granularity local descriptors without sacrificing global information. Proxies obtained from local image inputs may be of poor quality if they focus on blurred or irrelevant image regions (e.g., background). This problem can be addressed by learning stable features through self-supervised rotation loss and combining agent classification decisions with soft attention gates.

[0051] The examples described below focus on combining hybrid surrogate methods with metric learning-based methods because these approaches are simple, flexible, and offer state-of-the-art performance. However, this approach can also be applied to other FSL methods, metric learning-based methods, and meta-gradient learning-based methods.

[0052] Metric-based FSL methods focus on learning strong feature representations θ f These represent recombining images of the same class and separating different classes relative to a predefined distance metric γ(·). Depending on the method considered, the agent p associated with class c... c During training, it can be defined as (a) the support set images S cThe average representation of the classifier weights trained via standard backpropagation on the baseline dataset (e.g., "Prototypical networks for few-shot learning" by Snell, J., Swersky, K., and Zemel, R. at NeurIPS 2017) is given by column c. At test time, all methods preferably use option (a). Then, the unlabeled image x is assigned to the agent γ(x, p) of the different classes. c The embedding distance is used to classify them.

[0053] The goal is to use hybrid proxies to learn richer category representations to accurately represent the variability within a class. The support set representation can be decomposed into a set of N+1 proxy representations. Each proxy representation can be assigned a separate class based on distance.

[0054] As referenced above Figure 2 In summary, the model can be designed to maximize the use of multiple agents by considering factors from both local and global model components, increase stability to local inputs and improve local spatial inference by using an auxiliary task involving image rotation, and increase the impact of reliable agent predictions by using soft attention gates, potentially forcing high variance. These elements will be described in more detail below.

[0055] A key criterion in hybrid agent design is to maximize the variance between agents, thereby minimizing redundancy among these representations. This can be achieved using local and global agent learning methods.

[0056] Considering the data from the training dataset D base Annotated image x in b θ f (x b ) represents it, among which and These are the feature vector channels, width, and height, respectively. A trainable feature extraction network can be used (…). Figure 2 (As shown in Figure 202) Features are extracted from each item in the training dataset.

[0057] Instead of simply performing average pooling on the entire image, average pooling can be applied to N disjoint local regions (i.e., different relative local portions of the image). These N disjoint local regions can be obtained by uniformly segmenting the image feature representation along the height H, width W, or both of these regions, so that the nth local agent focuses on a specific region R of the input image. n The number of proxies along the height and / or width can constitute a hyperparameter.

[0058] By designing local proxies that focus on disjoint parts of the image, the proxies may be forced to provide supplementary information and limit redundancy. However, relying solely on fine-grained local representations may overlook high-level global information, which can also provide highly useful cues. Therefore, multiple local proxy representations p n The set n∈[1,…,N+1] can be combined with a global proxy considering the entire image, p. N+1 The global proxy uses global average pooling θ f (x b Parallel computation. The combination of local and global descriptors allows computation of different classes of proxies that focus on different aspects of the image.

[0059] However, in some embodiments, using only multiple local descriptors can lead to two problems that may limit the performance of multi-agent strategies. First, learning accurate embeddings and classifiers using local agents can be challenging and result in below-standard performance due to the potential ambiguity associated with partial image input. Second, local agents may focus on non-discriminatory image regions and therefore not provide relevant information. These potential problems can be addressed by using self-supervised methods to regulate the predictions of both local agents and attention-based agent combinations, as described in more detail below.

[0060] Recent advances in unsupervised and semi-supervised learning have demonstrated the advantages of self-supervision in canonicalizing model training and learning more robust feature representations. Training a classifier using local image information presents a similarly challenging scenario where the local information may be unclear or even lack the category of interest. In some implementations, this potentially unreliable signal can impair model training and potentially produce suboptimal proxy representations. Combining self-supervised auxiliary tasks can help learn more stable features by extracting features suitable for multiple high-level tasks, thereby learning the proxy. This effectively allows for optimizing the representational power of the local proxy.

[0061] In some embodiments, an auxiliary rotation task (such as...) can be used. Figure 2(Illustrated in 208). This can be particularly advantageous because strict rotation preserves spatial continuity and image properties helpful to the main task, unlike other common alternatives that might be used, such as jigsaw puzzle tasks (e.g., see "Boosting supervision with self-supervision for few-shot learning" by Su, J.-C., Maji, S., and Hariharan, B. in arXiv, 2019). Formally, given from D base Training image x b It can be done by x b Rotate the image by r degrees to generate four strictly transformed images, where r ∈ {0°, 90°, 180°, 270°}. The auxiliary rotation task can be formulated as a four-class classification problem, aiming to correctly identify the rotation r. Image local embedding can be passed through a 1×1 convolutional layer. and global embedding Post-trained linear classifier W r This is achieved by using an additional convolutional layer to transform the feature vector... The task is adjusted to a rotation task, and conflicts with the main classification task are implicitly prevented. The rotation branch can then be finally trained using standard softmax cross-entropy loss.

[0062]

[0063] Where Φ is the rotation embedding function, ρ c It is the rotational prediction score, δ c,y It is the Dirac delta function.

[0064] Therefore, in some embodiments, rotation prediction tasks can be added in parallel to class prediction to normalize the training process and improve performance. Thus, by employing a self-supervised rotation prediction-assisted training task, the representativeness of the formed agents can be strengthened in some implementations of the method.

[0065] Implementation examples including methods for using attention-based proxy prediction will now be described.

[0066] Local agent classification task utilities may vary. In the method embodiments described herein, the task utility and weighted agent are combined and learned using attention.

[0067] For a given input image x, the agent-specific classification score f n (x) and image region R n Associate and calculate for all C N Embedding of each class θf (x) n and agent p n Standardized distance between:

[0068]

[0069] in, and These are the classification score and proxy associated with class c, respectively.

[0070] A simple strategy could be to average all agent decisions to obtain a combined global score. However, in some implementations, this strategy may be affected by poorly informed local agents that focus on non-discriminatory regions. Alternatively, in a preferred implementation, soft attention gates could be aggregated to modulate the combination of agent decisions and weaken the signal propagated by low-quality agents.

[0071] Soft attention gate It can be designed as a single softmax and fully connected layer to represent the global image θ. f (x) is taken as input and reshaped into a vector. Then, each agent α = {α n The attention weights of} can be calculated as follows:

[0072] To mitigate any potential errors caused by noisy or difficult examples, the method described by Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., and Tang, X. in their 2017 CVPR presentation, "Residual attention network for image classification," combines gates with residual connections. This may result in more stable performance for inaccurate attention weights.

[0073] Finally, the classification score of image x can be calculated as follows:

[0074]

[0075] Then, the classification branch of the model can be trained using prediction and standard metric learning strategies.

[0076] The hybrid agent model described above can provide a general formula that can be easily combined with popular metric-based few-shot learning models.

[0077] As described above, in the preferred embodiment, such as Figure 2As illustrated in the diagram, the hybrid surrogate model can be implemented using the imprinted weight model described by Qi, H., Brown, M., and Lowe, DG at CVPR 2018, in their presentation, "Low-shot learning with imprinted weights." Other training strategies can also be used.

[0078] Imprint weighting method throughout the entire base class C b The classifier is trained on the set. The architecture includes a feature extraction network θ. f Then there is a classifier, which comprises an unbiased W∈F×C b A fully connected layer, where F is θ f The output dimension of W can be learned, such that w c The embedding θ of the input image of class c (column c of W) and class c f (x c The cosine distance between them is the smallest.

[0079] Therefore, w can be c Consider it as a proxy for the c-th class in the basic set. The objective function aims to minimize the cosine distance between the image and its corresponding proxy.

[0080] Using the imprinted weight model offers two main advantages. First, due to the training strategy, each row of the classifier matrix W constitutes a proxy, which helps new classes to be easily imprinted onto W using the support set proxies. This alleviates the need to retrain or fine-tune the model when new classes become available or the number of samples changes, resulting in an efficient model with continuous learning capabilities. Second, the classifier training method does not require a cumbersome set training process. However, traditionally, imprinting strategies can make the model highly sensitive to proxy quality and prone to failure in single-proxy scenarios.

[0081] The hybrid surrogate approach described in this paper focuses on strong multimodal representations and helps to fully leverage the benefits of this model while maintaining stable performance. In this case, the hybrid surrogate approach can be combined in a natural way, associating each of N local feature vectors and a single global feature vector with a different classifier.

[0082] As mentioned earlier, for example, classification decisions can be computed by evaluating the cosine distance between the input image and each column of a given classifier matrix, where each column corresponds to a class. Therefore, classifier weights can be learned to minimize the distance between embeddings and surrogates (classifier columns) of the same class.

[0083] Since each classifier focuses on different feature regions of the image, it can automatically learn N+1 different local and global proxies as each classifier matrix W1, W2, ..., W N+1 The columns. Specifically, for a given classifier W i The classification score of sample x of class c can be calculated as follows:

[0084]

[0085] Among them, w ij It is the weight matrix W i The j-th column corresponds to the region R i The proxy p associated with class j ij Scaling cosine similarity is defined as...

[0086] W i and θ f (x) can all be normalized using the L2 norm, where s is a trainable scalar (as described by Qi, H., Brown, M., and Lowe, DG at CVPR 2018, “Few-shot learning with imprinted weights”). This helps to avoid the risk of cosine distance producing a distribution that lacks discriminative power.

[0087] Then, the classification loss function is calculated as follows:

[0088]

[0089] Among them, formula (3) is used to extract from all f i c Calculate f c δ c,y It is the Dirac delta function. In formula (5), we retain only logf. i c The sum of (x) items, so that each agent can have authentication class information.

[0090] Then, the objective function can be used. The entire model is trained end-to-end. During testing, given D... novel The new category j in the dataset S has supporting datasets. j The new agent set can be computed as:

[0091]

[0092] Among them, S j Includes all annotation samples in class j.

[0093] By classifier Marked as And by repeating this process for any new category, new classes can be identified without retraining the model. By using W... n and Cascading allows the model to be used in all C... n +C b Tests are conducted on a category-by-category basis.

[0094] Figure 3 An exemplary method for training a machine learning system to perform a classification task by dividing input data into one of a plurality of classes is summarized. In step 301, the method includes receiving training data for each class, from which a representation of each class can be derived, wherein each class is described by a plurality of representations. In step 302, the method includes processing the training data to form a first agent for a relatively global portion of a training data item and a plurality of agents for different relatively local portions of the training data item, each agent corresponding to a representation of data belonging to that class. Then, steps 303 through 305 are performed for each training data item. In step 303, the method includes evaluating a match between the training data item and the agents. In step 304, the method includes estimating the class of the training data item based on a matching level. In step 305, the method includes adjusting the agents by updating a weighting matrix to reduce the distance between the training data item and the agent of the estimated class.

[0095] The method can be implemented on a computer system suitable for training a machine learning system to perform a classification task by dividing input data into one of a plurality of classes.

[0096] The trained model can be implemented on a computer system, including a machine learning system, for performing a classification task by dividing input data into one of a plurality of classes. The system is used to: store multiple proxies for each of the multiple classes, each proxy representing features of data belonging to that class; and classify the input data by evaluating the match between the input data and each proxy.

[0097] Figure 4 An example of a system 400 including device 401 is shown. Device 401 is used to train the system to perform a classification task and / or classify image data captured by at least one image sensor in the device using the methods described herein.

[0098] In the example described, device 401 includes image sensors 402 and 403. This device 401 typically includes some onboard processing capabilities. This capability may be provided by processor 404. Processor 404 can also be used to perform basic functions of the device. The device also includes memory 406. The memory may store code in a non-transitory manner, which can be executed by the processor to implement the operation of the method and the device.

[0099] Transceiver 405 is capable of communicating with other entities 410, 411 via a network. These entities may be physically located away from device 401. The network may be a publicly accessible network, such as the Internet. Entities 410, 411 may be cloud-based. Entity 410 is a computing entity. Entity 411 is a command and control entity. These entities are logical entities. In practice, these entities may be provided by one or more physical devices (such as servers and data storage), and the functionality of two or more entities may be provided by a single physical device. Each physical device implementing an entity includes a processor and memory. These devices may also include transceivers for sending data to and receiving data from transceiver 405 of device 401. The memory stores code in a non-transitory manner, which can be executed by the processor to implement the corresponding entity using the methods described herein.

[0100] Command and control entity 411 can train the artificial intelligence model used in the device. Even if the resulting model can be efficiently described, such training is typically a computationally intensive task. Developing the algorithm to be executed in the cloud can be efficient, as a large amount of energy and computing resources can be expected to be available in the cloud. It can be expected that this is more efficient than forming such a model in a typical imaging device.

[0101] In one implementation, once the algorithm is developed in the cloud, the command and control entities can automatically generate the corresponding model and transmit it to the relevant imaging device. In this example, the model is implemented by the processor 404 in device 401.

[0102] In another possible implementation, the image can be captured by one or both of sensors 402 and 403, and the image data can be sent to the cloud by transceiver 405 for processing, thereby classifying the image. The resulting image can then be sent back to device 401, such as... Figure 4 As shown in 412.

[0103] Therefore, the method can be deployed in various ways, such as in the cloud, on a device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new algorithms or refine existing ones. Depending on the computational power close to the data corpus, training can be performed near the source data or in the cloud using inference engines or similar methods. The method can also be implemented on a device, in dedicated hardware, or in the cloud.

[0104] Existing metric-based FSL methods typically restrict class representations to unimodal surrogates, while the method described in this paper offers a solution to a significant limitation often associated with these strategies. To address the limitations of previous methods, this paper describes a hybrid surrogate approach that learns multimodal class representations and can be combined with existing metric-based methods. The method described in this paper reduces the inherent biases and limitations associated with using a single representation and allows the learning of richer surrogate representations that accurately capture the underlying data distribution and enhance model stability. This addresses the FSL problem in image classification: teaching the model to handle new classes in a data-constrained regime (thus simulating relevant human capabilities).

[0105] As described above, each class learns a set of agents optimized to maximize individual performance (high representativeness) and combinatorial performance (high inter-agent variance). Class agents are modeled as a carefully designed set of feature representations to be highly diverse and maximize combinatorial performance. This can be achieved by computing a set of local and global feature vectors, which is beneficial for focusing on different regions and image attributes. Local agents can be regularized using soft attention gates to incorporate agent classification decisions, effectively allowing the ignoring of unreliable and non-discriminatory agents (image regions), and allowing the self-supervised rotation loss task to normalize the input to the local learning process and strengthen the representativeness of local agents, resulting in stable and class-representative local agents.

[0106] Therefore, image-level representations are combined with local descriptors, and the influence of local agents is carefully normalized using self-supervision and attention to maximize agent diversity and representation. This approach allows for accurate separation and generalization to new classes due to the resulting richer representations, and the model is designed to jointly optimize agent variance and representation.

[0107] The MP learning strategy of FSL described in this paper provides a simple and general approach that can be easily embedded into pre-existing metric learning-based methods.

[0108] The increased stability of the representation imparted by the hybrid proxy facilitates combining this method with the imprinted weights single proxy approach to produce an efficient formula that also maintains high accuracy due to the high-quality proxy representation. The model only needs to be trained once, providing an efficient and consistent model that does not require retraining when the number of training samples changes or new classes are introduced. Therefore, a no-sample model can be trained that continuously adapts to new classes without retraining.

[0109] Experiments on miniImageNet and tieredImageNet show that combining MP with metric learning methods can improve performance, and the imprinted weight MP model has achieved classification accuracy exceeding state-of-the-art by more than 3% (miniImageNet) and 1.5% (tieredImageNet) in some implementations in 1-shot and 5-shot settings.

[0110] Compared to previous multi-agent methods, such as those described by Allen, KR, Shelhamer, E., Shin, H., and Tenenbaum, JB in arXiv, “Infinite mixture prototypes for few-shot learning,” and those described by Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., and Luo, J. in CVPR, “Revisiting local descriptor based image-to-class measure for few-shot learning,” the MP method is highly diverse and can leverage attention to determine agent importance and self-supervision to optimize the representation of local agents. This helps to fully utilize mixture agent methods and can improve classification decisions for both single and combined agents.

[0111] The applicant hereby discloses individually each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, regardless of whether such features or combinations of features solve any of the problems disclosed herein, and without limiting the scope of the claims. This application demonstrates that aspects of the invention can be constituted by any such individual features or combinations of features. In view of the foregoing description, various modifications that can be made within the scope of the invention will be apparent to those skilled in the art.

Claims

1. A computer system, characterized in that, The computer system is used to train a machine learning system to perform a classification task by classifying input data into one of a plurality of classes; the computer system is used to: The system receives training data for each class, from which a representation of each class can be derived, wherein each class is described by multiple representations and each training data item is an image; the computer system is used to extract features from each image. The training data is processed to form a first agent for a relatively global portion of a training data item and multiple agents for different relatively local portions of the training data item for at least one class, each agent representing a feature of data belonging to that class; the different relatively local portions of the training data item include N disjoint local regions; the agents are defined by weights of a model learned by the machine learning system. For each training data item, evaluate the match between the training data item and the agent; estimate the class of the training data item based on the match level; adjust the agent by updating the weighting matrix to reduce the distance between the training data item and the agent of the estimated class.

2. The computer system according to claim 1, characterized in that, The step of processing the training data further includes: for at least one class, using a self-supervised rotation prediction training task to enhance the representational capabilities of the agent.

3. The computer system according to claim 1, characterized in that, The system is used to evaluate the matching between training data items and the agent through a soft attention mechanism.

4. The computer system according to claim 3, characterized in that, The soft attention mechanism includes processing the matching degree between the training data items and each agent in the agents according to the soft attention algorithm, and the computer system is used to train the soft attention algorithm to improve the system's habit of correctly classifying input data.

5. A computer system comprising a machine learning system trained by any one of claims 1-4 and used to perform a classification task by classifying input data into one of a plurality of classes, characterized in that, The system is used for: For each of the multiple classes, multiple proxies are stored, where each proxy represents a feature of the data belonging to that class; the proxies are defined by the weights of the model learned by the machine learning system. The input data is classified by evaluating the matches between the input data and each agent in the agent.

6. An image classification method for training a machine learning system to perform a classification task by classifying input data into one of a plurality of classes, characterized in that, The method includes: The system receives training data for each class, from which a representation of each class can be derived, wherein each class is described by multiple representations. The training data is processed to form a first agent for a relatively global portion of a training data item and multiple agents for different relatively local portions of the training data item for at least one class, each agent representing a feature of data belonging to that class; the different relatively local portions of the training data item include N disjoint local regions; the agents are defined by weights of a model learned by the machine learning system. For each training data item, the matching between the training data item and the agent is evaluated; the class of the training data item is estimated based on the matching level; the agent is adjusted by updating the weighting matrix to reduce the distance between the training data item and the agent of the estimated class; wherein each training data item is an image; the method further includes extracting features from each image.

7. The method according to claim 6, characterized in that, The steps for processing the training data include: for at least one class, using a self-supervised rotation prediction training task to enhance the representational capabilities of the agent.

8. The method according to claim 6, characterized in that, The matching between training data items and the agent is evaluated using a soft attention mechanism.

9. The method according to claim 8, characterized in that, The soft attention mechanism includes processing the matching degree between the training data items and each agent in the agents according to the soft attention algorithm, and the method includes training the soft attention algorithm to improve the system's habit of correctly classifying input data.

10. The method according to claim 6, characterized in that, The method is executed by a computer system comprising one or more processors programmed with executable code that is not transiently stored in one or more memories.

Images