Method for classifying objects in images using a machine learning model

Abstract

Method of classifying an object in an image, comprising: providing a machine-learning model comprising a base classification dataset consisting of prototype vectors and corresponding labels S406 and a

Description

Field

[001] The present application generally relates to a method for using a machine learning, ML, model to process an image to classify an object in image so that there is controllable forgetting of known base classes and a method of personalising a ML model for such use. Background

[002] In recent years, more and more mobile devices and home appliances rely on deep models for computer vision tasks. For example, to build application for food recognition on mobile device, one could pre-train a small machine learning, ML, model on a fixed set of Western food dishes, and deploy on the mobile device. An application, powered by the pretrained ML model, running on the mobile device would be able to recognize the food dish if it falls in one of the pre-trained classes. However, if the user asks the application to recognize an Asian food dish, which has not been seen during pre-training, the model would fail. Thus, to maintain the Quality of Service (QoS), it is necessary to include continual learning capabilities, so that the user is able to annotate a novel dish, so that the model recognizes other instances of that dish in the future.

[003] The applicant has therefore identified the need for an improved way to perform personalisation of ML models on user-specific data. Summary

[004] In a first approach of the present techniques, there is provided a computer-implemented method for using a vision machine learning, ML, model to classify an object in the image, the method comprising: obtaining an image depicting an object; generating, using the vision ML model, a feature vector for the object in the image wherein the ML model comprises a base classification dataset which comprises a plurality of base classification labels and each base classification label is associated with at least one feature vector which represents a base prototype for the base classification label and wherein the ML model comprises a user-specific classification dataset which comprises at least one user-specific classification label and each user-specific classification label is associated with a feature vector which represents a user-specific prototype for the user-specific classification label; calculating a distance in feature space between the generated feature vector and each feature vector in a base classification dataset; identifying, from the base classification dataset, the base prototype having a feature vector which is closest to the generated feature vector; determining whether a distance between the feature vector for the identified base prototype and the generated feature vector is less than or equal to the pre-defined distance threshold, when the determined distance is less than or equal to the pre-defined distance threshold, outputting the base classification label of the identified base prototype as the classification label for the object in the image, and when the determined distance is greater than the predefined distance threshold, using the user-specific classification dataset to output a classification label for the object in the image.

[005] In other words, the base classification label will continue to be used, i.e. not forgotten, when the generated feature vector is sufficiently close to any of the feature vectors for the prototypes in the base classification dataset. Otherwise, the user-specific classification dataset may be used to generate the classification label. The user-specific classification dataset may be termed a second dataset or a novel / new classification dataset which is personal to the user of the user device. As explained below, the user-specific classification dataset may be generated as part of a personalisation of the ML model to the user. Such a personalisation generally takes place before the first query sample is received, to improve efficiency. The technology may be considered to be a logic switch in the inference stage which allows for control and flexibility between inference based on a base task or a novel task.

[006] In other words, determining whether a distance between the feature vector for the identified base prototype and the generated feature vector is greater than / less than or equal to a pre-defined distance threshold may be considered to be implementing a decision rule (also termed a branching rule) that controls whether any base classification labels within the base classification dataset (which may be termed a first dataset) are to be forgotten, i.e. replaced with a novel classification label. Overtime, base classification labels which have been replaced may be reinstated by the user as part of the user-specific classification dataset. The implementation of the decision rule is lightweight and thus the proposed techniques have high accuracy, low footprint (in terms of memory requirements) and personal data is used efficiently and safely.

[007] Using a user-specific classification dataset to output a classification label for the object in the image may comprise calculating a distance between the generated feature vector and each feature vector in the user-specific classification dataset; identifying, from the user -i.a . ^pred.ncd ■ specific classification dataset, the user-specific prototype having a feature vector which is closest to the generated feature vector and outputting a classification label based on the identified user-specific prototype as the classification label for the object in the image. In other words, the user-specific classification label may be used as the classification label.

[008] This decision rule may be termed a novel class detection rule expressed mathematically as: clnc, if DRa(fl; a; Bp) is True clbc, otherwise, where Cp“edncd is the output class for the input image; is the closest prototype from the user-specific classification dataset Np, clbc is the closest prototype from the base classification dataset Bp, a is the pre-defined distance threshold, f1 is the generated feature vector for the input image and DRa(fl; a; Bp^ = 1 ( min dist( / 1, protoc) >a) protoceBp where 1 is the indicator function, protoc is the prototype, dist( / Tprotoc) is a function for calculating distance between the two vectors and the other terms are defined above.

[009] In other words, we also describe a computer-implemented method for using a personalised machine learning, ML, model to recognise user-specific classes, the method comprising: obtaining an image depicting an object; generating, using the ML model, a feature vector for the object in the image; comparing the generated feature vector to: a first dataset, comprising a plurality of feature vectors for base class prototypes; and a second dataset, comprising at least one feature vector for a user-specific class prototype; determining whether the generated feature vector is closest to a feature vector from the first dataset or second dataset; and outputting a class name for the object in the image based on a class name of the feature vector from the first dataset or second dataset that the generated feature vector is closest to. The determining may comprise using a decision rule that controls whether the first dataset is to be forgotten. The decision rule may specify that first dataset is to be forgotten when the generated feature vector is far away from all the feature vectors in the first dataset. When the generated feature vector is far away from all the feature vectors in the first dataset, the method may comprise: determining whether the generated feature vector matches or is similar to the at least one feature vector from the second dataset.

[010] A feature vector is a numerical representation of the object and / or image (or part of the image). A feature vector is a n-dimensional vector of numerical features that represent the object. A feature vector may also be termed an embedding. The feature vectors for each image in the plurality of images form a vector space which may be termed a feature space. A prototype feature vector may be termed a representative feature vector for the class. The prototype feature vector may be a feature vector which is selected from the generated feature vectors or may be calculated from one or more combinations of the generated feature vectors. The distance may be calculated using any suitable technique e.g. cosine distance. Suitable techniques are well-known in the art and are described for example on Wikipedia, e.g. https: / 7erhwikspedja.org / ^ Using prototype feature vectors means that the samples (either base samples or user-specific / new samples) do not need to be stored. This reduces the resource required on the user device to personalise the ML model but maintain accuracy.

[011] The base prototypes in the base classification dataset may have been determined using any suitable technique. The base classification dataset may be termed an array or list of prototypes. There may be more than one base prototype for each base classification, i.e. there may be a set of base prototypes (for example one, two or three) for each base classification. Clustering may be used to group the feature vectors into groups which are in a similar location in vector space. The clustering may be done using any suitable technique, for example k-means clustering which is described for example in “An efficient k-means clustering algorithm” by Kanungo et al published in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2002) or spectral clustering which is described for example in "Spectral clustering based on local linear approximations” by Arias-Castro et al published in Electronic Journal of Statistics in 2011. The method of generating the set of prototype feature vectors may comprise calculating a centroid feature vector for each cluster; and setting the centroid feature vector as the representative feature vector. The centroid of a cluster may be defined as the arithmetic mean of all the data points in that cluster.

[012] Outputting a classification label based on the identified user-specific prototype may comprise outputting the user-specific classification label for the identified user-specific prototype as the classification label for the object in the image. Before outputting the userspecific classification label, there may be a check to make sure that this label is appropriate / accurate. The method may thus further comprise determining whether a distance between the feature vector for the identified user-specific prototype and the generated feature vector is greater than a second pre-defined distance threshold. When the distance between the feature vector for the identified user-specific prototype and the generated feature vector is less than or equal to the second pre-defined distance threshold, the method may comprise outputting the user-specific classification label for the identified user-specific prototype as the classification label for the object in the image.

[013] When the distance between the feature vector for the identified user-specific prototype and the generated feature vector is greater than the second pre-defined distance threshold, the method may comprise outputting a null classification label and / or an error message. Alternatively, any suitable method for determining that the generated feature vector does not correspond to a user-specific classification in the dataset may be used. For example, feedback from a user that the output classification label is not correct may be received. Whether the base classification label or the user-specific classification label is output, it is possible that the output classification label is incorrect, for example because the system cannot classify the object in the image. The user feedback may thus be received in response to an output userspecific classification label or an output base classification label.

[014] When it is determined that the output classification label is incorrect, e.g. when a null classification label is output, an error message is produced, or when feedback from a user that the output classification label is not correct is received, the method may further comprise obtaining user input specifying a user-specific classification label for the incorrectly labelled (or not recognised) image; and updating the user-specific classification dataset using the user input. In this way, the personalisation of the ML model may be a continually updating process. The base classification labels may be coarse classifications (e.g. pet, pizza) and the userspecific classification labels may be fine classifications (e.g. my pet, your pet, frozen pizza, fresh pizza). It will be appreciated that overtime, some of the original base classification may be forgotten (i.e. replaced with user-specific classification). However, these base classifications may be reintroduced by a user as a user-specific classification. In this way, a coarse classification label may also be output for a subsequent query image.

[015] The method may further comprise receiving the user-input classification label via a text input or a voice input. The user input may be received via any suitable user interface, e.g. a touch screen or a microphone. The user may be prompted to input a correct classification label for the object and this may comprise prompting the user to select the correct classification label from a list of the classification labels in the user-specific classification dataset or the base classification dataset.

[016] Updating the user-specific classification dataset may comprise determining whether the input user-specific classification label matches a user-specific classification label in the user-specific classification database. When the input user-specific classification label does not match a user-specific classification label, updating the user-specific classification dataset using the user input may comprise adding the input user-specific classification label to the user-specific classification dataset and associating the generated feature vector for the incorrectly labelled image with the newly added user-specific classification label in the userspecific classification dataset.

[017] When the input user-specific classification label matches a user-specific classification label in the user-specific classification database, updating the user-specific classification dataset using the user input may comprise updating the at least one feature vector which represents a user-specific prototype for the user-specific classification label using the generated feature vector for the incorrectly labelled image. Any suitable updating technique may be used, e.g. replacement and / or averaging.

[018] Whether adding a new classification label or adding a new feature vector to an existing class, the resource constraint on the user device may need to be considered. Updating the user-specific classification dataset may further comprise: determining whether a pre-defined memory limit for the user-specific classification dataset and base classification dataset will be exceeded when adding the feature vector and user-input classification label and when the predefined memory limit is determined to not be exceeded, updating the user-specific classification dataset. When the pre-defined memory limit is determined to be exceeded, the method may comprise: removing an existing feature vector from the user-specific classification dataset and / or the base classification dataset; prior to updating the user-specific classification dataset. Removing an existing feature vector may comprise removing a least-used feature vector or a feature vector that has not been used recently, e.g. within the last few days or weeks depending on the intended use of the apparatus. For determining the least-used feature vector, there may be a counter vector having a shape of Ixtotal number of prototypes. For each input image, if the image at the i-th counter in the vector is the closest prototype to the feature extracted from the input image, then the i-th counter is increased by one. When determining which is the least-used feature vector, the one having the lowest count value, e.g. min(counter) is selected.

[019] In other words, when the generated feature vector does not match or is not similar to the at least one feature vector from the second dataset, the method may comprise: outputting a null class name for the object in the image or an indication that the object does not match a user-specific class. In other cases, when the generated feature vector does not match or is not similar to the at least one feature vector from the second dataset, the method comprises: outputting a coarse class name for the object based on a coarse classification of the object.

[020] Personalisation of the ML model may also or alternatively take place before the ML model is used to classify an image. Such a personalisation of the ML model may be termed a fine-tuning or incremental training. The method may thus further comprise: prior to obtaining an image depicting an object: receiving, from a server, a base ML model comprising a finetuned backbone and the base classification dataset; obtaining one or more user-specific sample images each of which have been annotated with at least one user-specific classification label which is not included in the base classification dataset; and personalising the base ML model using the one or more annotated user-specific sample images, by: freezing the fine-tuned backbone; and generating the user-specific classification dataset using the one or more annotated user-specific sample images. The generated user-specific classification dataset is used as explained above to personalise the model. The output personalised ML model comprises the base ML model, the user-specific classification dataset and the base classification dataset.

[021] Generating the user-specific classification dataset may be done using any known technique. The user-specific classification dataset comprises at least one user-specific classification label each of which is associated with at least one feature vector which represents a user-specific prototype for the user-specific classification label. The at least one feature vector which represents a user-specific prototype may be termed a user-specific prototype vector. As for the base classification dataset, the prototypes may be generated using any known technique and there may be more than one prototype for each class. Clustering may be used to generate the prototypes. By focusing on prototype updates rather than model retraining, efficient personalization is achieved without the need for extensive data collection or computational resources.

[022] When personalising the ML model, obtaining one or more annotated user-specific sample images may comprise receiving a single sample image for each user-specific classification label which is not included in the base classification dataset. In other words, the personalisation or fine-tuning of the ML model may be a one-shot training. The one-shot approach may be used in the continual training described above. One-shot training may be reduce the training time which is required to personalise the model.

[023] When implementing the ML model with the branching rule, i.e. when implementing the ML model which uses the pre-defined distance threshold when deciding on classification labels, the accuracy of the ML model when recognising base classes is typically reduced when compared to implementing the ML model without the branching rule. The pre-defined distance threshold may be adjustable / selectable to meet the user requirements. This selection may be part of the personalisation process. However, the adjustments may take into account a maximum tolerable forgetting rate, i.e. a value which determines the extent to which base classes can be forgotten. The maximum tolerable forgetting rate may be considered to allocate a portion of the feature space for the potential future user-specific classes. The maximum tolerable forgetting rate may be received from a server. In other words, the maximum tolerable forgetting rate may be set externally from the user device so that the forgetting of the base classes is controlled externally and thus there is an external QoS which must be met. The method may further comprise setting (selecting or adjusting - the terms can be used interchangeably) the pre-defined distance threshold to reduce the general / base class accuracy and improve novel class accuracy but still meet the externally defined QoS.

[024] It will be appreciated that the personalisation of the ML model may be a standalone aspect. Thus, according to another aspect of the present techniques, there is provided a computer-implemented method for personalising a machine learning, ML model for use on a user device to classify an object in an image, the method comprising receiving, from a server, a base ML model comprising a fine-tuned backbone and a base classification dataset; obtaining one or more user-specific sample images each of which have been annotated with at least one user-specific classification label; and personalising the base ML model using the one or more annotated user-specific sample images, by: freezing the fine-tuned backbone of the received base ML model; generating a user-specific classification dataset using the one or more annotated user-specific sample images, setting a pre-defined distance threshold which is used by the ML model when classifying images to determine whether to output a classification label based on the base classification dataset or the user-specific classification dataset and wherein the personalised ML model comprises the base ML model, the userspecific classification dataset, the base classification dataset and the pre-defined distance threshold. The pre-defined distance threshold may be used as described above to compare calculated distances between base prototypes and a feature vector for a query image. The pre-defined distance threshold may be used to control forgetting of base classes (base classification labels) within the base classification dataset.

[025] One method for setting the pre-defined distance threshold may include selecting a value for the pre-defined distance threshold such that a first base class accuracy of the ML model with the branching rule is greater than or equal to a difference between a second base class accuracy of the ML model without the branching rule and the maximum tolerable forgetting rate. The second base class accuracy may be termed a base recognition accuracy of the ML model. The base recognition accuracy is a fixed value which may be stored for use in selecting value for the pre-defined distance threshold. By contrast, the first base class accuracy will depend on the value for the pre-defined distance threshold. The first base class accuracy may be termed an a-priori base recognition accuracy of the ML model because although the accuracy varies with the value for the pre-defined distance threshold, the accuracy is not dependent on the nature or number of the user-specific classes. In other words, adding more user-specific classes to the user-specific dataset does not alter the first base class accuracy. The first base class accuracy of the model may be reduced relative to the second base class accuracy by an amount which is less than the maximum tolerable forgetting rate which may be termed F.

[026] For example, the specific value for pre-defined distance threshold a (aseiected) can be set so that the following expression is met: ACC(M^ncd,X^st)>BCR-F where ACC(M“Tncd;X^t) is the first base class accuracy of the ML model (this may also be termed an a-priori accuracy of the ML model, X^ is the test dataset, BCR is the base recognition accuracy (i.e. the second base class accuracy) and F is the maximum tolerable forgetting rate, ncd represents the branching rule (which may also be termed the novel class detection rule), namely the determination as to whether the distance between the feature vector for the identified base prototype and the generated feature vector is greater than a predefined distance threshold and the subsequent determination of the output label. The use of a variable pre-defined distance threshold allows control of the quality of service (QoS) and may help to maximise personal task accuracy given the pre-defined QoS budget which may be defined by the maximum tolerable forgetting rate.

[027] Both base class accuracies may be determined using a test dataset comprising a plurality of images each labelled with a base classification label. The test dataset may comprise selecting a subset of the base classification dataset / domain specific dataset as the test dataset. The selected subset of the base classification dataset is preferably a different subset of the base classification dataset than a training dataset which is used to train the ML model.

[028] The controlling of the user device may be any suitable control based on the use of the user device. For example, when the user device is an oven, the oven may be set to a certain temperature for a certain period of time. When the user device s is a moveable robotic device, e.g. a lawnmower, vacuum cleaner etc, the movement of the user device may be controlled based on the classification, e.g. to avoid certain objects. Similarly, any output recommendation or recommended action may be appropriate for the user device, e.g. a suggested recipe for an oven. It will also be appreciated that the controlling and / or recommendation may be based on the correct classification label which is provided by the user when the output classification is incorrect.

[029] Outputting the classification label may comprise modifying the obtained image to include the output classification label and outputting the modified image. After the correct classification label has been received from the user, the modified image may be further modified to include the correct classification label.

[030] Generating, using the personalised vision ML model, a feature vector for the object in the image a classification label for the object in the image may further comprise: determining a bounding box for the object in the image; and generating a feature vector for contents of the determined bounding box. Outputting the classification label may comprise: modifying the obtained image to include the bounding box and the output classification label and outputting the modified image. The modified image may be a cropped image which is cropped to show the part of the image within the bounding box or may be a modified image in which the bounding box has been superimposed on the original image. A bounding box is a rectangle which is drawn round a region of interest, e.g. an object in the image. Determining a bounding box may be done using any suitable technique, for example by including an object detection network in the pre-trained vision ML model. An example of a suitable object detection network is Y0I0V8 described in “Ultralytics YOLOv8” by Jocher et al. published on Github in 2023.

[031] In another approach of the present techniques, there is provided an apparatus for using a personalised machine learning, ML, model to recognise user-specific classes, the apparatus comprising: a user interface; and at least one processor coupled to memory, for: obtaining an image depicting an object; generating, using the ML model, a feature vector for the object in the image; comparing the generated feature vector to: a first dataset, comprising a plurality of feature vectors for base class prototypes; and a second dataset, comprising at least one feature vector for a user-specific class prototype; determining whether the generated feature vector is closest to a feature vector from the first dataset or second dataset; and outputting, via the user interface, a class name for the object in the image based on a class name of the feature vector from the first dataset or second dataset that the generated feature vector is closest to.

[032] In other words, in this approach there is provided a user device for using a personalised machine learning, ML, model to process an image to classify an object in the image, the user device comprising: a user interface; and at least one processor coupled to memory, for: obtaining an image depicting an object; generating, using the personalised ML model, a feature vector for the object in the image; calculating a distance between the generated feature vector and each feature vector in a base classification dataset, wherein the base classification dataset comprises a plurality of base classification labels each of which is associated with at least one feature vector which represents a base prototype for the base classification label; identifying, from the base classification dataset, the base prototype having a feature vector which is closest to the generated feature vector; determining whether a distance between the feature vector for the identified base prototype and the generated feature vector is less than or equal to a pre-defined distance threshold; and when the determined distance is less than or equal to the pre-defined distance threshold, outputting the base classification label of the identified base prototype as the classification label for the object in the image; and when the determined distance is greater than the pre-defined distance threshold, using a user-specific classification dataset to output a classification label for the object in the image, wherein the user-specific classification dataset comprises at least one user-specific classification label which is associated with a feature vector which represents a user-specific prototype for the user-specific classification label.

[033] The features described above with respect to the first approach apply equally to the this approach, and therefore, for the sake of conciseness, are not repeated.

[034] The apparatus may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV). More generally, the apparatus may be a constrained-resource device, but which has the minimum hardware capabilities to implement the personalised model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart washing machine, smart oven, or smart lawn mower). It will be understood that this is a non-exhaustive and non-limiting list of example devices.

[035] For example, the method may enable a smart fridge to detect new objects which have been inserted by the user, while maintaining recognition accuracy for old objects. The smart fridge may use the classification to recognise when items are running low in the fridge and may thus output a recommendation to order more food or even automatically place an order. As another example, the method may enable a robot vacuum cleaner to detect new user items / pets and avoid them while cleaning (e.g. because some pets are scared). As another example, a smart oven may be able to recognise new cuisines / dishes inserted by the user and may recommend / initiate a correct cooking program. As another example, a smart washing machine may be able to recognise new clothes inserted by the user and may recommend / initiate a correct washing cycle. Thus, the user device may make a recommendation for control of the user device and / or control the user device based on the output classification.

[036] In another approach of the present techniques, there is provided a system for training a machine learning, ML, model to recognise user-specific classes, the system comprising: a server comprising at least one processor coupled to memory, for: obtaining an ML model with a pre-trained backbone; obtaining a domain-specific training dataset, comprising a plurality of images and a pre-defined list of base classes; training the pre-trained backbone using the domain-specific training dataset to generate a domain-specific model with a fine-tuned backbone; and at least one user device, wherein each user device comprises at least one processor coupled to memory, for: obtaining one or more annotated user-specific sample images; obtaining a copy of the domain-specific model; personalising the domain-specific model using the one or more annotated user-specific sample images, by: freezing the finetuned backbone; and training the domain-specific model using the one or more annotated user-specific sample images.

[037] In other words, according to this approach of the present techniques, there is provided a system for training a machine learning, ML, model to recognise user-specific classes, the system comprising: a server comprising at least one processor coupled to memory, for: obtaining an ML model with a pre-trained backbone; obtaining a domain-specific training dataset, comprising a plurality of images each labelled with at least one base classification label; training the pre-trained backbone using the domain-specific training dataset to generate a base ML model with a fine-tuned backbone and a base classification dataset comprising a plurality of base classification labels each of which is associated with at least one feature vector which represents a base prototype for the base classification label; and at least one user device, wherein each user device comprises at least one processor coupled to memory, for: obtaining one or more annotated user-specific sample images each of which have been annotated with at least one user-specific classification label which is not included in the first dataset; obtaining a copy of the base ML model; personalising the base ML model using the one or more annotated user-specific sample images, by: freezing the fine-tuned backbone; and generating a user-specific classification dataset using the one or more annotated userspecific sample images, wherein the user-specific classification dataset comprises at least one user-specific classification label which is associated with at least one feature vector which represents a user-specific prototype for the user-specific classification label.

[038] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[039] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[040] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[041] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object-oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[042] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[043] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general-purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[044] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[045] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[046] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[047] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[048] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[049] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[050] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[051] Figure 1 is a flowchart of the steps during a base training session according to the present techniques;

[052] Figure 2 is a flowchart of the steps during an incremental learning session according to the present techniques;

[053] Figure 3 illustrates the testing and training samples which are used both in the base training session and the incremental training sessions of Figures 1 and 2;

[054] Figure 4 is a flowchart of the steps during inference using controllable forgetting according to the present techniques;

[055] Figure 5 is a flowchart of the steps for implementing a specific type of decision rule which may be implemented in the method of Figure 4;

[056] Figure 6 is a flowchart of the steps for determining a value for the pre-defined threshold rate for use in the decision rule of Figure 5;

[057] Figure 7 is a table showing NCR comparison between vanilla inference method (V-NCR) and inference based on the present NCD rule;

[058] Figures 8A and 8B show graphs from experiments comparing two inference methods;

[059] Figure 9 is a flowchart showing more detail of the novel class branch of Figure 4;

[060] Figure 10A is a flowchart showing continual training of the ML model;

[061] Figures 10B to 10D are example images which may be input or generated during the process of Figure 10A and

[062] Figure 11 is a block diagram of a system for implementing the present techniques. Detailed description of the drawings

[063] Broadly speaking, embodiments of the present techniques provide a method and apparatus for using a personalised machine learning, ML, model to process an image to classify an object in image so that there is controllable forgetting of known base classes and a method of personalising a ML model for such use. The controllable forgetting may be achieved by using a decision rule to branch between base classes and novel (user-specific) classes. Further control may be achieved by adjusting / setting a threshold which controls the decision rule.

[064] The present techniques focus on a set-up that is known in literature as Few-Shot Class-Incremental Learning (FSCIL). Formally, FSCIL consists of a base training session and several incremental training sessions. Figure 1 shows an example of the steps during the base training session using a training dataset which typically comprises a large number of samples from base classes (e.g. western food dishes). These steps are normally undertaken on a server or similar remote apparatus.. After that, the model is deployed on device, and may optionally be quantized or compressed before deployment. This is useful to reduce the computational burden of the ML model, particularly for low resource devices. Any suitable methods for quantization or compression may be used for example as described in “A new frontier of Al: On-device training and Personalization” by Moon et al. published in IEEE / ACM 46th International conference on software engineering: Software engineering in practice in 2024. Figure 2 shows an example of the steps on the user device, where the model continually observes a few annotated samples from novel classes (e.g. asian food dishes), and adapts the model to increase the accuracy of novel class recognition. Base training session

[065] In Figure 1, at step S100, a training dataset X^nis obtained. This step represnts an input. This training dataset comprises a plurality of images which have been labelled with at least one base classification c (each classification may indicate a class of an object within the image. The training dataset X^™ may be considered to contain domain-specific data and the domain may represent an intended use of the apparatus. Merely as an example, a suitable set of training data for food is Food-101 which is described for example in “Food-101 - mining discriminative components with random forests” by Bossard et al. published in Computer vision ECCV 2014. Food 101 comprises 101 different classes of objects in the form of different types of food. The domain “food” will be useful for apparatus such as fridges or ovens. As another example, when the apparatus is a smart appliance such as a lawn mower, the training data may be labelled images of flowers such Flowers-102 which is described in “Automated flower classification over a large number of classes” by Nilsback et al published in 2008 in Sixth Indian Conference on computer vision. Flowers-102 comprises 102 different classes of objects in the form of different types of flowers.

[066] At step S102, the initial ML model is updated / trained using the training dataset X^in. The ML model which is trained during the base training session may be any suitable model such as MobileNet-V2 which is described for example in “MobileNetv2: Inverted residuals and linear bottlenecks” by Sandler et al published in European Conference on Computer Vision in 2022, or DINO-v2 which is described for example in “Dinov2: Learning robust visual features without supervision” by Oquab et al. published in arXiv:2304.07193. The initial ML model may be termed Mma- Prior to the base training, the initial ML model may have been pretrained on even larger datasets.

[067] The ML model may be updated in any suitable way, for example by calculating any suitable loss at step S104 and backpropagating the loss into the initial ML model to adapt the parameters to minimise a loss between the prediction from the ML model and the label on the training data. As explained in more detail below, the inference stage is agnostic to the choice of the training procedure during this base training session. To evaluate effectiveness of the present inference method, it is applied on top of two base training procedures: the popular ProtoNet training which is described in “Prototypical networks for few-short learning” by Snell et al published in Advances in neural information processing systems and the state-of-the-art SAVC training described for example in “Semantic-aware knowledge distillation for few-shot class incremental learning” by Song et al published in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition in 2023. In knowledge distillation, a larger ML model may be used to train the ML model which is to be deployed on the user device wherein the larger ML model may have been pretrained. Merely as an example, a suitable larger ML model may be the MobileNet-V2 and a suitable smaller ML model may be DINO-v2.

[068] As shown in Figure 1, there is a step of calculating base class prototypes S106 using the trained model to generate at least one feature vector. This technique is used in both methods referenced above, as well as most current FSCIL methods. A prototype feature vector may be termed a representative feature vector for the class. A feature vector is a numerical representation of the object and / or image (or part of the image). A feature vector is a n-dimensional vector of numerical features that represent the object. A feature vector may also be termed an embedding. The feature vectors for each image in the plurality of images form a vector space which may be termed a feature space. The prototype feature vector may be a feature vector which is selected from the generated feature vectors or may be calculated from one or more combinations of the generated feature vectors. A prototype is a feature vector which represents the class and may be determined as a centroid of the feature vectors for a given class: protoc: = Avgj(M(x‘)), where M is a backbone model, and %' are annotated samples from class c.

[069] As shown, the protoypes are used in the determination of the loss in step S104. While ProtoNet employs prototypical loss, which is more effective and robust for few-shot learning applications, SAVC uses contrastive learning and augmented base classes in the base training session to effectively partition the feature space. Each of steps S102, S104 and S106 results in an update to the model backbone, the set of base case prototypes (which may be termed the base classification database) or the calculated loss. After base training stage, the base class prototypes are stored in memory for use during the inference stage.

[070] Incremental training session

[071] Figure 2 shows the steps in the incremental training session. During the incremental training stage, at step S200 a set of novel class prototypes rt.'\s obtained from few-shot novel class support samples. Following recent FSCIL works such as SAVC and / or FACT techniques, the backbone is frozen during incremental stage at step S202 to avoid uncontrollable catastrophic forgetting of base classes. FACT is described for example in “Forward compatible few-shot class-incremental learning” by Zhou et al published in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition in 2022. At step S204, there is a step of calculating novel class prototypes S204 using the novel class prototypes t and the frozen model. The prototypes may be generated as described above in relation to the base training. In Figure 2, the novel class prototypes X^°p t represent an input module, the backbone represents a frozen module and the calculation of the novel class prototypes is an update module.

[072] Figures 1 and 2 are thus block diagrams of a setup for K-shot Class-Incremental Learning. The training consists of a base training session (usually done on server and shown in Figure 1), and an incremental training session with a few annotated (support) samples from novel classes (usually done on a user device). Most FSCIL solutions consider incremental sessions with 5-10 classes in each session, and with 5 samples (or shots) in each class. However, this setting is often unrealistic in real-world applications where users may not be willing to provide 5 annotated samples for each novel class. Instead, the present techniques operate in the more challenging One-Shot Class-Incremental Learning (OSCIL) setup, in which each incremental session contains a single annotated sample from a number of novel classes. The focus of the present techniques is thus on the One-Shot case (K = 1), but it will be understood that the present techniques also apply to few-shot cases.

[073] Another significant challenge in Class-Incremental Continual Learning is maintaining base class recognition during incremental training sessions or updates. While fine-tuning the model on novel samples typically increases novel class accuracy, it often leads to a significant drop in base class performance or base class accuracy. This is a well-known phenomenon called catastrophic forgetting. This is especially true for low-shot settings and on low-resource devices, because retaining base class samples on the device is impractical, and one novel sample (i.e. a single novel sample) is not enough to adapt / fine-tune parts of the network with backpropagation.

[074] As an illustrative example of catastrophic forgetting, consider a pre-trained Al model, which is trained on a server to recognise food classes and then deployed to many user devices. The Al model is able to recognise a cantaloupe melon, because the model was pretrained on images depicting cantaloupe melons (among other things). However, the pretrained Al model is unable to recognise a watermelon, i.e. a novel class that it has not seen before during pre-training. To personalise the pre-trained Al model to recognise the novel class, the user provides one input image (sample) for the watermelon class. However, existing techniques enable the Al model to be personalised using novel classes, but they then lose accuracy in relation to previous / known classes. When catastrophic forgetting occurs, the personalised Al model is now able to recognise the novel class (watermelon), but it can no longer recognise the cantaloupe melon. This is unacceptable since it leads to a lower Quality of Service (QoS) for a user.

[075] Methodology

[076] The present solution for One-Shot Class-Incremental Recognition with Controllable Forgetting is now explained. A flowchart setting out the key steps of the method is shown in Figure 4. Before considering the steps of Figure 4 in detail, some terminology in the context of the present techniques is set out below.

[077] Few-Shot Class-Incremental Recognition

[078] As explained above and shown in Figure 1, the Few-Shot Class-Incremental Recognition pipeline usually starts with initial backbone model Minit, which might be pretrained on larger datasets (e.g. on lmageNet-1k for ResNet models) with cross-entropy or with selfsupervised contrastive loss (e.g. DINO visual transformer model). The model is then trained on domain-specific data with a pre-defined list of base classes, with many samples (» 100) available for each class. This is called a base training session which yields domain-finetuned model Mbt. The base class train split is denoted as X^in, test split as X^, and the number of base classes is denoted as No. This training is normally carried out on a server.

[079] Figure 2 shows the steps on the user device after the model is transferred to the personal device. The model is presented with a few annotated samples of novel classes, which have not been seen during the base training session of Figure 1. This leads to a continual stream of incremental training sessions, during which the model adapts to new data, yielding the final personalized model MIT.

[080] Figure 3 illustrates the testing and training samples which are used both in the base training session and the incremental training session. As set out above, the base class train split is denoted as and the test split as X^, and the number of base classes is denoted as No. For a training session in the incremental training session, s >0, the annotated (or support) samples from novel classes are denoted as X^pport, while the test (or query) samples from novel classes are denoted as x£>, the number of novel classes is denoted as and the number of support samples (or shots) per class is denoted as K. Without loss of generality, the present techniques focus on a case with a single incremental session s = 1. This means all the annotated samples can be combined into a single set X^ rt, and incremental training can be provided on this set. Similarly, the test (or query) samples from novel classes are denoted as X^ery.

[081] The main challenge in FSCIL is to balance between adapting to novel classes and retaining the knowledge about the base classes during the Incremental training stage. Overly fitting the model to the novel classes leads to forgetting of the base classes. Specifically, the present techniques focus on three metrics: Base class recognition (which is the accuracy on the test split of base classes after the base training session): BCR: = ACC(MBT;X^t); Novel class recognition (which is the accuracy on the novel class query samples after incremental training): NCR: = ACC(M / T;XSry); Base class forgetting rate (which is the decline in base class accuracy due to learning new classes): FOR: = BCR - ACC(M / r; X®)

[082] The present techniques focus on extreme low-data mode, i.e. the one-shot scenario, which more closely resembles the real-world applications. The one-shot scenario is less researched, since most current methods focus on 5- or 10-shot evaluation.

[083] Decision Rule for Inference

[084] During inference in a standard (vanilla) inference stage such as that described in “Distance-based image classification: generalizing to new classes at near-zero cost” by Mensink et al published in IEEE Transactions on Pattern Analysis and Machine Intelliegence in 2021, a query sample xq is assigned to the closest class prototype among base and novel classes as predicted class: cpred,van = argminc(dist( / % protoc)), where fq-. = M(xq) is the feature vector of a query sample xq, M is a backbone model, protoc are memorized or stored prototypes from base and novel classes {Bp u N^, and dist(-) is a distance function in the feature space (usually cosine distance).

[085] A big downside of this inference method for novel classes with only 1 support sample is very high variability of support samples in the novel class feature space. While with 5 shots the estimated novel class centroid is close to the true centroid, with a single shot the estimated class centroid will likely be far away from the true centroid. Figure 4 shows a method for alleviating this problem by reducing the dependency of noisy support samples on the final inference and relying more on the stable base class centroids.

[086] In a first step S400, a query sample xq is received. In step S402, the query is input to the personalized model MIT as output from the incremental training of Figure 2. A feature vector fq is generated at step S404 for the query sample xq using the personalized model Mit. The feature vector may be for the whole image or for a bounding box within the image. Determining a bounding box may be done using any suitable technique, for example by including an object detection network in the pre-trained vision ML model. An example of a suitable object detection network is Y0I0V8 described in “Ultralytics YOLOv8” by Jocher et al. published on Github in 2023. When training the ML, model, the object detection network may be added and the training may be as described above.

[087] The present techniques introduce the application of a decision rule to the Inference algorithm at step S410. The decision rule is a boolean function DR(fJ) also termed DRncd and controls logic flow for the inference. In general, a decision rule is based on both base class prototypes Bp and novel class prototypes Np as illustrated at steps S406 and S408, respectively, The decision rule can have different designs. The resulting output is based on the decision rule branch which is followed. As shown at step S412, a classification label corresponding to the best base class c^c may be output. Alternatively, as shown at step S414, a classification label corresponding to the best novel class c^c may be output.

[088] Figure 5 shows the steps of the specific type of decision rule which may be chosen for one-shot task and which may be termed a Novel Class Detection Rule. At step S500, a distance between the generated feature vector fq and each base prototype feature vector is calculated. At step S502, the closest base prototype feature vector is determined. At step S504, there is a decision as to whether the determined distance for the identfied base prototype feature vector is greater than a pre-defined distance threshold a. The threshold a may be termed a hyperparameter. In other words, the decision rule may be expressed as DRa(fq; a; Bp) = 1( min dist( / Q,protoc) >a) v pj protocEBp where 1 is the indicator function, protoc is a prototype vector from the set of base prototype vectors Bp, and fq is the generated feature vector for the query sample.

[089] It is decided that a sample belongs to a novel class if its feature vector is far away from all base class prototypes. In other words, if the distance is greater than the pre-defined distance threshold a, the output classification is based on the best novel class as shown at step S514 and if not, the best base class is output at step S508. In other words, the decision selects between the novel class branch and the base class branch. The best novel class may be determined by calculating a distance between the generated feature vector fq and each novel prototype feature vector at step S510. At step S512, the closest novel prototype feature vector is determined and it is the classification label associated with this closest novel prototype feature vector which is output at step S514.

[090] In other words, the resulting predicted class is based on the Decision Rule branch: c^c, if DRa(fl; a; Bp)isTrue clbc, otherwise, where c^edncd is the output class for the input image; c^c is the closest prototype from the user-specific classification dataset Np, cbc is the closest prototype from the base classification dataset Bp, a is the pre-defined distance threshold, fq is the generated feature vector for the input image.

[091] Controllable Forgetting

[092] Denoting M"Tdr the model with Novel Class Detection Rule and distance threshold a, it is noted that it possible to calculate the accuracy of the model on base class samples without the knowledge of the novel class samples, since the Decision Rule does not take the personal samples into account. Figure 6 shows the steps of one way which may be used to select the specific value for a (aselected). It will be appreciated that a can be selected in any appropriate way. In a first step S600, the test dataset X^ is obtained. The test comprises a plurality of images each labelled with a base classification label. In a next step S602, the base class recognition accuracy may be determined, for example using: a,a ■ pred,ncd' BCR: = ACC(MBr;X™t) In other words, the base class recognition accuracy uses the same test data and is the accuracy of the original base model as received by the user device, i.e. before incremental training. The base class recognition accuracy also does not use the pre-defined distance threshold to determine whether or not to use the base classification label. In other words, the base class recognition accuracy is a fixed value for each set of test data.

[093] At step S604, a maximum tolerable forgetting rate on the base classes; F, is received. This maximum tolerable forgetting rate is decided by the system provider for each specific application / use of the device. For example, F = 2 or F = 5 are values which have been used in the experiments. It will be appreciated that although the steps S600 to S604 are shown sequentially, they may be carried out in any order or simultaneously. Subtracting the maximum tolerable forgetting rate from the base class recognition accuracy is a value which defines the overall base class recognition accuracy when implementing the branching rule.

[094] At step S606, a specific value for the pre-defined distance threshold a (aseiected) is selected. Using the test dataset, the personalised model with the specific value of a, an base class accuracy of the personalised ML model (i.e. the ML model using the branching rule with the selected value of a) may then be determined as shown at step S608. This base class accuracy may be termed an a-priori accuracy, because the accuracy is determined without knowledge of the personal samples. In other words, adding more user-specific classes to the user-specific dataset does not alter the a-priori accuracy. The a-priori accuracy may be termed ACC^^X^

[095] The next step S610 is to check whether aseiected is a value of a such that: ACC(MfT,ncd^ If not, the method loops back to try another value of a. If the accuracy requirement above is met, at step S612, there is a determination as to whether there are other values of a to try. For example, aseiected could be empirically selected over a grid search of values. If there are no other values of a to try or if the a otherwise meets the requirements, at step S614, the value of a which is to be used for the personalised model is output for use in future inference on device.

[096] It is possible to calculate the accuracy ACC(M“Tdr; X^t) a-priori before using the model on device. This is because the a-priori accuracy needs knowledge of the branching rule and the base model but does not need knowledge of the user-specific classes. This is because the maximum tolerable forgetting rate may be considered to allocate a portion of the feature space for the potential future user-specific classes. The output value of a can then be used to calculate the forgetting rate FOR. FOR: = BCR - ACC(M / r; X^) It will be appreciated that by setting the appropriate distance threshold a as described in Figure 6 based on the pre-defined forgetting budget for base classes, the method provides controllable forgetting, in which the base class recognition accuracy will always be within the forgetting budget, regardless of encountered novel samples.

[097] Experiments

[098] Backbone Models and Datasets

[099] Backbone Models. The effectiveness of the Novel Class Detection rule is evaluated using three backbone architectures each having increasing complexity to account for different resource budgets during deployment. The three architectures are: MobileNet, ResNet18 and DINOv2s. MobileNet-V2 is described for example in “MobileNetv2: Inverted residuals and linear bottlenecks” by Sandler et al published in European Conference on Computer Vision in 2022; ResNet18 is described for example in “Deep residual learning for image recognition” by He et al published in Proceedings of the IEEE Conference on computer vision and pattern recognition in 2016; and DINO-v2 is described for example in “Dinov2: Learning robust visual features without supervision” by Oquab et al. published in arXiv:2304.07193

[0100] During a base training session, each model is initialised from a pre-trained checkpoint. For example MObileNetV2 and ResNet18 are pre-trained on the lmageNet-1k described for example in “ImageNet Large Scale Visual Recognition Challenge” by Russakovsky et al published in International Journal of Computer Vision in 2015 and DINOv2 is pre-trained on a collection of multiple training datasets. For base training, either ProtoNet loss or other methods such as SAVC, FACT or OrCo are used. ProtoNet training is described in “Prototypical networks for few-short learning” by Snell et al published in Advances in neural information processing systems and SAVC training is described for example in “Semantic-aware knowledge distillation for few-shot class incremental learning” by Song et al published in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition in 2023. FACT is described for example in “Forward compatible few-shot class-incremental learning” by Zhou et al published in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition in 2022 and OrCo is described for example in “Oreo: Towards better generalization via orthogonality and contrast for few-shot class-incremental learning” by Ahmed et al published in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition in 2024. The resulting backbones are denoted as MobileNet-PN, MobileNet-SAVC, ResNet18-PN, ResNet18-SAVC, ResNet18-OrCo, ResNet18-FACT and DINOv2s-PN, respectively. A non-adapted DINOv2s is also included with a checkpoint taken from initial contrastive learning pre-training on large vision dataset denoted as DINOv2s-init. The checkpoint with the best validation accuracy on the base classes after fine-tuning with a slow learning rate is selected.

[0101] Evaluation datasets. The evaluation datasets are CLIB200, a common FSCIL finegrained dataset, CIFAR100 and CORe50 chosen specifically to evaluate DINOv2 model on a dataset not seen during self-supervised training. CLIB200 is described for example in “The caltech-ucsd birds 200-2011 dataset” by Wah et al published in 2011. CIFAR100 is described for example in “Learning multiple layers of features from tiny images” by Krizhevsky et al published in 2009. CORe50 is described for example in “Core50: a new dataset and benchmark for continuous object recognition” by Lomonaco et al published in Conference on robot learning in 2017. For each dataset No fixed base classes were chosen, and the remaining classes were used as possible novel classes during the incremental training sessions. In CLIB200, No = 100, CIFAR100, No = 50, and in CORe50, / V0 = 40.

[0102] Average metrics are computed over 25 evaluation episodes. To perform each single episode of few-shot evaluation, N1 novel classes are randomly subsampled, and a single sample in each novel class is randomly chosen as the support sample, while the other samples serve as query samples. The chosen novel classes and support samples are then used in few-shot evaluation. Results for two ultra-low data scenarios are presented: ^ = 1 (one novel class) or N1 = 5 (five novel classes), focussing on the challenging one-shot setting (K=1), where only one support sample is provided for each novel class.

[0103] Evaluation Metrics

[0104] The evaluation metrics are BCR, NCR and FOR as set out above. Accuracy of the proposed decision rule of Figure 5 is compared to the baseline vanilla inference method described above.

[0105] For base class recognition accuracy, BCR scores are included using simple nearest centroid method for base classes. The BCR metric is the same for both inference methods.

[0106] For the vanilla inference method, the NCR metric (denoted by V-NCR in Figure 7) is included. The FOR metric is not included, since it is negligible yet uncontrolled for the frozen backbone during incremental training stage.

[0107] For inference with the present decision rule (i.e. according to the present techniques), the distance threshold a is selected depending on the bearable forgetting budget for the target application as described above for Figure 6. For example, for a = 0, all incoming samples are detected as novel, resulting in a high NCR, but complete forgetting of the base classes (FOR = BCR). On the other hand, a big a level would result in 0% FOR but also 0% NCR.

[0108] To mimic a practical application where it is possible to trade some BCR for increased NCR, two levels of forgetting budget are chosen: FOR = 2% and FOR = 5%. The a values corresponding to those two levels of forgetting are found, and two NCR metrics are reported, denoted in Figures 8a and 8b as NCR@2FOR and NCR@5FOR, respectively.

[0109] Main Results

[0110] Figure 7 is a table showing the NCR comparison between vanilla inference method (V-NCR) and inference based on the present techniques (NCR@2FOR and NCR@5F0R). The comparison is done across various backbones and datasets. The effectiveness of the proposed techiques, including the decision rule, show some insight into organization and evolution of the feature space for different base training methods.

[0111] As shown by Figure 7, NCR accuracy improves greatly when applied on top of simple ProtoNet base training. For MobileNet-PN and ResNet18-PN backbones with 1 novel class on the CLIB200 dataset, NCR is improved by 9.3-10.8% for the price of 2% FOR, and by 2731.3% for the price of 5% FOR. With 5 novel classes on the CLIB200 dataset with the same backbones, NCR is improved by 10-16.1% for the price of 5% forgetting. Similar gains are also achieved by ResNet18-PN on the CIFAR100 dataset.

[0112] Intuitively, ProtoNet training on base classes with slow learning rate gradually deforms the feature space to cluster the base class samples together. In the process, the feature space corresponding to novel classes becomes more deformed, so the one-shot prototypes from those classes are more separated from the actual novel class centroid. The decision rule was designed to reduce the dependency of the inference result on the choice of the support sample in the novel class. While NCR@xFOR for = 1 measures pure out-of-distribution capabilities in the feature space (i.e. how well are the base class clusters separated from any novel samples), the same metric for ^ = 5 also measures the separability of the novel classes between each other.

[0113] The results of MobileNetv2-PN and ResNet18-PN indicate that there is a lot of accuracy gain for = 1, but for = 5, the gains are much smaller since the discrimination of those models between unseen classes is worse compared to the contrastive learning-based training methods.

[0114] A better separation of the feature space achieved during pre-training by using SAVC, OrCo and FACT allows these methods to reach higher results in terms of NCR than ProtoNet based pre-training. While vanilla NCR shows good results when applied to SOTA methods, the proposed techniques can still improve the performance of novel classes, but for a higher FOR price.

[0115] For example, in ResNet18-SAVC trained for FSCIL task, the NCR gains are lower but still notable for CLIB200: 22.3% gain for 1 novel class, and 13.5% gain for 5 novel classes for the price of 5% forgetting, with similar gains for MobileNetv2-SAVC backbone. This suggests superior novel class separation capabilities of backbones trained with contrastive learning (e.g. in SAVC and FACT) and orthogonality promoting terms (e.g. in OrCo).

[0116] Notably, the proposed decision rule helps a lot with DINOv2 transformer architecture, with NCR gains on CORe50 dataset of 18.7% and 26.2% for 1 novel class at a price of 2% and 5% forgetting, respectively.

[0117] Comparing DINOv2-PN training with MobileNetv2-PN and ResNet18-PN, it is clear that the initial checkpoint makes a big difference because DINOv2-PN was trained via contrastive learning on a large vision dataset, already starting with a highly separable feature space for classes. The clustering in DINOv2 feature space is also well-transferable to never-seen-before classes of CORe50, as seen in DI NOv2-init metrics. Starting from that checkpoint, ProtoNet training with small learning rate increases base class recognition rate, but also improves out-of-distribution capabilities (as seen in the ^ = 1 case) as well as keeping the novel classes separated (as seen in the ^ = 5 case).

[0118] Ablations

[0119] Figures 8a and 8b show graphs from experiments comparing two inference methods for a varying number of novel class numbers N1 and shot numbers K. Figure 8a is the NCR accuracy for increasing number of novel classes, and single shot K = 1. Figure 8b is the NCR accuracy for increasing number of shots and the number of novel classes ^ = 5. Experiments are done on ResNet18-PN backbone with CLIB200 dataset.

[0120] The effectiveness of the proposed techniques for a decision rule is measured for varying numbers of novel classes N± and numbers of shots K. As shown by Figure 8a, it is possible to improve NCR accuracy considerably for up to 20 novel classes for one-shot recognition.

[0121] However, the proposed techniques for a decision rule with 5% forgetting performs the same or even worse than vanilla inference when the number of shots is increased. In practice, it could be useful to switch to vanilla inference mode when three or more samples have been collected for novel classes. Moreover, as explained above, the controllable forgetting rate can be adjusted on device as described above depending on the forgetting strategy.

[0122] In the method shown in Figures 4 and 5, the decision rule (e.g. distance >threshold) acts as a logic switch to select between the base class branch and the novel class branch. During inference, the model may encounter objects in images which do not belong to the personal classes (or the base classes), but which need to be detected nevertheless. Figure 9 is a flowchart showing the adaptation of the methods of Figures 4 and 5 to this scenario.

[0123] In a first step S900, a query image xq is received at a user device. For example, on a smartphone, the query image may for example depict a dog (Bruno), for which there is no personal label even though there are personal / user-specific labels for other dogs such as “Archie”. As another example, on a smart oven or a smart fridge, the input image may be an image of a food which has not been seen previously. As in Figure 4, the query image xq is input into the personalized backbone of the ML model at step S902 and the feature vector for the query image is generated. As before the decision rule is applied and thus there is a determination as to whether the distance between the feature vector and all the prototype vectors for the base classes is greater than the threshold. In this example, the distance is greater and thus the method has switched to the novel class branch. For simplicity the detail of the base class branch is omitted.

[0124] At step S908, the distance between the feature vector and each of the feature vectors for the novel classes is calculated. The novel prototype vector with the closest feature vector may be identified at step S910. This identified novel prototype vector may be termed a personal class confidence vector because it is the vector having the highest confidence value. There is a comparison of the distance between the identified novel prototype vector and the feature vector to a second pre-defined threshold at step S912. This second pre-defined distance threshold may be termed a personal class threshold and like the first pre-defined distance threshold may be set externally by a system provider. When the distance is below the threshold, the classification label corresponding to the identified prototype may be output as shown at step S916. However, when the distance is too high, a coarse label may instead be applied to the image during inference. This mechanism is referred to as “fall back to coarse classes”. The coarse label may be applied to the sample if the distance to known prototypes exceeds the threshold. Merely as an example, for the smartphone example the coarse label may be “dog”.

[0125] As set out above, the present techniques are well-adapted to the one-shot scenario. Thus, as an alternative to outputting a coarse classification, the user device may output an error message. Alternatively, the user may provide input that the output classification, whether the base classification or novel classification is incorrect. Figure 10A shows how the model can be further updated. As shown in Figure 10A, the further updating is triggered by either outputting an error message as shown at step S1000 or receiving user feedback that the output classification is incorrect at step S1002. For example, as shown in Figure 10B, an example of an image which has triggered the need for updating is shown. The image is an onion. As shown in Figure 10C, the output classification label is “apple” which is clearly incorrect and thus the process of Figure 10A is triggered.

[0126] There is an optional output request for the user to input a correct classification at step S1004. It will be appreciated that this step can be omitted. At step S1006, the user device then receives a correct classification label for the image. For example, in Figure 10C, the correct classification label is “onion”.

[0127] The model is then updated in a similar manner to the on-device incremental training described above. In particular, the user-specific dataset is updated at step S1008. This means that the new classification label (onion) is added to the dataset together with a corresponding prototype feature vector, e.g. the feature vector which was generated when processing the image (this may be feature vector for the whole image of Figure 10B / or just the feature vector for the bounding box of Figure 10C). At step S1010, another query image which includes an object in the same class as the new class is then received. This new query image is processed as described above at step S1012 to generate a feature vector which is then compared to all prototype feature vectors for the base classes. As before the distance threshold will be exceeded but in this case, the new user-specific class will be output as the classification label as shown at step S1014. An example output is shown in Figure 10D, in which the output is a modified version of the query image showing a bounding box (applied using known techniques) and the correct new classification label “onion”.

[0128] Merely as an example, the present techniques enable class-incremental recipe recognition for a smart oven. A user may input an image or other data item representing a recipe or food item to be cooked. If this image is one the trained model has not previously seen, recognition of the recipe / food item will fail and an error message is output. The user can then annotate the image so that the trained model is able to recognise the recipe / food item in the future. This may ensure the smart oven chooses the right cooking settings for the recipe / food item and can monitor the cooking (using one or more sensors, cameras, etc).

[0129] Similarly, the present techniques enable class-incremental recipe recognition for a smart fridge. A user may add an unknown grocery item to a smart fridge (for example an onion). The smart fridge may have at least one camera to capture images of items put into and taken out of the smart fridge. The smart fridge may not be able to recognise the unknown grocery item in one or more images captured of the item. Thus, the user may annotate the new grocery item so that the model can recognise this item in the future.

[0130] The present techniques may also be used for personalised pet recognition. For example, during the incremental training of Figure 2, the user annotates a few images of their dog (“Archie” or “Atlas”) as support images and the trained model learns to recognise their dog in the future. New images can then be analysed to determine whether they depict the user-specific classes (e.g. “Archie”, or “Atlas”). When the new images do not show these userspecific classes, the output may indicate that the images do not depict the user’s dogs. That is, in one example, if the image does not contain the user’s dog Archie, the model may simply not output a label or may say that Archie has not been identified in the image. As an alternative, as described above, a fall-back mechanism to a coarse class may be used. That is, when the trained model is shown an image that does not contain the user’s dog Archie, instead of not outputting a label or saying that Archie has not been identified in the image, the trained model may output a coarse class instead, such as “dog”. In other words, when the trained model is shown images of other dogs (Bruno, Luna), the trained model may output a coarse label “dog” as the personal label (“Archie”) does not apply.

[0131] Figure 11 illustrates an example system which may be used to implement the new techniques described above. The system comprises a server 1000 which may be a single server or collection of servers (e.g. the cloud). The server 1000 comprises at least one processor 1002 coupled to memory 1004. The at least one processor 1002 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The at least one processor 1002 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The memory 1004 may comprise volatile memory, such as random-access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example.

[0132] A base model (ML model) 1006 may be stored on the server 1000, e.g. during the training described above. The base model (ML model) 1006 may be a vision ML model which is used to generate feature vectors from images. The server 1000 also comprises an input / output interface 408 (or similar communication module) which connects the device to a database 1010. The database 1010 may comprise training dataset(s) for training the ML model. The database 1010 may also comprise additional ML models from which the base ML model may be selected as described above. The server may comprise a clustering module 101 for generating the base prototypes for the base classification dataset 1007 and the base classification dataset 1007 may be stored on the server 1000. The server 1000 is also coupled to at least one apparatus / user device 1020.

[0133] The user device 1020 also comprises similar standard components to the server 1000. The user device 1020 comprises at least one processor 1022 coupled to memory 1024. The at least one processor 1022 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The at least one processor 1022 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The memory 1024 may comprise volatile memory, such as random-access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example. The personalized model (ML model) 1026 is stored on the electronic device 1020 together with the base classification dataset 1029 and the user-specific classification dataset 1027. User data 1030 which is used to personalize the model, e.g. to create the user-specific classification dataset may be stored on the user device and for security reasons may not leave the user device. The pre-defined distance threshold and / or the forgetting rate may also be stored on the user device. The base recognition accuracy may also be stored. There is also an input / output interface 1028 which connects the user device 1020 to the server 1000.

[0134] As detailed above, the present techniques provide at least the following contributions: (1) A novel inference method for OSCIL based on branching Decision Block which can increase the novel class recognition accuracy significantly. (2) Controllable forgetting, a novel feature of the present method which allows the system to predictably control the level of base class forgetting while adapting to new classes. (3) A solution that works on low-resource devices, and which does not require storage of any of the old samples. (4) An inference method that can be applied on top of any state-of-the-art base training methods, although it works best in combination with simple ProtoNet training. (5) Out-of-distribution prediction for query images (i.e. whether or not an image belongs to the previously-seen base classes). OOD predictions can be useful for user applications, where the system itself determines whether the user needs to annotate an incoming image.

[0135] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

1. A computer-implemented method for using a machine learning, ML, model on a user device to classify an object in an image, the method comprising:obtaining an image depicting an object;generating, using the ML model, a feature vector for the object in the image, wherein the ML model comprises a base classification dataset which comprises a plurality of base classification labels and each base classification label is associated with at least one feature vector which represents a base prototype for the base classification label and wherein the ML model comprises a user-specific classification dataset which comprises at least one userspecific classification label and each user-specific classification label is associated with a feature vector which represents a user-specific prototype for the user-specific classification label;calculating a distance in feature space between the generated feature vector and each feature vector in the base classification dataset;identifying, from the base classification dataset, the base prototype having a feature vector which is closest to the generated feature vector;determining whether a distance between the feature vector for the identified base prototype and the generated feature vector is greater than a pre-defined distance threshold;when the determined distance is less than or equal to the pre-defined distance threshold, outputting the base classification label of the identified base prototype as the classification label for the object in the image; andwhen the determined distance is greater than the pre-defined distance threshold, using the user-specific classification dataset to output a classification label for the object in the image.

2. The method as claimed in claim 1, wherein using the user-specific classification dataset to output a classification label for the object in the image comprises:calculating a distance between the generated feature vector and each feature vector in the user-specific classification dataset; andidentifying, from the user-specific classification dataset, the user-specific prototype having a feature vector which is closest to the generated feature vector.

3. The method as claimed in claim 2, wherein outputting a classification label based on the identified user-specific prototype comprises:outputting the user-specific classification label for the identified user-specific prototype as the classification label for the object in the image.

4. The method as claimed in claim 3, further comprising, prior to outputting the userspecific classification label:determining whether a distance between the feature vector for the identified userspecific prototype and the generated feature vector is greater than a second pre-defined distance threshold; andwhen the distance between the feature vector for the identified user-specific prototype and the generated feature vector is less than or equal to the second pre-defined distance threshold outputting the user-specific classification label for the identified user-specific prototype as the classification label for the object in the image.

5. The method as claimed in claim 4, further comprising:when the distance between the feature vector for the identified user-specific prototype and the generated feature vector is greater than the second pre-defined distance threshold, outputting one or both of a null classification label and an error message.

6. The method as claimed in claim 4, further comprising:when the distance between the feature vector for the identified user-specific prototype and the generated feature vector is greater than the second pre-defined distance threshold, outputting a coarse classification label, wherein the coarse classification label is selected from the base classification dataset.

7. The method as claimed in claim 5 or claim 6, wherein when a null classification label, error message or coarse classification is output, the method further comprises:obtaining user input specifying a user-specific classification label for the object in the image; andupdating the user-specific classification dataset using the user input.

8. The method as claimed in claim 7, wherein updating the user-specific classification dataset using the user input comprises:determining whether the input user-specific classification label matches a user-specific classification label in the user-specific classification database; andwhen the input user-specific classification label does not match a user-specific classification label, adding the generated feature vector and the input user-specificclassification label to the user-specific classification dataset wherein the generated feature vector is the feature vector which represents a user-specific prototype for the user-specific classification label.

9. The method as claimed in claim 7 or claim 8, wherein updating the user-specific classification dataset using the user input comprises:determining whether the input user-specific classification label matches a user-specific classification label in the user-specific classification database; andwhen the input user-specific classification label matches a user-specific classification label, using the generated feature vector to update the at least one feature vector which represents a user-specific prototype for the user-specific classification label.

10. The method as claimed in any one of the preceding claims, wherein the ML model is personalised before being used, wherein the personalisation comprises:receiving, from a server, a base ML model comprising a fine-tuned backbone and the base classification dataset;obtaining one or more user-specific sample images each of which have been annotated with at least one user-specific classification label; andpersonalising the base ML model using the one or more annotated user-specific sample images, by:freezing the fine-tuned backbone of the received base ML model;generating the user-specific classification dataset using the one or more annotated user-specific sample images,wherein the personalised ML model comprises the base ML model, the userspecific classification dataset and the base classification dataset.

11. A computer-implemented method for personalising a machine learning, ML model for use on a user device to classify an object in an image, the method comprising:receiving, from a server, a base ML model comprising a fine-tuned backbone and a base classification dataset which comprises a plurality of base classification labels and each base classification label is associated with at least one feature vector which represents a base prototype for the base classification label;obtaining one or more user-specific sample images each of which have been annotated with at least one user-specific classification label; andpersonalising the base ML model using the one or more annotated user-specific sample images, by:freezing the fine-tuned backbone of the received base ML model;generating a user-specific classification dataset using the one or more annotated user-specific sample images, wherein the user-specific classification dataset comprises at least one user-specific classification label and each user-specific classification label is associated with a feature vector which represents a user-specific prototype for the user-specific classification label; andsetting a pre-defined distance threshold which is used by the ML model when classifying images to determine whether to output a classification label based on the base classification dataset or the user-specific classification dataset;wherein the personalised ML model comprises the base ML model, the userspecific classification dataset, the base classification dataset and the pre-defined distance threshold.

12. The method as claimed in claim 10 or claim 11, wherein obtaining one or more annotated user-specific sample images comprises receiving a single sample image for each user-specific classification label.

13. The method as claimed in any one of the preceding claims, further comprising:setting the pre-defined distance threshold based on a maximum tolerable forgetting rate for the base classification labels in the base classification dataset.

14. The method as claimed in claim 13, further comprising:receiving, from a server, the maximum tolerable forgetting rate.

15. The method as claimed in claim 13 or claim 14, wherein setting the pre-defined distance threshold comprises:obtaining a test dataset comprising a plurality of images each labelled with a base classification label;determining, using the test dataset, a base recognition accuracy, wherein the base recognition accuracy is the accuracy of the ML model when classifying images in the test dataset without using the pre-defined distance threshold;selecting a value for the pre-defined distance threshold;determining, using the test dataset and the selected value for the pre-defined distance threshold, a base class accuracy of the ML model, wherein the a-base class accuracy is the accuracy of the ML model when classifying images in the test dataset when using the predefined distance threshold; andwhen the base class accuracy is greater than or equal to a difference between the base recognition accuracy and the maximum tolerable forgetting rate, setting the selected value for the pre-defined distance threshold as the pre-defined distance threshold.

16. The method of claim 15, wherein obtaining the test dataset comprises selecting a subset of the base classification dataset as the test dataset.

17. The method as claimed in any one of the preceding claims, wherein outputting the classification label comprises:modifying the obtained image to include the output classification label; and outputting the modified image.

18. The method as claimed in any one of the preceding claims, further comprising: controlling the user device based on the output classification label and / or outputting a recommendation based on the output classification label.

19. A user device for using a personalised machine learning, ML, model to classify an object in the image, the user device comprising:a user interface; andat least one processor coupled to memory, for:obtaining an image depicting an object;generating, using the personalised ML model, a feature vector for the object in the image, wherein the ML model comprises a base classification dataset which comprises a plurality of base classification labels and each base classification label is associated with at least one feature vector which represents a base prototype for the base classification label and wherein the ML model comprises a user-specific classification dataset which comprises at least one user-specific classification label and each user-specific classification label is associated with a feature vector which represents a user-specific prototype for the user-specific classification label;calculating a distance in feature space between the generated feature vector and each feature vector in the base classification dataset;identifying, from the base classification dataset, the base prototype having a feature vector which is closest to the generated feature vector;determining whether a distance between the feature vector for the identified base prototype and the generated feature vector is less than or equal to a pre-defined distance threshold; andwhen the determined distance is less than or equal to the pre-defined distance threshold, outputting the base classification label of the identified base prototype as the classification label for the object in the image; andwhen the determined distance is greater than the pre-defined distance threshold, using the user-specific classification dataset to output a classification label for the object in the image.

20. A system fortraining a machine learning, ML, model to recognise user-specific classes, the system comprising:a server comprising at least one processor coupled to memory, for:obtaining an ML model with a pre-trained backbone;obtaining a domain-specific training dataset, comprising a plurality of images each labelled with at least one base classification label;training the pre-trained backbone using the domain-specific training dataset to generate a base ML model with a fine-tuned backbone and a base classification dataset comprising a plurality of base classification labels each of which is associated with at least one feature vector which represents a base prototype for the base classification label; andat least one user device, wherein each user device comprises at least one processor coupled to memory, for:obtaining one or more annotated user-specific sample images each of which have been annotated with at least one user-specific classification label which is not included in the first dataset;obtaining a copy of the base ML model;personalising the base ML model using the one or more annotated userspecific sample images, by:freezing the fine-tuned backbone; andgenerating a user-specific classification dataset using the one or more annotated user-specific sample images, wherein the user-specific classification dataset comprises at least one user-specific classification label which is associated with at least one feature vector which represents a user-specific prototype for the user-specific classification label.

21. A computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out the method of any one of claims 1 to 18.

Images