Method for personalising a machine learning model

Abstract

The application describes techniques related to a method for personalising a trained ML model, on a user device, to be able to perform personalised object detection in images such that the personalise

Description

Field

[001] The present techniques generally relate to a method for personalising a machine learning, ML, model. In particular, the present techniques provide a method for personalising a trained ML model, on a user device, to be able to perform personalised object detection in images such that the personalised ML model can recognise objects that are personal to the user of the user device. Background

[002] A recent trend in robotic home appliances is to include vision systems on personal devices for performing a task of personal object search. This task involves localization and identification of personal items of interest on images captured by robotic appliances, with each item referenced only by a few annotated images. The task is crucial for robotic home appliances and mobile systems, which need to process personal visual scenes or to operate with particular personal objects (e.g., for grasping or navigation). There are several challenges associated with this task. For example, a robot vision system needs to be able to distinguish between many fine-grained classes, in the presence of occlusions and clutter. Additionally, there are strict resource requirements for the on-device system restrict usage of most state-of-the-art methods for few-shot learning, and often prevent on-device adaptation.

[003] So-called semantic segmentation is a common technique employed for computer vision tasks, where a scene is segmented, based on a reference image and a mask, to identify objects in the scene. However, there are multiple disadvantages associated with existing semantic segmentation techniques.

[004] Firstly, existing solutions only evaluate the intersection over union (loll) metric for the mask corresponding to the ground truth class on the image, thus not accounting for the multiclass scenario. Secondly, they require adaptation training on coarse datasets, thus making fine-grained classes indistinguishable in the feature space (which is part of the effect known as the neural collapse). Thirdly, current transformer-based solutions rely on large foundation models which may be too costly for on-device implementation.

[005] The present applicant has identified the need for an improved technique for on-device personalisation of machine learning, ML, models. Summary

[006] In a first approach of the present techniques, there is provided a computer-implemented method for personalising, on a user device, a trained machine learning, ML, model for classifying images, the method comprising: receiving a personalisation dataset comprising at least one data pair, each data pair comprising: an image depicting an object which is personal to a user of the user device, and label data comprising location information indicating a location of the object in the image, and a label for the object; for each data pair in the personalisation data set: generating, using a trained feature extractor of the ML model and the location of the object in the image, a feature map for the object in the image of the data pair; determining, using the generated feature map for the object, a class prototype comprising a feature vector for the object and a class threshold that enables selection of relevant patches of interest on query images to be classified; and assigning, to each class prototype, the label for the object; and storing, on the user device, the class prototype and assigned label as parameters of the ML model, thereby personalising the ML model to the user of the user device.

[007] Advantageously, the present techniques enable a pre-trained ML model, that has been pre-trained to perform object detection in images, to be personalised such that the personalised ML model can detect user-specific objects within images. This is advantageous because the pre-trained ML model is trained using generic datasets that may not fully represent objects of interest to the user. For example, the pre-trained ML model may be able to recognise dogs or carpets within images, but it cannot identify and distinguish between a user’s specific dog and other dogs, or a user’s specific carpet and other carpets. The personalisation may be useful in a number of contexts, such as in ML-controlled consumer electronic devices. For example, the control / operation settings of a robotic vacuum cleaner may be personalised for a user’s home based on the specific objects within the home. This example, and others, will be described in more detail below.

[008] The personalisation uses an image depicting an object that is personal to a user, information on the location of the object within the image, and a label for the object. The object is also referred to herein as a “class”. The object, location information, and label are used by a trained feature extractor of the ML model to generate a feature map for the object. The feature map is then used to determine a class prototype, which comprises a feature vector (e.g. an embedding vector) that defines key features of the object. The feature vector is used to identify this object in other images after the personalisation is complete. The feature map is also used to determine a class threshold which is determined by identifying patches of the image, determining distances between feature vectors for these patches and the feature vector of the class prototype and generating the class threshold from the determined distances. Depending on the patch, the determined distances may indicate whether a feature is likely or unlikely to be a feature of the personal object. The class threshold helps the personalised model to determine whether other new images contain the personal object, for example by considering whether a distance between a feature vector of a patch in the new image and the feature vector of the class prototype is within the class threshold, i.e. whether a feature of the feature vector of the patch is likely or unlikely to be a feature of the personal object. In other words, the personalisation process involves generating a feature vector and a class threshold for each object that is personal to a user, such that the object can be readily identified in other images. Once the feature vector and class threshold are determined, they are stored together with the label for the object, so that the model can be used to detect the object in other images. This is advantageous because the model can be personalised using a single image of the object that is personal to the user (i.e. one-shot personalisation), or just a few images (i.e. few-shot personalisation). Furthermore, the present techniques are advantageous because they do not require wholesale re-training of the trained ML model on the user device using user-specific data, which could lead to catastrophic forgetting or neural collapse. Instead, the ML model is provided with information that enables the ML model to determine whether there are any personal objects present in the image and, if so, where the objects are located in the image.

[009] The term “patch” is used herein to mean a small portion or region of an image. Each patch of the image may be the same size, where a patch is defined by a number of pixels of the image. The number of pixels in the patch may be N x M, where N and M are integers, and N and M may be the same (square patch) or different (rectangular patch).

[010] In some cases, the location information may correspond to a segmentation map that isolates the object in the image. In other words, the segmentation map may be considered to segment the personal object in the image from a remainder of the image. In such cases, determining the class prototype may comprise: selecting a plurality of patches of the image using the segmentation map, wherein each patch in the identified plurality of patches at least partially intersects with the segmentation map. That is, in some cases, the location information is provided in the form of a segmentation map that just represents the object location. In other words, the image is divided into two distinct regions, where each region represents different objects. In some cases, the segmentation map may effectively divide the image into two regions - one region containing / representing only the object that is personal to the user which is the segmentation map, and the other region containing / representing everything else in the image which is not in the segmentation map.

[011] Alternatively, the location information may correspond to a bounding box around the object in the image. In such cases, the bounding box indicates where the object is located, but the object itself has not been isolated completed. Therefore, in such cases, the first step is to segment / isolate the object from the rest of the image. Thus, determining the class prototype may comprise: generating a segmentation map of the image using the bounding box, wherein the segmentation map isolates the object in the image; and selecting a plurality of patches of the image using the segmentation map, wherein each patch in the identified plurality of patches at least partially intersects with the segmentation map.

[012] In both cases, a plurality of patches are selected, where each patch at least partially intersects with the isolated object in the image. These patches are used to generate the feature vector and class threshold for the object / class that is personal to the user.

[013] When only a bounding box is received, the step of generating the segmentation map may comprise: identifying a plurality of object patches and a plurality of background patches of the image, wherein each object patch at least partially intersects with the bounding box and wherein each background patch at least partially borders at least one object patch. The step of generating the segmentation map may further comprise: combining the plurality of object patches and the plurality of background patches to generate a set of patches; clustering the set of patches into a plurality of patch clusters; and filtering the plurality of patch clusters to remove patch clusters which include background patches to generate the segmentation map. In other words, in order to convert a bounding box into a segmentation map, it is necessary to consider all of the patches of the image which have a non-empty intersection with the bounding box, as well as patches that border the bounding box. These patches are then grouped into a set of clusters based on the features of each patch. That is, a feature vector for each cluster is generated, where the feature vector defines key features of the content of the patch. Then, the feature vectors for all the patches are clustered using a clustering algorithm (such as, but not limited to, k-means clustering). A maximum number of clusters may be pre-defined to avoid too-small and too-large clusters being generated. Each cluster may contain at least one patch. Since patches that taken from outside of the bounding box do not contain any features of the object being segmented, the clusters which contain background patches are filtered out. This enables the object of interest in the bounding box to be separated from the background / other 4 objects in the image / bounding box, thereby generating a segmentation map in which the object of interest is highlighted or segmented from other content in the image.

[014] Once the segmentation map has been obtained / generated and the object of interest has thereby been isolated from other objects in the image, as noted above, a plurality of patches are selected, where each patch partially intersects with the segmentation map. (These are different patches to those used when generating the segmentation map). The step of determining the feature vector of the class prototype may then comprise: generating a feature vector for each patch of the selected plurality of patches of the image; and averaging the feature vectors to generate the feature vector for the object. By selecting patches which partially overlap with the segmentation map, it is possible to determine features of interest in the object of interest that can help to segment the object from other images in the future. The averaging may comprise calculating the mean of the feature vectors. The generated feature vector defines features that help to segment the object from other images in the future.

[015] As noted above, it is desirable to provide some information to the model which helps the model ascertain, in the future, whether an object in an image is likely to be the personal object of the user. That is, it is desirable to avoid misclassification of personal objects, or for false positives to arise when images contain objects but do not contain personal objects. To do so, the present techniques make use of a class threshold, which helps to determine whether a personal object is present within an image. The step of determining the class threshold of the class prototype may further comprise: identifying a plurality of negative patches of the image using the segmentation map, wherein each negative patch has an empty intersection with the segmentation map. In other words, the negative patches do not intersect with the segmented personal object in the image and therefore, do not contain any features of interest in identifying the personal object.

[016] Preferably, determining the class threshold of the class prototype may further comprise: calculating, for each patch of the selected plurality of patches, a positive patch distance corresponding to a distance in feature space between the feature vector of the class prototype and the feature vector of the selected patch; generating a feature vector for each patch of the plurality of negative patches of the image; and calculating, for each negative patch of the plurality of negative patches a negative patch distance corresponding to a distance in feature space between the feature vector of the class prototype and the feature vector of the negative patch. The positive patch distances indicate how far, in feature / embedding space, a feature can be from the class prototype and still be indicative of a feature of the personal object. That 5 is, the positive patch distances indicate whether a feature is likely to be a feature of the personal object. The negative patch distances indicate how far, in feature / embedding space, a feature can be from the class prototype and not be indicative of a feature of the personal object. That is, the negative patch distances indicate whether a feature is unlikely to be a feature of the personal object. Determining the class threshold may further comprise determining a positive patch distance threshold from the calculated positive patch distances; determining a negative patch distance threshold from the calculated negative patch distances; and determining the class threshold of the class prototype from the positive patch distance threshold and the negative patch distance threshold. For example, the class threshold may be the minimum of the positive patch distance threshold and the negative patch distance threshold.

[017] Determining the class threshold of the class prototype may further comprise: filtering, using the calculated positive patch distances, the selected plurality of patches having a positive patch distance over a predefined positive distance value; filtering, using the calculated negative patch distances, the plurality of negative patches having a negative patch distance below a predefined negative distance value; determining, using remaining patches from the selected plurality of patches, a positive patch distance threshold; and determining, using remaining patches in the plurality of negative patches, a negative patch distance threshold. In other words, outliers may be filtered before determining the class threshold. The positive patch distance threshold may be the maximum positive patch distance of the remaining (i.e. filtered) positive patches. Similarly, the minimum patch distance threshold may be the minimum negative patch distance of the negative patches. The class threshold may be defined as the minimum of the positive patch distance threshold and the negative patch distance threshold, after filtering. That is, some possible patch outliers are removed from the set of positive patch distances and the set of negative patch distances to fine-tune the patch distances and ensure they are most helpful for identifying whether features are indicative of the personal object. For example, the predefined positive distance value may be set so that the highest 5% (or another similar fixed percentage) of positive patches (those which are most similar to the features of the personal object) are excluded. The predefined negative distance value may be set so that the lowest 5% (or another similar fixed percentage) of negative patches (those which are most dissimilar to the features of the personal object) may be removed.

[018] In a second approach of the present techniques, there is provided a user device for performing personalised object recognition, the user device comprising: at least one processor coupled to memory, for: receiving a personalisation dataset comprising at least one data pair, each data pair comprising: an image depicting an object which is personal to a user of the user device, and label data comprising location information indicating a location of the object in the image, and a label for the object; for each data pair in the personalisation data set: generating, using a trained feature extractor of a trained ML model for classifying images and the location of the object in the image, a feature map for the object in the image of the data pair; determining, using the generated feature map for the object, a class prototype comprising a feature vector for the object and a class threshold that enables selection of relevant patches of interest on query images to be classified; and assigning, to each class prototype, the label for the object; and storing, on the user device, the class prototype and assigned label as parameters of the ML model, thereby personalising the ML model to the user of the user device.

[019] The features described above with respect to the first approach apply equally to the second approach and therefore, for the sake of conciseness, are not repeated.

[020] In a third approach of the present techniques, there is provided computer-implemented method for personal object detection, on a user device, using a personalised, trained machine learning, ML, model comprising at least one class prototype for a personal object, the method comprising: receiving an image depicting at least one object to be classified; and for each object depicted in the received image: selecting a plurality of patches of interest in the received image; determining, using the at least one class prototype, a likelihood of the selected patches representing the personal object of the at least one class prototype; and outputting, when the determined likelihood for the selected patches is above a classification threshold for one class prototype of the at least one class prototype: a label for the personal object corresponding to a label associated with the class prototype for which the determined likelihood is above the classification threshold, and a location of the personal object in the received image.

[021] Thus, once the at least one personal object has been added to the ML model (as explained above with respect to the first and second approaches), the personalised ML model can be used to perform personal object detection in new images. Specifically, the class prototype (i.e. feature vector and class threshold) for each personal object that has been added to the ML model, can be used to determine whether new received images contain a personal object. This is done by selecting patches of interest in a new received image, and then using the class prototype to determine whether the patches contain a personal object or not. If the patches do contain a personal object, the ML model outputs a label for the personal object and a location of the personal object in the received image. This is useful because then a user can be provided with information on what personal object has been identified, and where it is within the image. Providing both the object name / label and where it is located in the image is helpful when an image contains many objects (personal and non-personal).

[022] Outputting a location of the personal object may comprise outputting a segmentation map that segments the personal object in the image from a remainder of the image. The segmentation map helpfully isolates the personal object. The segmentation map may be helpful for a user device’s machine vision to understand where the personal object is, particularly if the user device needs to interact with (or avoid interacting with) the personal object once it has been identified.

[023] The method may further comprise: adding a bounding box to the received image, where the bounding box contains the personal object depicted by the segmentation map. The bounding box may be useful for a human or a machine to understand where the personal object is located in the image.

[024] The method may further comprise: generating, using a feature extractor of the ML model, a feature map for the received image.

[025] Preferably, the selecting of a plurality of patches may comprise: identifying a plurality of patches; generating, using the feature map, a feature vector for each selected patch; calculating, for each identified patch, a distance between the generated feature vector and a feature vector of the at least one class prototype; and for each class prototype: determining whether the calculated distance is within the class threshold of the class prototype; and retaining an identified patch as a selected patch when the calculated distance for that patch is less than the class threshold for the class prototype. As explained above, the class threshold may be determined from a negative patch distance threshold and a positive patch distance threshold, for example as the minimum of the negative patch distance threshold and the positive patch distance threshold. Thus, for each class prototype, patches that are greater than the positive or negative patch distance thresholds are not considered useful in determining whether the personal object corresponding to that class prototype is present within the image, and therefore may be discarded. It will be understood that if the plurality of patches are not relevant to any of the class prototypes, a different plurality of patches may be identified and the process repeated.

[026] Identifying a plurality of patches may comprise: grouping together a plurality of patches based on the patches being spatially close together in the received image. This may ensure that the plurality of patches are not selected from very spatially disparate places in the image, where they are less likely to relate to the same object.

[027] Determining, using the at least one class prototype, a likelihood of the selected patches representing the personal object of the at least one class prototype, may comprise: for each class prototype for which a plurality of the selected patches are retained: calculating a class score by averaging the generated feature vector for the selected plurality of patches for the class prototype; comparing the class score with the feature vector of the class prototype to determine whether the class score is greater than or equal to a classification threshold for the class prototype; and determining, when the class score is greater than or equal to the classification threshold, that the selected patches contain the personal object corresponding to the class prototype.

[028] In some cases, selecting a plurality of patches may comprise selecting a plurality of sets of patches, each set comprising at least one patch. That is, multiple sets of patches may be considered, which can be useful for accurately determining the edge(s) of the personal object, because one set of patches may better identify the object and its edge(s) than another.

[029] When two or more sets of patches are determined to contain the personal object corresponding to a class prototype, the method may comprise: selecting a set of patches, from the two or more sets of patches, with a maximum class score.

[030] Outputting a location of the personal object may then comprise outputting a segmentation map, based on the selected set of patches with a maximum class score, that segments the personal object in the image from a remainder of the image.

[031] When the determined likelihood for the selected patches is below a classification threshold for the at least one class prototype, the method may comprise: outputting a negative classification result for the object indicating that the object is not a personal object. Thus, in some cases, the personalised ML model may simply state that a personal object has not been found in the image.

[032] In other cases, the method may further comprise: performing a coarse classification of the object using the trained ML model; and outputting from the trained ML model a coarse class name for the object. Thus, when no personal object has been found, the ML model may simply output a class name for any non-personal object that has been found within the image. For example, the ML model may output “dog” when a dog has been identified, but the user’s own dog has not been identified.

[033] In a fourth approach of the present techniques, there is provided a user device for performing personalised object recognition using a personalised, trained machine learning, ML, model comprising at least one class prototype for a personal object, the user device comprising: at least one processor coupled to memory, for: receiving an image depicting at least one object to be classified; and for each object depicted in the received image: selecting a plurality of patches of interest in the received image; determining, using the at least one class prototype, a likelihood of the selected patches representing the personal object of the at least one class prototype; and outputting, when the determined likelihood for the selected patches is above a classification threshold for one class prototype of the at least one class prototype: a label for the personal object corresponding to a label associated with the class prototype for which the determined likelihood is above the classification threshold, and a location of the personal object in the received image.

[034] The user device may be a smart device. The user device may be a smartphone. A smartphone is an example of a smart device. The user device may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV), a smart fridge, a smart oven, a smart vacuum cleaner, a smart robotic device, and so on. More generally, the user device may be a constrained-resource device, but which has the minimum hardware capabilities to personalise a trained ML model, and to use the personalised ML model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart lawn mower, smart oven, etc). It will be understood that this is a non-exhaustive and non-limiting list of example devices.

[035] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[036] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[037] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[038] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[039] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[040] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[041] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[042] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[043] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[044] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[045] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[046] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[047] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[048] Figure 1 is a schematic diagram of existing techniques for personalizing ML models on-device;

[049] Figure 2A is a flowchart of a example steps for personalising, on a user device, a trained machine learning, ML, model for classifying images;

[050] Figure 2B is a block diagram showing a high-level overview of the present techniques;

[051] Figure 3 is a block diagram showing more details of the feature map generation step S102 of Figure;

[052] Figure 4A is a block diagram showing more details of the support image processing step S102 of Figure 2A;

[053] Figure 4B is a schematic diagram showing how multiple parts of different sizes of an object can be obtained in order to generate a class prototype for a personal object;

[054] Figure 4C shows examples of how location information / a bounding box may be given;

[055] Figure 5 is a block diagram showing example steps for personal object detection using a personalised, trained machine learning, ML, model comprising at least one class prototype for a personal object;

[056] Figure 6A is a block diagram showing more details of the query image processing stage of Figure 2A;

[057] Figure 6B schematically illustrates an example of how the cosine similarity distance is calculated.

[058] Figure 6C is a block diagram showing an example of how a personalized ML model may be used to perform image classification;

[059] Figures 7 to 10 show tables of results from experiments to test the present techniques, and

[060] Figure 11 is an example of a method of the present techniques which may be implemented in a home appliance or another user device such as a smart phone. Detailed description of the drawings

[061] Broadly speaking, the present techniques generally relate to a method for personalising a trained ML model, on a user device, to be able to perform personalised object detection in images such that the personalised ML model can recognise objects that are personal to the user of the user device. Advantageously, the present techniques enable a pre-trained ML model, that has been pre-trained to perform object detection in images, to be personalised such that the personalised ML model can detect user-specific objects within images. This is advantageous because the pre-trained ML model is trained using generic datasets that may not fully represent objects of interest to the user. For example, the pre-trained ML model may be able to recognise dogs within images, but it cannot identify and distinguish between a user’s specific dog and other dogs. The personalisation may be useful in a number of contexts, such as in ML-controlled consumer electronic devices. For example, the control / operation settings of a robotic vacuum cleaner may be personalised for a user’s home based on the specific objects within the home.

[062] Computer vision has a pivotal role in mobile systems and home appliances to understand the surroundings and navigate in complex environments. Scene understanding deep neural networks have obtained outstanding results and have been successfully deployed to mass-accessible personal devices: for example industrial or domestic service robots (e.g., vacuum cleaners), assistive robots, and smartphones.

[063] Recently, increasing attention has been devoted on the personalization of on-device Al vision models to tackle a variety of practical use cases. The present techniques focus on personal item search, whereby robot vision systems aim to localize and recognize personal user classes (or fine-grained classes, e.g. my dog Archie, her dog Rocco, my favourite cup, your favourite flower, etc.) on scenes. Specifically, a user provides a small number of reference images with location annotations (either a segmentation map or a bounding box) for each personal item. Then, given a new scene, a visual system needs to i) determine which of the personal objects are present in the scene, and ii) provide the location (in the form of a segmentation map or bounding box) for each of the personal objects present in the scene. This task has found significant applications for personal assistants and service robots: for navigation (e.g., reach my white sofa), HRI (e.g., find my dog Archie), grasping (e.g., bring me my phone), etc.

[064] Figure 1 is a schematic diagram of existing techniques for personalizing ML models on-device. Current techniques focus on different aspects of the task. The closest comparison task to the present techniques is so-called few-shot semantic segmentation, which aims to segment an object on the scene given a provided reference image and mask. Common adaptive semantic segmentation methods adapt models to coarse datasets and do not account for multiple personal objects or unseen personal objects on a scene, thus generating false positive errors. While early works on few-shot semantic segmentation resorted to fine-tuning large parts of models, more recent approaches are based on training adaptation layers on top of a feature extractor with the prototypical loss. These latter approaches compute class prototypes as the average embedding of all images of a class. The label of a new (query) image is predicted by identifying the nearest prototype vector computed from the training (support) set. The training and evaluation are usually performed on popular segmentation datasets with coarse-level classes (e.g. person, cat, car, chair1), namely PASCAL - 5' (Shaban, A. et al) and COCO - 20‘ (Nguyen, K. et al).

[065] Recent advancements in large vision models have led to novel few-shot scene understanding works, especially applied to semantic segmentation, such as PerSAM (Zhang, R. et al.), and Matcher (Liu, Y et al.). PerSAM, a training-free approach, uses a single image with a reference mask to localize and segment target concepts. Matcher, utilizing off-the-shelf vision foundation models, can showcase impressive generalization across tasks. On the other hand, both approaches are computationally expensive and not applicable on low-resource devices.

[066] The present techniques tackle the task of personal object search, which is closely related to the practical scenarios. In contrast to existing methods, the present techniques do not rely on coarse dataset training and are very lightweight, allowing seamless implementation on device. Pre-trained feature extractor models like the DINOv2 (Oquab, M. et al) model show great versatility for downstream tasks. Herein, the present techniques are also interchangeably referred to as Sw / ss DINO, after the Swiss Army Knife, for its incredible versatility and adaptability.

[067] It has been shown that the common classification architectures trained on coarse-level datasets have low accuracy with fine-grained classes when applied out-of-the-box. This is due to the fact that the fine-grained classes become indistinguishable in the feature space after long training on coarse class classification, part of the effect known as neural collapse. Therefore, fine-tuning or adaptation methods are often employed to separate the feature vectors for fine-grained datasets.

[068] In SSL, significant contributions have been made to the development of pre-trained models, such as DINO (Caron, M. et al) and DINOv2. These models have demonstrated remarkable capabilities in feature extraction and object localization, making them highly transferable to the task of few-shot personalization. Simeoni et al. presents a method named LOST to leverage pre-trained vision transformer features for unsupervised object localization. Melas-Kyriazi et al. reframed image decomposition as a graph partitioning problem, using eigenvectors from self-supervised networks to segment images and localize objects. These methods not only provide a strong foundation for the few-shot personalization method of the present techniques but also highlight the potential of SSL transformer backbones in overcoming the challenge of neural collapse.

[069] The present techniques will now be described in more detail. In particular, the problem of personal object search will be formulated and the terminology relating to stages of the present techniques will be introduced.

[070] The present techniques adopt a simple yet effective framework for one-shot personal object search based on transformer models, such as the recent DINOv2 model which was shown to have strong zero-shot generalization properties. Advantagesouly, the present techniques avoid false positive errors by performing open-set classification on parts of the image prior to generating segmentation mask and are able to handle challenging on-device personalized scene understanding requirements, and does not require any adaptation training.

[071] Pre-training Stage. The first step is to pre-train a backbone model of an ML model on a large dataset. The backbone should provide localization information of objects on an image, and have a strong ability to transfer to new personal classes, in particular avoiding neural collapse of generated features.

[072] On-device Personalization Stage. The second step is an on-device personalization stage where the ML model is personalized. After the system is implemented on a mobile or robotic device (e.g., a robot vacuum cleaner or service robot), it is shown a few images of personal objects, together with their label (e.g., dog Archie, dog Rocco, my mug, etc.), and a prompt indicating the location of the object on the image, in the form of a bounding box or a segmentation map. Those images are also known in the few-shot literature as support images.

[073] Although the present techniques can be applied to any number of support images per personal object, to simplify evaluation and notations, the most challenging one-shot setup is considered, i.e., a single support image Sc for each personal object index c = 1,2,..., C is obtained, where C is the number of personal objects.

[074] On-device Open-set Personal Object Segmentation, Detection, and Recognition. During the on-device inference step, a new test image Q is given (also known in the literature as the query image, and the terms will be used interchangeably herein). For this image, the following need to be determined: i) which personal objects, if any, are present on the image; ii) for each of the personal objects present on the image, find its location in the form of segmentation map or a bounding box.

[075] More formally, a personal object search (POS) method is defined by: POS(Q) := (oloc^Q).....olocc(Q)), (1) oloc (l°cc(.Q)> if th eobject c is present , . c (None, otherwise where locc(Q) can take the form of a bounding box or a segmentation map for the object c on the image Q.

[076] The present techniques will now be described in more detail.

[077] Figure 2A is a flowchart of a example steps for personalising, on a user device, a trained machine learning, ML, model for classifying images. The method comprise: receiving a personalisation dataset comprising at least one data pair, each data pair comprising: an image depicting an object which is personal to a user of the user device, and label data comprising location information indicating a location of the object in the image, and a label for the object (step S100); for each data pair in the personalisation data set: generating, using a trained feature extractor of the ML model and the location of the object in the image, a feature map for the object in the image of the data pair (step S102); determining, using the generated feature map for the object, a class prototype comprising a feature vector for the object and a class threshold that enables selection of relevant patches of interest on query images to be classified (step S104); and assigning, to each class prototype, the label for the object (step S106); and storing, on the user device, the class prototype and assigned label as parameters of the ML model, thereby personalising the ML model to the user of the user device (step S108).

[078] Figure 2B is a block diagram showing a high-level overview of the present techniques. The present techniques comprise feature extraction, which is used to generate a class prototype for a personal object (“support image processing”) and to use class prototypes to analyse new images to determine if they contain a personal object (“query image processing”). The support image processing and query image processing is described in more detail below.

[079] Patch-level feature map extractor. Figure 3 is a block diagram showing more details of the feature map generation step S102 of Figure 2A. A pre-trained feature extractor 100 is used. The pre-trained feature extractor 100 may be a transformer-based patch-level feature extractor. The pre-trained feature extractor is typically trained on a server or on an alternative computing device capable of training large foundation models. The pre-trained feature extractor 100 may be distributed to user device(s) once (pre-)trained.

[080] Transformer models, such as the DINOv2 model, have localization and fine-grained separation capabilities for the feature maps they produce. Thus, the DINOv2 is an example of a suitable transformer backbone. Another example of a suitable pre-trained feature extractor 100 is the SAM segmentation model (Kirillov, A. et al). However, it will be appreciated that any appropriate pre-trained feature extractor 100 may be used.

[081] The backbone 100 B takes an image X as an input (which may be a so-called support image or query image, as explained in more detail below) and produces i) patch-wise feature map XF = (XFlt -,X^pNp) 104, where NP is the number of patches along each side of the image, and XFj is the D-dimensional vector corresponding to the (jj)-th spatial patch on the image; and ii) a D-dimensional class token Xc, such that B(X) = (XF,Xcy

[082] Given support images 102 {Sc}^=1 for each personal-level class and a query image 104 Q, corresponding feature maps 106 {Sf}£=1 (denoted the support feature map in Figure 2B) and 108 QF (denoted the query feature map in Figure 2B), respectively, are computed.

[083] Support feature map processing. Figure 4A is a block diagram showing more details of the support image processing step S102 of Figure 2A. For each personal class c = 1,2,..., C, the same processing steps are applied to the feature map SF 106. In the following, the index c is dropped to make the notation less cluttered.

[084] In some cases, receiving, at step S100 of Figure 2A, location information may comprise receiving a segmentation map (as shown by 112) that isolates the object in the image (102). In such cases, determining the class prototype may comprise: selecting a plurality of patches of the image using the segmentation map, wherein each patch in the identified plurality of 19 patches at least partially intersects with the segmentation map. That is, in some cases, the location information is provided in the form of a segmentation map that divides the image into distinct regions, where each region represents different objects. In some cases, the segmentation map may divide the image into two regions - one region containing / representing only the object that is personal to the user - i.e. the segmentation map itself, and the other region containing / representing everything else in the image.

[085] Bounding box into segmentation map. If a ground truth bounding box b (see 110 in Figure 4A) for the support image 102 S is given, the union of all patches P^- of the image that have non-empty intersection with b may be considered, denoted by bp, as well as patches bordering bp, denoted by dbp. A set of corresponding feature vectors {Spj\Pij e bp udbp} may be partitioned into ks clusters using, for example, the / c-means method (in the implementation described below, ks was empirically chosen to be ks = 5), denoting the set of patches in each cluster as Kp, with r = 1,2, ...,ks.

[086] Given that the patches from dbp are outside of the bounding box, and thus do not belong to the object of interest, patch clusters which contain those ‘negative’ patches are filtered out, thus resulting in an (approximate) segmentation map: seg = U {Kp I dbp n Kp = 0}. (3)

[087] This process allows the separation (i.e. isolation and the terms may be used interchangeably) of the object of interest within the bounding box from the background.

[088] Thus, alternatively, receiving, at step S100 of Figure 2A, location information may comprise receiving a bounding box around the object in the image. In such cases, the bounding box indicates where the object is located, but the object itself has not been isolated completed. Therefore, in such cases, the first step is to segment / isolate the object from the rest of the image. Thus, determining the class prototype may comprise: generating a segmentation map of the image using the bounding box, wherein the segmentation map isolates the object in the image; and selecting a plurality of patches of the image using the segmentation map, wherein each patch in the identified plurality of patches at least partially intersects with the segmentation map.

[089] When only a bounding box is received, the step of generating the segmentation map may comprise: identifying a plurality of object patches (e.g. bp in the equation above) and a plurality of background patches (e.g. dbp in the equation above) of the image, wherein each object patch at least partially intersects with the bounding box and wherein each background patch at least partially borders at least one object patch. The step of generating the segmentation map may further comprise: combining the plurality of object patches and the plurality of background patches to generate a set of patches; clustering the set of patches into a plurality of patch clusters; and filtering the plurality of patch clusters to remove patch clusters which include background patches to generate the segmentation map. In other words, in order to convert a bounding box into a segmentation map, it is necessary to consider all of the patches of the image which have a non-empty intersection with the bounding box, as well as patches that border the bounding box. These patches are then grouped into a set of clusters based on the features of each patch. That is, a feature vector for each cluster is generated, where the feature vector defines key features of the content of the patch. Then, the feature vectors for all the patches are clustered using a clustering algorithm (such as, but not limited to, k-means clustering). A maximum number of clusters may be pre-defined to avoid too-small and too-large clusters being generated. Each cluster may contain at least one patch. Since patches that taken from outside of the bounding box do not contain any features of the object being segmented, the clusters which contain background patches are filtered out. This enables the object of interest in the bounding box to be separated from the background / other objects in the image / bounding box, thereby generating a segmentation map in which the object of interest is highlighted or segmented from other content in the image.

[090] In both cases, a plurality of patches are selected, where each patch at least partially intersects with the isolated object in the image. These patches are used to generate the feature vector and class threshold for the object / class that is personal to the user.

[091] Figure 4B is a schematic diagram showing how multiple parts of different sizes of an object can be obtained in order to generate a class prototype for a personal object. Typically, in existing techiques, image segmentation for the purpose of object detection is performed using holistic features representing larger regions in images (e.g. a region containing a dog). Therefore, existing techniques are sensitive to local occulsions and deformations, which may happen when, for example, an object is obscured by another object in an image. In contrast, the present techniques may perform segmentation - both when generating a class prototype, and when using class prototypes to perform object detection - using multiple scales. That is, different parts of an object may be obtained (e.g. the paws and ears of a dog), and feature 21 exrtraction may be performed on these individually to build up a richer representation of the object.

[092] Figure 4C shows examples of how location information / a bounding box may be given. As noted above, the present techniques make use of support images 102 for which location information is given. That is, each support image 102 is annotated or labelled with location information relating to the location of each object depicted in the image. The support image(s) may be labelled e.g. by a user providing a “click” or “touch” input on a user device at the location of a personal object within the image. For example, an image may depict a user’s pert dog. In labelling the support image, the user would provide a touch input via a user interface of the user device to provide location information for the dog depicted in the image. The location information may be in the form of a bounding box representing a region of the image 102 containing the dog. Alternatively, the location information may be in the form of a segmentation map corresponding to a representation of regions of the support image 102 which depict and do not depict the object. It will be appreciated location information may be provided for any type of object (i.e. not only pet dogs), and for any number of objects depicted in the image 102.

[093] Patch pooling from segmentation map. Given a (ground truth or approximate) segmentation map seg of the support image, the patches P^- which partially intersect with seg are picked, denoting the set of those patches as segp. A patch prototype is computed by averaging over patches in segp by proto := avg^Spj\Pij e segp). (4)

[094] Thus, once the segmentation map has been obtained / generated and the object of interest has thereby been isolated from other objects in the image, as noted above, a plurality of patches are selected, where each patch partially intersects with the segmentation map. (These are different patches to those used when generating the segmentation map). The step S104 in Figure 2A of determining the feature vector of the class prototype may then comprise: generating a feature vector for each patch of the selected plurality of patches of the image; and averaging the feature vectors to generate the feature vector for the object. By selecting patches which partially overlap with the segmentation map, it is possible to determine features of interest in the object of interest that can help to segment the object from other images in the future. The averaging may comprise calculating the mean of the feature vectors. The generated feature vector defines features that help to segment the object from other images in the future.

[095] Adaptive threshold for class prototype. To pick the patches of interest from the query object, a feature distance metric and a threshold are chosen to determine patches of interest on the query image 104. As a distance metric between feature vectors, the widely used cosine similarity metric is chosen. To determine the distance threshold, information about positive and negative patches on the support image is used.

[096] More concretely, segp denotes a set of patches that have non-empty intersection with the segmentation map seg, and nsegp denotes a set of patches that have empty intersection with seg. A set of positive patch distances and a set of negative patch distances is computed according to: pd = {dist(Sppproto) | Pt j e segp], (5) nd = {dist(Spj, proto) | Ptj e nsegp}. (6)

[097] As noted above, it is desirable to provide some information to the model which helps the model ascertain, in the future, whether an object in an image is likely to be the personal object of the user. That is, it is desirable to avoid misclassification of personal objects, or for false positives to arise when images contain objects but do not contain personal objects. To do so, the present techniques make use of a class threshold, which helps to determine whether a personal object is present within an image. The step S102 of determining the class threshold of the class prototype may further comprise: identifying a plurality of negative patches of the image using the segmentation map, wherein each negative patch has an empty intersection with the segmentation map. In other words, the negative patches do not intersect with the segmented personal object in the image and therefore, do not contain any features of interest in identifying the personal object.

[098] Preferably, determining the class threshold of the class prototype may further comprise: calculating, for each patch of the selected plurality of patches, a positive patch distance corresponding to a distance in feature space between the feature vector of the class prototype and the feature vector of the selected patch; generating a feature vector for each patch of the plurality of negative patches of the image; and calculating, for each negative patch of the plurality of negative patches a negative patch distance corresponding to a distance in feature space between the feature vector of the class prototype and the feature vector of the negative patch. The positive patch distances indicate how far, in feature / embedding space, a feature can be from the class prototype and still be indicative of a feature of the personal object. That is, the positive patch distances indicate whether a feature is likely to be a feature of the personal object. The negative patch distances indicate how far, in feature / embedding space, a feature can be from the class prototype and not be indicative of a feature of the personal object. That is, the negative patch distances indicate whether a feature is unlikely to be a feature of the personal object. A positive patch distance threshold may be defined as the maximum of the positive patch distances, i.e. the maximum distance that a feature in the positive patch can be from the class prototype and still be indicative of a feature of the personal object. A negative patch distance threshold may be defined as the minimum of the negative patch distances, i.e. the minimum distance that a feature in a negative patch can be from the class prototype and still be indicative of a feature of the personal object. The class threshold may be defined as the minimum of the positive and negative thresholds.

[099] Before calculating the thresholds, patch outliers may be removed. For example, possible patch outliers are removed (by removing the highest 5 percent from pd and the lowest 5 percent from nd) to find positive and negative thresholds by ptr = percentile (pd, 95) and ntr = percent Ue (nd, 5). The final adaptive class threshold is taken as the minimum of positive and negative thresholds by tr: = mm(ptr,ntr).

[100] Thus, determining the class threshold of the class prototype may further comprise: filtering, using the calculated positive patch distances, the selected plurality of patches having a positive patch distance over a predefined positive distance value; filtering, using the calculated negative patch distances, the plurality of negative patches having a negative patch distance below a predefined negative distance value; determining, using remaining patches from the selected plurality of patches, a positive patch distance threshold; and determining, using remaining patches in the plurality of negative patches, a negative patch distance threshold. The class threshold may be defined as the minimum of the positive patch distance threshold and the negative patch distance threshold. The positive patch distance threshold may be the maximum positive patch distance of the remaining (i.e. filtered) positive patches. Similarly, the minimum patch distance threshold may be the minimum negative patch distance of the negative patches. That is, some possible patch outliers are removed from the set of positive patch distances and the set of negative patch distances to fine-tune the patch distances and ensure they are most helpful for identifying whether features are indicative of the personal object. For example, the predefined positive distance value may be set so that the highest 5% (or another similar fixed percentage) of positive patches (those which are most similar to the features of the personal object) are excluded. The predefined negative distance value may be set so that the lowest 5% (or another similar fixed percentage) of negative patches (those which are most dissimilar to the features of the personal object) may be removed.

[101] Figures 4A to 4C above show how a ML model may be personalized by generating class prototypes for novel or personal classes. Figure 5 is a block diagram showing example steps for personal object detection using a personalised, trained machine learning, ML, model comprising at least one class prototype for a personal object. The method comprises: receiving an image depicting at least one object to be classified (S500); and for each object depicted in the received image: selecting a plurality of patches of interest in the received image (S502); determining, using the at least one class prototype, a likelihood of the selected patches representing the personal object of the at least one class prototype (S504); and outputting, when the determined likelihood for the selected patches is above a classification threshold for one class prototype of the at least one class prototype: a label for the personal object corresponding to a label associated with the class prototype for which the determined likelihood is above the classification threshold, and a location of the personal object in the received image (S506).

[102] Query feature map processing. Figure 6A is a block diagram showing more details of the query image processing stage of Figure 2A. That is, Figure 6A corresponds to the inference-time stage of the present techniques. Given a tuple (protoc, tr^ for each personal class c = 1,..., C and query feature map 108 QF, the following steps are used to find the patches belonging to the objects of interest.

[103] Object location candidates. For each personal class c = 1,..., C, patches in QF are found identified which are close enough to the class prototype, resulting in a set of patches denoted as psegcaw' pseg™w ■.= {Pt j | dist(QFj,protoc) <trc], (8) where dist is the cosine similarity between feature vectors.

[104] Thus, the present method (as shown in Figure 5) may further comprise: generating, using a feature extractor of the ML model, a feature map for the received image.

[105] Preferably, the selecting (at step S502 of Figure 5) of a plurality of patches may comprise: identifying a plurality of patches; generating, using the feature map, a feature vector for each selected patch; calculating, for each identified patch, a distance between the generated feature vector and a feature vector of the at least one class prototype; and for each class prototype: determining whether the calculated distance is less than a class threshold of the class prototype; and retaining an identified patch as a selected patch when the calculated distance for that patch is less than the class threshold for the class prototype. In the example above, the class threshold may the minimum of the positive patch distance threshold and the negative patch distance threshold where the positive and negative patch distance thresholds may be determined after discarding outliers as described above. Thus, for each class prototype, patches that are greater than the minimum positive or negative patch distance thresholds are not considered useful in determining whether the personal object corresponding to that class prototype is present within the image, and therefore may be discarded. It will be understood that if the plurality of patches are not relevant to any of the class prototypes, a different plurality of patches may be identified and the process repeated.

[106] Coordinate-adjusted patch k-means. Optionally, agnostic to the set of support classes, a pre-processing step may be performed on the query feature map 108 QF. To group together the patches corresponding to the same object, / c-means clustering may be applied to the patch feature vectors. In addition, feature vectors may be augmented with spatial information to reinforce the connectivity of patch clusters: Q^^concatlQ^.^-). (7). \ IN p IN p / where aco is a coordinate scaling factor aiming to control the effect of spatial information on the resulting clusters.

[107] The augmented patch features QF’aus may be clustered into kQ clusters %$, r = 1, ...,kQ and may be saved for a segmentation map refinement step later.

[108] Thus, identifying a plurality of patches may comprise: grouping together a plurality of patches based on the patches being spatially close together in the received image. This may 26 ensure that the plurality of patches are not selected from very spatially disparate places in the image, where they are less likely to relate to the same object.

[109] Figure 6B schematically illustrates an example of how the cosine similarity distance is calculated.

[110] If pseg™™ is empty, the patch which is closest to the prototype: pseg™™ = argminp. dist^Qi,, protoc) is chosen. To account for cluttered scenes with similar objects, pseg™™ is split into L connected subsets (pseg^, ...,pseg^, thus generating L candidates for the location of the object c in the image 104.

[111] Calculating class scores. For each candidate set of patches pseglc, I = 1,..., L, the class score is found via patch prototype distance with the support image 102: scored := dist(avg(Qtj | Pij G pseglc), protoc). (9)

[112] The candidate lmax with the maximum class score is chosen as the predicted segmentation map psegc := [J {Pi:] | Pij G pseglcmax] (10) i,j and classification score scorec: = maxiscorel-

[113] From the class score scorec, a determination is made as to whether object c is in the image 104. Similar to other score-based approaches in open-set classification, a classification threshold needs to be selected for a given dataset to control which predicted masks psegc would be accepted, and which ones would be rejected. In the actual implementation, the classification threshold needs to be selected for each scenario empirically, while in what follows the capability of the method to separate positive examples from negative examples is measured via score precision metric (as described below).

[114] Thus, the step of S504 in Figure 5 of determining, using the at least one class prototype, a likelihood of the selected patches representing the personal object of the at least one class prototype, may comprise: for each class prototype for which a plurality of the selected patches are retained: calculating a class score by averaging the generated feature vector for the selected plurality of patches for the class prototype; comparing the class score with the feature vector of the class prototype to determine whether the class score is greater than or equal to a classification threshold for the class prototype; and determining, when the class score is greater than or equal to the classification threshold, that the selected patches contain the personal object corresponding to the class prototype.

[115] In some cases, selecting (step S502) a plurality of patches may comprise selecting a plurality of sets of patches, each set comprising at least one patch. That is, multiple sets of patches may be considered, which can be useful for accurately determining the edge(s) of the personal object, because one set of patches may better identify the object and its edge(s) than another.

[116] When two or more sets of patches are determined to contain the personal object corresponding to a class prototype, the method of Figure 5 may comprise: selecting a set of patches, from the two or more sets of patches, with a maximum class score.

[117] Segmentation map refinement. The predicted map psegc may be used as a segmentation map for the object of interest, the map usually covers only part of the object or contains holes. To capture the whole object, the patches from psegc with clusters obtained from the previous step are refined using / c-means by: pseg™^ := [J {%r° | n psegc * 0} (11)

[118] Thus, outputting a location of the personal object (step S506 in Figure 5) may comprise outputting a segmentation map that segments the personal object in the image from a remainder of the image. The segmentation map helpfully isolates the personal object. The segmentation map may be helpful for a user device’s machine vision to understand where the personal object is, particularly if the user device needs to interact with (or avoid interacting with) the personal object once it has been identified.

[119] Bounding box from segmentation map. A detection bounding box may be genereated using the refined segmentation map pseg^ by taking the extreme coordinates of the segmentation map.

[120] Thus, the method of Figure 5 may further comprise: adding a bounding box to the received image, where the bounding box contains the personal object depicted by the segmentation map. The bounding box may be useful for a human or a machine to understand where the personal object is located in the image.

[121] Figure 6C is a block diagram showing an example of how a personalized ML model may be used to perform image classification. In this example, the query image 104 depicts two dogs, where one of the dogs is the pet of a user. That is, in this example, the query image 104 depicts one object which is personal to the user of a user device. As noted above, in a first step, the pre-trained feature extractor 100 generates a feature map from the query image 104, as described in Figure 3. The personalized ML model comprises class prototypes associated with personal classes depicted in the support image 102, which are obtained as a result of the support image processing described above in relation to Figures 4A to 5. The class prototypes are used to classify the query image 104 by determining whether any personal classes are depicted in the image. In this example, there is a support image 102 which is labelled with the personal classification “Archie” corresponding to the user’s pet dog. Therefore, the support image 102, and any other support images which also depict the user’s pet dog “Archie”, have a class prototype comprising a classification and class threshold for the personal class of “Archie” the pet dog.

[122] As described above, the query image 104 is classified as containing “Archie” by computing a set of classification scores and determining whether any of the computed classification scores are above the classification threshold for the personal class “Archie.” Further optional post-processing steps of personal detection and personal segmentation may be performed. That is, the location of the personal object may be detected in the image e.g. by the generation of a bounding box surrounding the personal object, and a segmentation map based on the location of the personal object may be generated.

[123] A series of experiments for validating the performance of the present techniques will now be described. The evaluation of the present techniques focuses on multi-instance personalization (i.e., adaptation to multiple personal objects) via one-shot transfer on multiple tasks (image classification, object detection and semantic segmentation) and datasets (iCubWorld (Fanello, S. R. et al) and PerSeg (Zheng, R. et al)). As will be shown below, for one-shot segmentation task, the present techniques improve the memory usage by up to 10 x and backbone inference time by up to 100 x compared to few-shot semantic segmentation competitors based on foundational models, while maintaining similar segmentation accuracy, as well as improving segmentation accuracy of lightweight solutions on cluttered scenes by 46%. To evaluate multi-instance identification accuracy, metrics from an open-set recognition task are adopted. Existing segmentation methods are adapted for the multi-instance setup and show that compared to lightweight competitors pre-trained on coarse classes, the present tehcniques achieve 55% identification improvement on simple scenes, and 42% improvement on cluttered scenes.

[124] Datasets. In this section, datasets used for the evaluation of the present techniques will be described. In particular, datasets are chosen which i) contain images of personal objects with different position / scale / background / lighting variations with fine-level class annotations, and ii) include either a segmentation map or bounding box annotations.

[125] Object Detection Datasets for Robotic Applications. As discussed above, object detection and fine-grained identification are crucial tasks for robotic manipulators. To boost the development of object detection methods, several datasets have been introduced. In particular, iCubWorld is a collection of images recording the visual experience of the iCub humanoid robot observing personal user objects in its typical environment, such as a laboratory or an office. CORe50 (Lomonaco, V. et al) further enriches the field, offering a new benchmark for continuous object recognition, designed specifically for real-world applications such as fine-grained object detection in robot vision systems. These datasets align with the present task as they represent scenarios where few-shot personalization can be used to enhance the robot’s ability to recognize new or fine-grained objects, serving as practical representations of the use cases where the present techniques can be applied.

[126] PerSEG. The PerSEG dataset (Zheng, R. et al) is a convenient choice for one-shot segmentation tasks due to a collection of 40 personalized classes and high-quality segmentation maps. The images contain salient objects that take a large part of the image and with simple, non-cluttered backgrounds, making the segmentation and classification task easier compared to other, noisier, datasets. For few-shot evaluation, the first image for each class is taken as a reference image, and test one-shot open-set classification and segmentation on the rest of the images in the class.

[127] iCubWorld. The iCubWorld dataset (Fanello, S. R. et al) is aimed specifically for robotics application of fine-grained object identification. The dataset contains images from several sessions where a single object is being moved in hand across the scene, and several additional sessions where various objects are filmed in cluttered environments. 30

[128] A subset of the sessions within the dataset that contain bounding box annotations are considered, namely i) MIX sessions where 50 personal objects are captured in various poses, scales, and lighting conditions, one session per object, and ii) TABLE, FLOOR1, FLOOR2, SHELF sessions where a subset of personal objects are scattered on the same scene (altogether 19 personal objects are included in those sessions). For the evaluation of our framework, the first image of the MIX session is taken as the support image. Detection and open-set classification accuracy are evaluated separately on the collection of MIX sessions (called iCW-single here) and the collection of cluttered sessions TABLE, FLOOR1, FLOOR2, SHELF (called iCW-cluttered here).

[129] Metrics. In this section, metrics for the personal object search task are presented, which include a localization metric and two open-set identification metrics. Firstly, to measure localization, the common mloU metric (Csurka, G. et al) between ground truth localization and predicted localization is employed for a given personal item on the image: mloU := avgt ^[ / (pseg^jQiXgtseg;)), (12) where (&, gtct, gtseg;) are triplets of reference image, index of a personal object on the image, and ground truth localization (segmentation map for PerSEG and bounding box for iCubWorld) of the object on the image, respectively (for cluttered scenes, different personal objects on the same image would correspond to different tuples with same Qt).

[130] Secondly, to measure identification accuracy (denoted by ACC), the predicted score for the ground truth class is checked to be the highest among candidate locations of comparison classes near the ground truth location: ACC := avgt I acc I argmax (score^^Qi)} I j, (13) where scorelcloc(Qi) is the score of the location candidate of personal class c with the highest intersection with ground truth map gtseg; (the score is 0 if there is no intersecting candidate).

[131] Thirdly, to measure open set identification accuracy, the Average Precision metric across class scores (denoted by cPREC) is employed, which measures how well the class scores for positive examples are separated from the scores from negative examples: cPREC := avgc ^APi (scorelcloc((?;))) ■ (14)

[132] The footprints of the methods are also compared in the form of i) inference time: how much time does it take for a backbone to process a single image; and ii) GPU memory consumption (vRAM): how much GPU memory is required to pass a single image through the backbone, without gradients.

[133] Since the pre- and post-processing steps are done on the CPU, the timings for those steps depend on I / O throughput and specific implementation of those steps. However, since the k-means pre-processing step takes a considerable amount of time in Swiss DINO, the impact of k-means on the time footprint will also be discussed.

[134] Experimental setup. For the experiments, the DINOv2 backbone (version without registers) is used, with input resized to 448x448 resolution, patch size 14, and patch number NP = 32. To measure the footprints, an NVIDIA A40 single GPU is used, with batch size 1 during inference.For segmentation refinement hyperparameters, these are empirically chose to be kQ = 30, aco = 200 for the iCubWorld dataset, and kQ = 150, aco = 200 for the PerSEG dataset. An efficient k-means++ method (Arthur, D. et al) is employed to speed up the clustering step.

[135] Comparison methods. To compare the present techniques against the existing solutions, training-free methods for semantic segmentation or detection are primarily focussed on. To be able to adapt semantic segmentation methods for few-shot prototype-based identification task, methods that provide a feature map or a prototype vector for each predicted segmentation mask or bounding box are chosen.

[136] Matcher / PerSAM / DINOv2+M2F. Matcher and PerSAM are state-of-the-art training-free methods for one-shot semantic segmentation. The methods are based on prompt engineering for a large SAM (Kirillov, A. et al) segmentation model either based on positivenegative pairs or on DINOv2’s features. Matcher and PerSAM with default SAM ViT-h backbones are included, as well as PerSAM with smaller SAM ViT-b backbone to compare the footprint efficiency of those methods. DINOv2+Mask2Former is also considered, using an M2F (Cheng, BH. Et al) segmentation head on top of DINOv2 specifically trained for segmentation on the ADE20k dataset (Zhou, B. et al) with coarse-level classes.

[137] To adapt and evaluate PerSAM and Matcher for multi-class identification, i) the feature map from the respective backbones (DINOv2 for Matcher, SAM for PerSAM) is extracted; ii) the features over the support and predicted query masks are averaged to get the prototypes; and Hi) cosine distance is applied to calculate scorec(Q) for each query image.

[138] To adapt DINOv2+M2F for personal object search task, pre-softmax feature vectors of M2F head are used. The same steps as those of the present techniques are performed, but using DINOv2+M2F as an alternative backbone.

[139] The methods above require a precise segmentation map extracted from the reference image, without good extension to bounding box annotations. Therefore, those methods for the iCubWorld dataset are not considered, which only provides bounding box annotation for the reference images of personal objects.

[140] YOLOv8-seg. YOLOv8-seg is an instance segmentation model based on the state-of-the-art YOLOv8 (Jocher, G. et al) lightweight detection method, and pre-trained on COCO (Lin, Tsung-Yi et al) dataset. It is particularly convenient, since the model outputs feature vectors for each of the candidate bounding boxes and masks.

[141] To adapt the model for the personal object search task, i) the support prototype vector for the bounding box with the highest loll and the ground truth bounding box is extracted; ii) the query bounding box with the closest prototype on the query image is found to get loU score; and Hi) the query prototype from the ground-truth query bounding box is extracted to calculate cPREC and ACC. YOLOv8-seg is applied on the iCubWorld dataset for detection and on the PerSEG dataset for segmentation.

[142] DINOv2 bounding box oracle. To have an upper bound reference for the identification metrics cPREC and ACC, is it assumted that the ground truth location of the personal object on the test image is known. This method is termed “DINOv2 bbox oracle" which has similar computational resources.

[143] Knowing ground truth bounding boxes, support images Sc and a query image Q are cropped into Sbb and Qbb respectively. DINOv2 class tokens (Sbb)c and (Qbb)c are used as prototypes and the class score scorec(Q) is computed as the cosine distance between corresponding prototypes. Then, cPREC and ACC metrics using scorec(Q) may be calculated, as utilized above.

[144] Results on iCubWorld. Figure 7 is a table of results showing the performance of the present techniques on the iCubWorld dataset. As can be seen, Swiss DINO significantly outperform the lightweight comparison method YOLOv8-seg on personal object detection on the iCubWorld dataset. Swiss DINO achieves 16% / 46% IOU improvement on single-object and cluttered scenes, respectively. Swiss DINO also shows significant improvement in personal object identification, showing 55% / 40% cPREC open-set score improvement, and 57% / 52% classification accuracy improvement on single and cluttered scenes respectively.

[145] Significant rnlOU gap in cluttered scenes compared to YOLOv8-seg method is caused by the wrong bounding box picked as the prediction, and the large gap in cPREC and ACC metrics is caused by a large number of false positive predictions near the ground truth location of the object. Given that the adaptation of YOLOv8-seg chooses the bounding box with the feature vector closest to the ground truth class prototype, this shows the poor separation of the feature vectors for fine-grained classes.

[146] Compared to the bounding box-oracle method, on the iCubWorld-single dataset the cPREC and ACC scores of Swiss DINO are only about 5% below the upper bound, while on the iCubWorld-cluttered dataset a more significant 25% accuracy gap is observed, likely due to smaller object scale and presence of similar objects on the cluttered images.

[147] Results on PerSEG. Figure 8 is a table of results showing the performance of the present techniques on the PerSEG dataset. As can be seen Swiss DINO also outperforms YOLOv8-seg on semantic segmentation on the PerSEG dataset in terms of personal object identification metrics (50% cPREC improvement) and classification accuracy (48% improvement), while maintaining similar computational footprint and slightly smaller loll on segmentation maps.

[148] Compared to DINOv2+M2F, Swiss DINO shows 25% loU, 45% cPREC, and 49% ACC improvement, while using a much smaller backbone. This again shows how fine-tuning of segmentation head on a coarse-level dataset harms discriminative properties of the features in personalized scenarios. Compared to Matcher and PerSAM-b / h, Swiss DINO achieves 100 x backbone inference time speedup and 10 x improved GPU memory usage while maintaining competitive segmentation and identification scores.

[149] Overall, the results show outstanding capabilities for zero-shot transfer of DINOv2 feature maps onto new tasks (i.e., segmentation and detection) and personalized classes compared to other backbones trained on large datasets, namely: CNN-based YOLOv8 architecture, specialized Mask2Former segmentation head and SAM foundation model.

[150] Impact of k-means. In the on-server implementation of the present techniques, most of the inference time is spent on the k-means pre-processing step for the query images, which is done on the CPU. This is because k-means is performed on Np = 1024 feature vectors of D = 384 / 768 / 1024 dimensions from vit-s / b / l backbones respectively.

[151] The k-means method takes 0.19 / 0.23 / 0.25 seconds per query image for vit-s / b / l backbones respectively, which is about 95% of the overall inference time, compared to about 10 milliseconds spent on query feature map extraction, and 1 millisecond spent on the rest of the query post-processing. The experiments were performed on 64 Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz cores.

[152] Note that the k-means refinement step for query images is optional and does not affect identification metrics cPREC and ACC, since the class score scorec(Q) is computed using the non-refined segmentation map. Therefore, the non-refined version is perfect for applications that only need partial localization information (e.g., a single point on the object of interest). For mIOU scores for segmentation maps without refinement (see Figure 10 below). Also note that even with costly refinement step, the overall inference time is still 2 x lower compared to heavy-weight methods like PerSAM and Matcher, while still maintaining significant gains by 10 x on vRAM footprint.

[153] ViT backbone ablations. In this section, the choice of DINOv2 as a backbone is motivated by comparing the accuracy of the method using popular self-supervised vision transformer backbones: DINO, DeiT (Touvron, H. et al) and OpenCLIP (Cherti, M. et al). Figure 9 is a table of results of abliation studies for different backbones on the iCW-single dataset. As can be seen, DINOv2 outperforms other backbones by 7-11% in loll and 1-5% in cPREC metrics, while maintaining similar or better computational footprints.

[154] Hyperparameter ablations. Figure 10 is a table of results of ablation studies for different choices of hyperparameters. In this section, the effect of hyperparameters is analyzed. In particular, it is seen how kQ and aco used in segmentation mask refinement affect loU score for the iCW-single, iCW-cluttered, and PerSEG datasets. As can be seen, reducing the number of clusters KQ improves the loU score by 32% / 42% on the iCW single and cluttered respectively, while the number of clusters needs to be high on the cleaner PerSEG dataset to exclude the false positive patches. Coordinate scaling aco does not affect the accuracy metrics much (yielding 5% improvement on cluttered scenes), however, it can be seen from qualitative observations that the inclusion of coordinate scaling makes the results more robust to scene variation. Following this result, the following hyperparameter choices are made: kQ = 30, aco = 200 for the iCubWorld dataset, and kQ = 150, aco = 200 for the PerSEG dataset.

[155] In summary, the present techniques encompass the formulation of a novel problem and metrics for the personal object search task, which is directly related to practical robot vision tasks performed by mobile and robotic systems such as home appliances and robotic manipulators, in which the system needs to localize all present objects of interest in a cluttered scene, where each object is only referenced by a few images. To address this task, the present techniques leverage SSL-pretrained DINOv2’s feature maps having strong discriminative and localization properties. Swiss DINO presents novel clustering-based segmentation / detection mechanisms to alleviate the need for additional specialized modules for such dense prediction tasks.

[156] The present techniques are compared to common lightweight solutions, as well as heavy transformer-based solutions. A significant improvement (up to 55%) of segmentation and recognition accuracy is shown compared to the former common lightweight methods, and significant footprint reduction of backbone inference time (up to 100 x) and GPU consumption (up to 10 x) compared to the latter heavy trasnformer-based methods, allowing seamless implementation on robotic devices.

[157] Altogether, the present techniques show the power and versatility of self-supervised transformer models on personal object search and various downstream tasks. A potential future extension of the present techniques is an extension to the continual learning of new generic as well as new personal objects.

[158] There are a variety of use cases for the techniques described above. For example, they may be used in home appliances such as fridges, ovens, washing machines and robot vacuum cleaners. By using the personalisation techniques, the home appliances may have improved recognition of objects for both objects which are not detected by an original baseline model which is typically used on such devices, when there is occlusion of an object, e.g. due to overlapping or adjacent objects or when there are different scales of objects. When an object is not detected by the original baseline model, the home appliance can update its knowledge to recognise new classes (new objects) using a few shots only (including as few as one shot) as described above. Thus, the home appliances can recognise and / or detect new objects which previously may not have been recognised / detected by the home appliances. In a fridge, an example of a new object may be a new food product which the user has not previously inserted into the fridge.

[159] In a fridge, examples of occlusion include occlusion by a user’s hand holding the food product as it is being inserted into or removed from the fridge or occlusion by overlapping food products within the fridge, for example other food products which are close to, in front of or on top of the food product to be identified. By more accurately identifying products despite the occlusions, the fridge may be able to make more accurate predictions of the food products within the fridge due to the part-based matching described above. Thus, the fridge may make more accurate recommendations regarding food products to be purchased.

[160] For a robot vacuum cleaner, examples of occlusion include a target object (e.g. a lamp or a plant) behind another object (e.g. a sofa), a table behind a chair and so on. The target object may be an object around which a user has instructed the vacuum cleaner to clean, an object which the user has instructed the vacuum cleaner to avoid or an object such as a pet which is to be watched. By using the personalisation techniques, the robot vacuum cleaner may have improved recognition of objects despite occlusion. It will be appreciated that scale of an object may also be important, for example two potted plants may have similar appearances but different sizes, and thus by improving personalisation to account for scale, there will be improved accuracy in the recognition.

[161] Figure 11 is an example of a method which may be implemented in a home appliance or another user device such as a smart phone. At step S1100, the user device receives an image depicting at least one object to be classified. At step S1102, each object in the image is detected. For example, at step S1100, a smart oven may receive an image of some food (e.g. a roast or a tart) which has been inserted in the oven and may then detect all the objects, e.g. ingredients, at step S1102. Similarly, a washing machine may receive an image of some clothes which have been inserted in the machine for washing at step S1100 and at step S1102 may detect all the cloth types for the clothes which have been inserted.

[162] At step S1104, the user device determines whether each detected object corresponds to a known class protype. The determination may be done as described above, for example in Figure 5. The known class prototypes may include pre-defined classes for a standard model, e.g. cotton, wool for the washing machine example or user specific classes which have been added as described above, for example as described, in relation to Figure 2A. If the classes are known, there is an output at step S1106. The output may be label (e.g. class) and / or location for each known object (where the location is important). Once there is an output, the user device may make a recommendation based on the output at step S1108. For example, the washing machine may output a recommendation for a washing program or an oven may output a recommendation for a cooking program or a recipe. These recommendations may be implemented automatically for some user devices.

[163] When the classes for one or more identified objects are not known, the user device may implement personalisation of the model as described above. For example, the image may be annotated (e.g. by the user) for the new object(s) at step S1116 and then the model may be updated with the new class(es) at step S1120. In other words, there is class-incremental recognition. For example, the washing machine may be personalised to recognise one or more new cloth types such as a delicate cloth and an oven may be trained to recognise one or more new ingredients at each use. The next time that the new class(es) are presented to the user device, they will then be recognised and steps S1106 and S1108 will be carried out. In the washing machine example, this means that the new delicate cloth will be correctly identified and thus the delicate wash cycle will be recommended and not wash as synthetic or another incorrect recommendation. This overcomes an issue in known systems, in which even when the user provides feedback, the user device continues to make incorrect recommendations for objects which are not correctly recognised by the user device.

[164] In a similar manner to the home appliances, the personalisation described above may be used in applications running on user devices such as smart phones or similar devices. For example, the application Bixby™ vision which provides descriptions for input image of the depicted scenes including identifying objects may be improved to recognise personal user objects within the scene. In other words, instead of an input image of a dog simply being described as “a dog sitting on the grass”, the input image may be described as “Rocco’s dog (Archie) sitting on the grass”. Alternatively, in response to a request to identify an object, the application Bixby™ vision may state “your kitchen clock” rather than simply “clock”. As another example, image editing may be enhanced to allow not just selection of whole objects from a scene as may currently be done but additionally / alternatively selection of parts of user-specific objects from within a scene. Image editing may be combined with generative Al methods. Another example of an application is a gallery search of personal classes. As an final example, the above techniques may also be incorporated in robotic appliances, e.g. to improve grasping of a specific objection.

[165] References: • Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots, “One-shot learning for semantic segmentation,” arXiv preprint arXiv:1709.03410, 2017. • Khoi Nguyen and Sinisa Todorovic, “Feature weighting and boosting for few-shot segmentation,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2019, pp. 622-631. • Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li, “Personalize segment anything model with one shot,” in ICLR, 2024. • Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen, “Matcher: Segment anything with one shot using all-purpose feature matching,” in ICLR, 2024. • Maxime Oquab, Timothee Darcet, Theo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al., “Dinov2: Learning robust visual features without supervision,” arXiv:2304.07193, 2023. • Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin, “Emerging properties in self-supervised vision transformers,” in ICCV, 2021, pp. 9650-9660. • Oriane Simeoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Perez, Renaud Marlet, and Jean Ponce, “Localizing objects with self-supervised transformers and no labels,” arXiv:2109.14279, 2021. • Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi, “Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization,” in CVPR, 2022. • Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li, “Personalize segment anything model with one shot,” in ICLR, 2024. • Sean Ryan Fanello, Carlo Ciliberto, Matteo Santoro, Lorenzo Natale, Giorgio Metta, Lorenzo Rosasco, and Francesca Odone, “iCub World: Friendly Robots Help Building Good Vision Data-Sets,” in CVPRW, 2013. • Gabriela Csurka, Diane Larlus, Florent Perronnin, and France Meylan, “What is a good evaluation measure for semantic segmentation?.,” in BMVC, 2013, vol. 27, pp. 10-5244. • David Arthur, Sergei Vassilvitskii, et al., “k-means++: The advantages of careful seeding,” in Soda, 2007, vol. 7, pp. 1027-1035. • Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al., “Segment anything,” ICCV, 2023. • Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar, “Masked-attention mask transformer for universal image segmentation,” in CVPR, 2022, pp. 1290-1299. • Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba, “Scene parsing through ade20k dataset,” in CVPR, 2017, pp. 633-641. • Glenn Jocher, Ayush Chaurasia, and Jing Qiu, “Ultralytics YOLOv8,” 2023. • Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick, “Microsoft coco: Common objects in context,” in ECCV, 2014, pp. 740-755. • Vincenzo Lomonaco and Davide Maltoni, “Core50: a new dataset and benchmark for continuous object recognition,” in Conference on robot learning. PMLR, 2017, pp. 17-26. • Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou, “Training data-efficient image transformers &distillation through attention,” in International conference on machine learning. PMLR, 2021, pp. 10347-10357. • Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel llharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev, “Reproducible scaling laws for contrastive language-image learning,” in CVPR, 2023, pp. 2818-2829. 5

[166] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the 10 embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

1. A computer-implemented method for personalising, on a user device, a trained machine learning, ML, model for classifying images, the method comprising:receiving a personalisation dataset comprising at least one data pair, each data pair comprising:an image depicting an object which is personal to a user of the user device, andlabel data comprising location information indicating a location of the object in the image, and a label for the object;for each data pair in the personalisation data set:generating, using a trained feature extractor of the ML model and the location of the object in the image, a feature map for the object in the image of the data pair;determining, using the generated feature map for the object, a class prototype comprising a feature vector for the object and a class threshold that enables selection of relevant patches of interest on query images to be classified, wherein the class threshold indicates how far apart, in vector space, a feature vector for a query image can be from the feature vector of the class prototype while still representing the object; andassigning, to each class prototype, the label for the object; andstoring, on the user device, the class prototype and assigned label for use by the ML model, thereby personalising the ML model to the user of the user device.

2. The method as claimed in claim 1 wherein the location information corresponds to a segmentation map that isolates the object in the image, and wherein determining the class prototype comprises:selecting a plurality of patches of the image using the segmentation map, wherein each patch in the identified plurality of patches at least partially intersects with the segmentation map.

3. The method as claimed in claim 1 wherein the location information corresponds to a bounding box around the object in the image, and wherein determining the class prototype comprises:generating a segmentation map of the image using the bounding box, wherein the segmentation map isolates the object in the image; andselecting a plurality of patches of the image using the segmentation map, wherein each patch in the identified plurality of patches at least partially intersects with the segmentation map.

4. The method as claimed in claim 3 wherein generating the segmentation map comprises: identifying a plurality of object patches and a plurality of background patches of the image, wherein each object patch at least partially intersects with the bounding box and wherein each background patch at least partially borders at least one object patch;combining the plurality of object patches and the plurality of background patches to generate a set of patches;clustering the set of patches into a plurality of patch clusters; andfiltering the plurality of patch clusters to remove patch clusters which include background patches to generate the segmentation map.

5. The method as claimed in any of claim 2 to 4 wherein determining the feature vector of the class prototype comprises:generating a feature vector for each patch of the selected plurality of patches of the image; andaveraging the feature vectors to generate the feature vector for the object.

6. The method as claimed in claim 5 wherein determining the class threshold of the class prototype further comprises:identifying a plurality of negative patches of the image using the segmentation map, wherein each negative patch has an empty intersection with the segmentation map.

7. The method as claimed in claim 6 wherein determining the class threshold of the class prototype further comprises:calculating, for each patch of the selected plurality of patches, a positive patch distance corresponding to a distance in feature space between the feature vector of the class prototype and the feature vector of the selected patch;determining a positive patch distance threshold from the calculated positive patch distances;generating a feature vector for each patch of the plurality of negative patches of the image;calculating, for each negative patch of the plurality of negative patches a negative patch distance corresponding to a distance in feature space between the feature vector of the class prototype and the feature vector of the negative patch;determining a negative patch distance threshold from the calculated negative patch distances; anddetermining the class threshold of the class prototype from the positive patch distance threshold and the negative patch distance threshold.

8. The method as claimed in claim 7 wherein determining the class threshold of the class prototype further comprises:filtering, using the calculated positive patch distances, the selected plurality of patches having a positive patch distance over a predefined positive distance value;filtering, using the calculated negative patch distances, the plurality of negative patches having a negative patch distance below a predefined negative distance value;determining the maximum positive patch distance of the remaining patches from the selected plurality of positive patches;determining the minimum negative patch distance of the remaining patches from the selected plurality of negative patches; andsetting the maximum positive patch distance as the positive patch distance threshold and the minimum negative patch distance as the negative patch distance threshold.

9. A user device for performing personalised object recognition, the user device comprising: at least one processor coupled to memory, for:receiving a personalisation dataset comprising at least one data pair, each data pair comprising:an image depicting an object which is personal to a user of the user device, andlabel data comprising location information indicating a location of the object in the image, and a label for the object;for each data pair in the personalisation data set:generating, using a trained feature extractor of a trained ML model for classifying images and the location of the object in the image, a feature map for the object in the image of the data pair;determining, using the generated feature map for the object, a class prototype comprising a feature vector for the object and a class threshold that enables selection of relevant patches of interest on query images to be classified, wherein the class threshold indicates how far apart, in vector space, a feature vector for a query image can be from the feature vector of the class prototype while still representing the object; andassigning, to each class prototype, the label for the object; andstoring, on the user device, the class prototype and assigned label for use by the ML model, thereby personalising the ML model to the user of the user device.

10. A computer-implemented method for personal object detection, on a user device, using a personalised, trained machine learning, ML, model comprising at least one class prototype for a personal object, the method comprising:receiving an image depicting at least one object to be classified; andfor each object depicted in the received image:selecting a plurality of patches of interest in the received image;determining, using the at least one class prototype, a likelihood of the selected patches representing the personal object of the at least one class prototype; andoutputting, when the determined likelihood for the selected patches is above a classification threshold for one class prototype of the at least one class prototype, wherein the classification threshold indicates how far apart, in vector space, a feature vector for a query image can be from the feature vector of the class prototype while still representing the object:a label for the personal object corresponding to a label associated with the class prototype for which the determined likelihood is above the classification threshold, anda location of the personal object in the received image.

11. The method as claimed in claim 10 wherein outputting a location of the personal object comprises outputting a segmentation map that segments the personal object in the image from a remainder of the image.

12. The method as claimed in claim 11 further comprising:adding a bounding box to the received image, where the bounding box contains the personal object depicted by the segmentation map.

13. The method as claimed in claim 10, 11 or 12 further comprising:generating, using a feature extractor of the ML model, a feature map for the received image.

14. The method as claimed in claim 13 wherein the selecting comprises:identifying a plurality of patches;generating, using the feature map, a feature vector for each selected patch;calculating, for each identified patch, a distance between the generated feature vector and a feature vector of the at least one class prototype; andfor each class prototype:determining whether the calculated distance is within the class threshold of the class prototype; andretaining an identified patch as a selected patch when the calculated distance for that patch is less than the class threshold for the class prototype.

15. The method as claimed in claim 14 wherein identifying a plurality of patches comprises: grouping together a plurality of patches based on the patches being spatially close together in the received image.

16. The method as claimed in claim 14 or 15 wherein determining, using the at least one class prototype, a likelihood of the selected patches representing the personal object of the at least one class prototype, comprises:for each class prototype for which a plurality of the selected patches are retained: calculating a class score by averaging the generated feature vector for the selected plurality of patches for the class prototype;comparing the class score with the feature vector of the class prototype to determine whether the class score is greater than or equal to a classification threshold for the class prototype; anddetermining, when the class score is greater than or equal to the classification threshold, that the selected patches contain the personal object corresponding to the class prototype.

17. The method as claimed in claim 14, 15 or 16 wherein selecting a plurality of patches comprises selecting a plurality of sets of patches, each set comprising at least one patch.

18. The method as claimed in claim 17 wherein when two or more sets of patches are determined to contain the personal object corresponding to a class prototype, the method comprises:selecting a set of patches, from the two or more sets of patches, with a maximum class score.

19. The method as claimed in claim 18 wherein outputting a location of the personal object comprises outputting a segmentation map, based on the selected set of patches with amaximum class score, that segments the personal object in the image from a remainder of the image.

20. The method as claimed in any of claims 11 to 19 wherein when the determined likelihood for the selected patches is below a classification threshold for the at least one class prototype, the method comprises:outputting a negative classification result for the object indicating that the object is not a personal object.

21. The method as claimed in claim 20 further comprising:performing a coarse classification of the object using the trained ML model; and outputting from the trained ML model a coarse class name for the object.

22. A computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out the method of claims 1 to 8, or 10 to 21.

23. A user device for performing personalised object recognition using a personalised, trained machine learning, ML, model comprising at least one class prototype for a personal object, the user device comprising:at least one processor coupled to memory, for:receiving an image depicting at least one object to be classified; andfor each object depicted in the received image:selecting a plurality of patches of interest in the received image;determining, using the at least one class prototype, a likelihood of the selected patches representing the personal object of the at least one class prototype; andoutputting, when the determined likelihood for the selected patches is above a classification threshold for one class prototype of the at least one class prototype, wherein the classification threshold indicates how far apart, in vector space, a feature vector for a query image can be from the feature vector of the class prototype while still representing the object:a label for the personal object corresponding to a label associated with the class prototype for which the determined likelihood is above the classification threshold, anda location of the personal object in the received image.

Images