Method and System for Training Models

Abstract

Method of personalising the classification of an image, comprising: receiving an image 208; obtaining a coarse first classification for an object in the image and a set of intermediate object features

Description

Field

[001] Embodiments of the present techniques relate to a method and a system for training models, including fine-tuning on device using few-shot instance-level personalisation. The models may be used for classifying images, for example for processing images to control a device, e.g. a home appliance. Background

[002] Object Detection (OD) systems have been a staple in robotics and on a variety of personal devices such as home appliances (e.g., robot vacuum cleaners) and personal assistants for many years. This has highlighted a challenge in their design, i.e., they cannot efficiently update their knowledge to distinguish between general classes and user-specific instances (e.g., a dog vs. user’s dog). We refer to this challenging task as Instance-level Personalized Object Detection (IPOD).

[003] Indeed, several applications require precise localization and recognition of the surrounding objects {e.g., for navigation or object grasping). While most of them need only coarse-level classes (e.g., person, door, plant, etc.), emerging user-centric scenarios demand fine-grained detection abilities to identify personal instances (e.g. , user’s friend, kitchen door, chamomile). For example, a user watching the television may ask an assistive robot to retrieve its remote, rather than the remote for the hi-fi; a robot smart vacuum cleaner to clean the floor in front of a specific piece of furniture; a lawn mower to avoid specific plants or flowers; a phone to find photos of the user’s pets, and so on. We envision that OD systems should allow instance-level personalized understanding (e.g., dog vs. user’s dog) when deployed on resource-constrained robotic devices through a very limited set of user-provided reference samples. We define this task as Few-Shot Instance-level Personalized Object Detection (FS-IPOD).

[004] The personalization task requires many samples for model tuning and optimization in a centralized server, raising privacy concerns. An alternative is provided by approaches based on recent large-scale Foundation Models, but their compute costs preclude on-device applications. In other words, the task referred to as FS-IPOD presents various challenges, chief among which are privacy concerns related to personal data processing. Viable solutions, indeed, should update the OD system on the device itself, while learning with a few samples provided by the user for training.

[005] Regrettably, deep learning based OD systems cannot efficiently update their knowledge once pre-training is complete, and fine-tuning them is typically very computationally expensive. To cope with this, several post-training few-shot learning (FSL) strategies have been proposed, for example as described in “Prototypical networks for few-shot learning,” by Snell et al published in Proc, of NIPS, 2017. These approaches, however, require the features processed by the architecture to be descriptive enough to distinguish between instances.

[006] The present applicant has identified the need for an improved technique for personalising machine learning models. Summary

[007] In a first approach of the present techniques, there is provided a computer-implemented method for classifying, on a user device, an image using a machine learning, ML, model, the method comprising: receiving an image; obtaining, using an object detector of the ML model, a first classification for at least one object in the input image; generating, using a feature extractor of object detector, a set of intermediate features for the image; converting, using a translator of the ML model, the set of intermediate features into a set of auxiliary features, wherein the set of auxiliary features is defined within an auxiliary feature space which has a different spatial resolution to an intermediate feature space for the set of intermediate features; generating, using a pooling module of the ML model, an embedding vector which encodes at least some of the set of auxiliary features; and identifying, using a classifier of the ML model, a second classification for the at least one object in the image by selecting, from a storage on the user device, a plurality of stored vectors each of which have a first classification and a second classification, wherein the first classification of each selected vector is the same as the first classification obtained using the object detector; comparing the generated embedding vector to the selected plurality of stored vectors; identifying the stored vector which is most similar to the generated embedding vector; and assigning the second classification of the identified stored vector to the at least one object.

[008] Assigning the second classification to the at least one object typically happens in realtime (i.e. within seconds) because of the efficient architecture in which the parameters of the ML model, including the translator, are fixed. As explained in more detail below, the translation parameters are learned at pre-training time so there is no need for expensive on-device tuning of the architecture. Nevertheless, by changing (or translating) the set of intermediate features into a set of auxiliary features, we are able to retain the richness of the feature set before it is passed to the classifier to generate the second classification.

[009] The image may be obtained from a camera on the user device and an output based on the assigned classification may be generated and may comprise controlling the user device based on the assigned classification. Controlling may comprise outputting an edited version of the image, e.g. with the second classification and optionally a bounding box. Controlling may include moving the user device, or controlling the user device to respond to a user request. The user device may be a robotic device, such as an indoor domestic and service robot like a vacuum cleaner. For example, the personalised model may be used to navigate the movement of a robot in a user’s home. For example, the user can ask the robot to clean a specific location and the robot can use the personalised model to identify the specific location (e.g. room). Another use case is in a robotic lawnmower. A user may instruct the lawnmower to ignore specific flowers and plants and the robotic lawnmower can use the personalised model to automatically identify these flowers and plants and to cut the grass around them. There are also industrial uses, for example in a robotic inventory manager. The robot may be instructed to retrieve the scaffolding (also termed pallet) containing a specific item and the robot can use the personalised model to automatically identify and retrieve the correct pallet.

[010] It will be appreciated that the control may be a separate aspect and thus in another approach of the present techniques, there is provided a method for controlling an electronic device using a trained machine learning, ML, model (where the model has been trained using the techniques described herein), the method comprising: receiving an input image for processing, the input image depicting at least one object; processing the input image by: obtaining, using an object detector of the ML model, a first classification for at least one object in the image; generating, using a feature extractor of object detector, a set of intermediate features for the image; converting, using a translator of the ML model, the set of intermediate features into a set of auxiliary features, wherein the set of auxiliary features is defined within an auxiliary feature space which has a different spatial resolution to an intermediate feature space for the set of intermediate features; generating, using a pooling module of the ML model, an embedding vector which encodes at least some of the set of auxiliary features; and identifying, using a classifier of the ML model, a second classification for the at least one object in the image by selecting, from a storage on the user device, a plurality of stored vectors each of which have a first classification and a second classification wherein the first classification of each selected vector is the same as the first classification obtained using the object detector; comparing the generated embedding vector to the selected plurality of stored vectors; identifying the stored vector which is most similar to the generated embedding vector; and assigning the second classification of the identified stored vector to the at least one object; and controlling the user device based on the assigned second classification.

[011] The first classification may be a coarse-level classification Cc representing a class of objects and for each class of objects, there is at least one the second classification which may be a fine-grained classification representing a user-specific instance of an object within the class. A user of the user device typically wants personalised object recognition services (e.g. the ability to add new classes to the model or to personalize the users’ instances of objects) on the user device. The method described above extracts representative embedding vectors from an auxiliary feature space via pooling and employs them for personalized instance-level retrieval with minimal computation. In other words, we have a ML model which includes a conditional coarse-to-fine classifier. The object detector may be considered to be personalized to instance-level classes for their significant real-world implications in various robot systems such as humanoid, service, and rescue robotics.

[012] The ML model which is used in the classification comprises an object detector which includes a feature extractor (e.g. an encoder), a translator (which may also be termed a translator block or translation module), a pooling module and a classifier (which may also be termed a few shot learning module). The object detector may be considered to generate a primary output, e.g. the first classification and may thus be considered to be a primary neural network. The object detector may also be used to obtain a bounding box around each object for which a first classification was obtained. The translator, pooling module and classifier of the ML model may also be used to generate a secondary output, in this case the second classification and may be considered to form a secondary neural network. Using such components may be considered to be using a modular approach and may provide a computationally efficient framework for on-device deployment.

[013] The generated embedding vectors are compared with vectors which are stored on the user device (i.e. locally stored). As explained in detail below, this means that during the training process, personalisation of the ML model may be achieved without sharing data with a central server to preserve data privacy. The user device may have a relatively small storage and thus, the stored vectors are preferably lightweight and thus require less storage than the images. By using an embedding vector, there is also no need to store private replay user data (e.g. the images themselves) on the user device. The use of such vectors may thus result in efficiencies may be with respect to storage and / or memory requirements as well as training and / or inference speed.

[014] When identifying the second classification, each embedding vector is compared to stored vectors which have the same first classification as the first classification which is obtained. That is, we limit the search space of the fine-level class (i.e. second classification) on the basis of the coarse class (first classification) predicted by the detector (rather than searching among all possible options). This allows our method to be about three times faster than the standard non-conditional FSL classifier, without significant impact on the reclassification accuracy.

[015] Converting each set of intermediate features may comprise changing, using a spatial differential module, the spatial resolution. Converting each set of intermediate features may comprise changing, using a channel differential module, a channel depth by mapping the intermediate feature space to the auxiliary feature space so that a channel depth of the intermediate feature space matches the channel depth of the auxiliary feature space. Thus, the translator may comprise a spatial differential module and a channel differential module. The channel differential module (Dc) and spatial differential module (Ds) may together translate the intermediate feature space into an auxiliary space where all feature maps have the same resolution and channel depth. One expression for this translation may be: Ri = lj=iDs(Dc^j^ Vi = 1,2.....n. where R; is the set of auxiliary features in the auxiliary feature space, F, is set of intermediate features in the auxiliary feature space, Ds is the operation by the spatial differential module and Dc is the operation by the channel differential module. It will be appreciated that the order of the operations by the modules may typically be reversed when the operations are independent.Ri may be considered to be the full translated feature map and R; is a partially translated feature map.

[016] The resolution may also be termed the size of the feature space. For example, each input image may have a resolution defined by the height (H) and width (W), e.g. H x W, the set of intermediate features has a resolution defined by H' x W' and the set of auxiliary features has a resolution defined by H” x W". In other words, all three resolutions may be different. The channel depth represents the number of channels in each neural network (teacher or student ML model). For example, in image processing, each channel represents a colour, e.g. red, green, blue. Each set of intermediate features may have a channel depth of lE and the set of auxiliary features may have a channel depth of l0. These channel depths may also be different. As explained in more detail below, the translator can be used to generate an auxiliary feature space in which features generated by a teacher model (e.g. a self-supervised model such as DINOv2) can be distilled without impacting the performance of the object detector. Thus, the target channel depth and resolution for the auxiliary feature space may be determined by the channel depth and resolution of a teacher feature space. The channel differential module may be considered to be unifying two feature spaces which have different channels. Similarly, the spatial differential module may be considered to be unifying two feature spaces which have different spatial resolutions.

[017] Generating an embedding vector for each bounding box may comprise using each bounding box to identify features within the set of intermediate features which are to be summarised into the embedding vector. Each bounding box may be labelled as a list of coordinates, e.g. (x0,y0>xnyi) which specify the corners of the bounding box. A suitable framework for generating the embedding vectors vk may be: *i-i y^i 1 1 y RJ^y] V Yk = (x0,yo> Xi,yi, c,p) e M(X). where x and y are coordinates, i.e. pixel locations, of the bounding boxes, c is the predicted coarse class, p is the prediction confidence (the probability of class assignment), Yk are the predicted bounding boxes which are output from the object detector and only Ri of the set of auxiliary features R( is used. It is noted that xO <x1 and yO <y1, conventionally x direction is along the image width and increases left-to-right, while y direction is along the image height and increases up-to-down.

[018] Prior to receiving an input image, the method may comprise locally training a pre-trained ML model to personalise the ML model. In other words, the user device may receive a pretrained ML model, for example from a server, and perform some fine-tuning to personalise the received ML model. Local training may be done using few-shot learning and thus there may only be a small number of training images, e.g. between 1 and 5 training images. Local training may comprise receiving a training image, e.g. from a camera on the user device; obtaining, using the object detector of the pre-trained ML model, a bounding box around at least one object in the training image and a first classification for the at least one object in the bounding box; generating, using the feature extractor of the object detector, a set of intermediate features for the training image; converting, using the translator of the pre-trained ML model, the set of intermediate features into the set of auxiliary features; generating, using the pooling module of the pre-trained ML model, an embedding vector for each bounding box by encoding from the set of intermediate features within the bounding box; receiving user input in the form of a selection of a bounding box and a second classification for the selected bounding box; and training the classifier of the pre-trained ML model to personalise the ML model using the received user input, wherein parameters of the object detector are frozen during training of the classifier.

[019] This local training can be done as an independent method. Thus, according to another aspect of the present techniques, there may be provided a computer-implemented method for locally training a pre-trained ML model to personalise the ML model, wherein local training comprises: receiving, at a user device, a pre-trained ML model comprising an object detector, a translator, a pooling module and a classifier; receiving a training image; obtaining, using the object detector, a bounding box around at least one object in the training image and a first classification for the at least one object in each bounding box; generating, using a feature extractor of the object detector, a set of intermediate features for the training image; converting, using the translator, the set of intermediate features into a set of auxiliary features, wherein the set of auxiliary features is defined within an auxiliary feature space which has a different spatial resolution to an intermediate feature space for the set of intermediate features; generating, using the pooling module, an embedding vector for each bounding box by encoding the auxiliary features within the bounding box; receiving user input in the form of a selection of a bounding box and a second classification for the selected bounding box; and training the classifier to personalise the ML model using the received user input, wherein parameters of the object detector are frozen during training of the classifier.

[020] During the local training, parameters of the object detector is frozen and only parameters in the classifier (which is also termed a few-shot learning module) are adapted. Moreover, during the local training, the object detector is used in a similar way to its use at inference to obtain bounding box(es) and / or associated first classification(s), generate a set of intermediate features, translate these into a set of auxiliary features and generate the embedding vector for each bounding box. Accordingly, the description of these steps in the first aspect applies equally to this aspect.

[021] As described in more detail below, the auxiliary feature space allows the classifier to mimic the descriptive features of a teacher model (e.g. a self-supervised learning model). The auxiliary feature space supports few shot learning (e.g. training using a few samples) which is not possible with the intermediate feature space. The translation of the features from the intermediate feature space to the auxiliary feature space allows the gap between the intermediate features and the features from the teacher model to be bridged, perhaps by more than 80%.

[022] Training the classifier may comprise teaching the classifier to associate the embedding vector for the selected bounding box with the second classification provided by the user input. To reduce the computational cost of the operation, the classifier training may not use standard methods such as minimising losses or gradient backpropagation. In other words, the classifier may not contain any optimizable parameters. The training may be simply involve storing a tuple (embedding vector, second classification) in the internal database corresponding to the first (coarse) class predicted originally. In other words, the training may require user interaction to provide the second classification. Each embedded vector may be stored in internal storage on the user device with its associated first and second classifications. For each first classification, there may be a set of stored vectors J>[c] = {p1; p2, ■■■}, each having a different second classification. Each set may be used as the search space in the first aspect.

[023] The method may further comprise deriving an additional vector for each first classification from all the stored vectors having the same first classification; and storing the derived additional vector in the internal storage with the set of stored vectors having the same first classification. This additional vector represents a fallback when a determined embedding vector (also termed a query vector) does not match any stored vectors and hence does not match any second classification (fine class). Merely as an example, the additional vector may represent the centroid (centre) of the cluster of all the stored vectors. If the query vector matches this additional vector, there is no associated fine-grained classification. The method may thus comprise identifying the additional stored vector as the stored vector which is most similar to the generated embedding vector and assigning the second classification of the identified stored vector by computing a new second classification using a distance between in embedding space between the generated embedding vector and the set of stored vectors. As an example, the new second classification may be computed from: 1 V fl r n ) d = -—- >softmax]—----Vi = 1,..., IJ’MI k ld(q.p,) lJIJ c' = argmaxi=1.....|j>[c]|d[j] where d is the distance which is calculated for a set of distance functions D = {d^ d2,...}, q is the query vector q, c is the first classification, J>[c] = {p^ p2,...} is the set of stored vectors p. for the first classification, and c' is the calculated new second classification.

[024] Before fine-tuning the ML model on the user device, there may also be pre-training of the ML model prior to deployment on the user device. Pre-training may comprise using a previously trained teacher ML model to teach the ML model. In this way, we can distill the knowledge of the teacher ML model (e.g. an SSL-pretrained model) into the auxiliary feature space of the detector of the ML model. Pre-training may comprise receiving a set of labelled training image, each of which is an image which is labelled with a bounding box and a first classification for at least one object in the image. For each labelled training input, the method comprises generating, using the teacher ML model, a set of teacher features for the labelled training input, wherein the set of teacher features is defined in a teacher feature space which has a first resolution; generating, using the feature extractor of the object detector, the set of intermediate features for the labelled training input,; converting, using the translator, the set of intermediate features into a set of auxiliary features, wherein the auxiliary feature space has the same resolution as the first resolution; generating, using the pooling module, an embedding vector for each bounding box by encoding the auxiliary features within the bounding box. Training the ML model comprises training the object detector using the set of labelled input data and the obtained bounding boxes and first classifications for each object within each labelled input data and training the feature extractor and the translator using the sets of auxiliary features and the sets of teacher features.

[025] Like the fine-tuning, this pre-training can be done as an independent method and can be applied to another types of neural networks. Thus, according to another aspect of the present techniques, there may be provided a computer-implemented method for pre-training a student ML model using a teacher ML model, wherein pre-training comprises: receiving a set of labelled training inputs. For each labelled training input, the method comprises generating, using the teacher ML model, a set of teacher features for the labelled training input, wherein the set of teacher features is defined in a teacher feature space which has a first resolution; generating, using a feature extractor of an object detector in the student ML model, a set of intermediate features for the input image, wherein the set of intermediate features is defined in an intermediate feature space which has a second resolution which is different to the first resolution; converting, using a translator in the ML model, the set of intermediate features into a set of auxiliary features, wherein the set of auxiliary features is defined in an auxiliary feature space which has the same resolution as the first resolution; generating, using a pooling module in the student ML model, an embedding vector which encodes at least part of the set of auxiliary features; and obtaining, using a primary network within the student ML model, a primary output for the labelled training input. Training the student ML model comprises training the feature extractor and the translator using the sets of auxiliary features and the sets of teacher features and training the primary network using the set of labelled input data and the obtained primary outputs.

[026] The aspect above has been generalised to specify that the student network generates a primary and a secondary aspect. In the specific arrangement of image classification, it will be appreciated that each labelled training input may be an image which is labelled with a bounding box and / or a first classification for at least one object in the image. The method may comprise generating an embedding vector comprises encoding the set of auxiliary features within the bounding box; obtaining a primary output comprises obtaining, using an object detector, a bounding box around at least one object in the training image and a first classification for the at least one object in each bounding box; and training the primary network comprises training the object detector.

[027] Training the feature extractor and the translator may comprise adjusting parameters within the feature extractor and the translator to minimise a loss between the set of auxiliary features and the set of teacher features. The loss may be termed a distillation loss because the auxiliary features are used to distil the knowledge from the teacher model to the ML model) Training the object detector may comprise adjusting parameters within the object detector, for example within both the encoder and decoder to minimise a loss (which may be termed a detection loss) between the obtained bounding box and first classification and the labelled bounding box and first classification. By minimising both the distillation loss and the detection loss for the object detector simultaneously, the overall performance may be improved because adjusting the parameters to account for one loss is likely to affect the other loss.

[028] During the pre-training, parameters of the classifier are frozen and only parameters in the object detector and the translator are adapted. Moreover, during the pre-training, the object detector is used in a similar way to its use at inference and during the local training to obtain bounding box(es) and / or associated first classification(s), generate a set of intermediate features, translate these into a set of auxiliary features and generate the embedding vector for each bounding box. Accordingly, the description of these steps in the aspects above applies equally to this aspect.

[029] During the converting step, multiple sets of auxiliary features may be generated when there are multiple sets of intermediate features in the intermediate feature space. However, the auxiliary feature space will only have the same number of sets of auxiliary features as the teacher feature space. In one example, for example using an oracle SSL as the teacher ML model, there may be only one set of features. It is also noted that for example when calculated using the formula above, may be expressed as the sum of all the other auxiliary features and thus will typically have the highest quality and retain all the information. Thus, only Ri may be used. When the primary and secondary tasks (e.g. object detection and feature extraction of the teacher model) do not match, the auxiliary feature space allows the distiallation of the knowledge from the teacher model into the student model. There is nevertheless no impact on the performance of the primary task (e.g. object detection).

[030] It is also noted that although the detail described above relates to image processing, the approaches can be applied in any scenario where the knowledge of a compute-heavy model needs to be distilled into a more efficient one, especially in the presence of a mismatch in the architectures or in the task performed by the networks.

[031] In each approach, the user device (or client device) may be a constrained-resource device but typically has the minimum hardware capabilities to use a trained neural network / ML model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, a vacuum cleaner, a lawnmower, an image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge). It will be understood that this is a non-exhaustive and non-limiting list of example client devices. For some of these devices, e.g. a vacuum cleaner, there may not be the minimum hardware capabilities to perform the local training and this may be done on a second user device, e.g. a smartphone, tablet, laptop, computer or computing device which has more capability including a user interface for a user to input labelled training data. The second user device may be local to the first user device (e.g. within the same household) whereby the training may be described as local training. The second user device may be connectable to the first user device to transfer the locally trained ML model.

[032] According to another aspect, there may be provided a system comprising a server and a user device which is connected to the server and which comprises a processor, wherein the processor of the user device is further configured to: receive a pre-trained ML model from the server; and locally train the ML model to personalise the model as described above. The server may comprise a processor which is configured to pre-train the ML model prior to deployment on a user device as described above.

[033] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[034] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[035] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[036] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[037] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[038] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD- ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[039] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[040] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[041] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[042] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[043] The ML / AI model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), deep Q-networks, and transformer or visual transformer networks.

[044] The learning algorithm is a method for training a predetermined target device (for example, a user device or client device) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semisupervised learning, or reinforcement learning. Brief description of the drawings

[045] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[046] Figure 1 is a flowchart showing example steps of how a global machine learning ML model may be personalised and used on a user device;

[047] Figure 2 is a schematic diagram of an example cross-architecture system which can be used to pre-train the global ML model;

[048] Figures 3a, 3b and 3c are example images which are input to and output from the global ML model, respectively;

[049] Figure 4 is a schematic diagram showing more detail of the cross-architecture system of Figure 2;

[050] Figures 5a and 5b are two alternative modules which can be used in the arrangement of Figures 2 and 4;

[051] Figure 6 shows more detail of a module which can be used in the arrangement of Figures 2 and 4;

[052] Figure 7 is a schematic illustration of a system implementing the cross-architecture system described above;

[053] Figure 8 is a flowchart showing the steps which may be considered a planning or preparation stage;

[054] Figure 9 is a flowchart showing the server-side steps of the method for pre- training the global ML model;

[055] Figure 10 is a flowchart showing the device-side steps of the method for fine-tuning the global ML model which has been pre-trained as shown in Figure 9;

[056] Figure 11 is a flowchart showing the steps of using the fine-tuned ML model on a device;

[057] Figure 12a is a table showing quantitative results for one- and five-shot learning for each dataset and for the proposed methods and comparative prior art methods;

[058] Figure 12b is a table with a subset of the results from Figure 12a shown as percentage changes; and

[059] Figure 13 plots the score against expansion rate of the ground truth (GT) padding size for non-square and expanded square boxes. Detailed description of the drawings

[060] Broadly speaking, embodiments of the present techniques provide methods, systems and apparatuses fortraining and using ML models including artificial neural networks, e.g. to process images. For example, the training may personalise a ML model by training a classifier of the ML model using few-shot learning. There may also be pre-training of the ML model using cross-architecture knowledge transfer approach. The ML model may be used to generate an output based on a classification identified from an input image. For example, the output may comprise instructions to control the operation of a robot storing the ML model.

[061] Figure 1 is a flowchart showing example steps of how a machine learning ML model may be personalised and used. At step S100, a global model is pre-trained. The global model may be pre-trained (or trained, and the terms will be used interchangeably herein) on a central server. As explained in more detail below, the global model (in this example includes an object detector (OD)) may be pre-trained using a pre-trained teacher model (in this example an oracle SSL feature extractor) which is typically a much larger model. In other words, our system is based on an efficient object detector network, that is modified by our feature Translator Block. This block generates an auxiliary feature space where the knowledge of the pre-trained teacher model is distilled. The auxiliary feature space is used for personalization, while keeping the detection inference unaltered. The training uses a large-scale global training dataset (general purpose dataset) on a pre-defined set of classes. In other words, after the pre-training, we have an OD model that recognizes coarse-level classes Cc (e.g., dog, cat, person, etc.) but also has distilled knowledge of the descriptive features from the teacher model. Model pre-training requires large computational power (for example hundreds of GPUs / TPUs) to update the whole model via large batches. That is, the parameters of the whole model are updated during training. Large storage is also required for the large batch sizes.

[062] At step S102, the global model is deployed from the central server for personalization to the user environment. The global model may be deployed to at least one user device. In other words, the model may be embedded into user devices, e.g., home robots, and shipped to users. The or each user device may be any electronic device which utilizes computer vision. For example, the or each user device may be a smart vacuum cleaner, smart domestic appliance (such as a smart oven, e.g. to recognise new dishes or smart fridge, e.g. to recognise new items in the fridge), smartphone, tablet, laptop or other robotic device. However, it will be appreciated that the user device(s) may be any suitable electronic device. Once deployed to the user device(s), the global model is personalised to generate a personalised model. In particular, we want to expand the class set C = Cc u Cf, to include fine-level classes Cf which are typically more relevant to the user {e.g., dog1, dog2, cat1, etc.), without forgetting the original class set.

[063] User devices such as robots do not have large storage due to privacy and storage constraints and cannot update the whole model due to computational limitations. Large batches are also unavailable, since no storage and low RAM is assumed. This means that the refinement operations must be performed in a compute-efficient manner and with a minimal amount of labeled samples since the end-user will need to provide them. To address these issues, we resort to few shot learning (FSL), where the global model is trained on a chosen scenario with very few samples available for tuning (order of 1-5 samples per class). As explained in more detail below, in contrast to the training of the global model on the central server, there is no update of the parameters of the whole model during the personalisation and only the parameters of the classifier module are updated. The auxiliary features which are generated by the translator module are also used at the personalization stage to compute descriptive embeddings for the objects in the scene. Personalisation in this way requires a small storage, RAM, and computation capability.

[064] As further background, we noted that personalized scene understanding tasks trace their roots in NLP, for example as described in “Motivating personality-aware machine translation,” by Mirkin et al published in Conference on Empirical Methods in Natural Language Processing, 2015, pp. 1102-1108. In these early works, it is recognised that the conversation tone tends to change dramatically for different users, and architectures should adapt accordingly. The first paper ““Personalized image semantic segmentation,” by Zhang et al published in Proc, of ICCV Vision, 2021 to investigate its application to vision tasks was applied the concepts to semantic segmentation. Since then, some few-shot techniques have been introduced, thanks to the development of Foundation Models (FM) such as CLIP which is described in “Learning transferable visual models from natural language supervision,” by Radford et al published in 2021. These architectures allow the use of textual prompts to guide the behavior of vision models. In other examples, there is an idea is to prompt an FM and use it to generate either segmentation masks or descriptive features, which can be used for the personalization.

[065] At step S104, the personalised model is used in the user environment. For example, the personalised model may be used to navigate the movement of a robot in a user’s home. The robot may be a robot vacuum which during inference may identify the user’s pet using the personalised model. The user can ask the robot for the location of the pet using a controller app. Alternatively, the user can ask the robot to clean a specific location using the controller app and the robot can use the personalised model to identify the specific location (e.g. room) for example based on an image of the door. Another use case is in a robotic lawnmower. A user may instruct the lawnmower to ignore specific flowers and plants and the robotic lawnmower can use the personalised model to automatically identify these flowers and plants and to cut the grass around them. There are also industrial uses, for example in a robotic inventory manager. The robot may be instructed to retrieve the scaffolding (also termed pallet) containing a specific item and the robot can use the personalised model to automatically identify and retrieve the correct pallet.

[066] Other uses are in smartphones. For example, in a camera app with text driven detection, the personalised model on the phone can be used to identify the objects (e.g. white flowers) in a cluttered scene from a text prompt by the user. The personalised model can be used to name people in a scene based on a text prompt such as “person” and can give the names. Alternatively, the user can provide these names and the phone can add metadata to the image for faster retrieval in a gallery app. There is a similar use in a camera app with image or text based retrieval. For example, a user can ask for “Fido’s images” and the gallery app can then use the personalised model to retrieve these specific images whilst ignoring images of other dogs. These examples focus on personalizing object detectors to instance-level classes for their significant real-world implications in various robot systems such as humanoid, service, and rescue robotics. Nonetheless, in principle, our approach can be applied in any scenario where the knowledge of a compute-heavy model needs to be distilled into a more efficient one, especially in the presence of a mismatch in the architectures or in the task performed by the networks.

[067] Steps S102 and S104 may be alternated at any time. In other words, as the robot navigates a user’s home, the robot encounters new classes which the robot must accurately classify / recognize to effectively navigate the user environment. In other words, adding new instances / personal classes can be performed asynchronously to allow a user to one or more initial personal classes, use the device for a bit, and then decide to add one or more additional personal (fine-grained) classes.

[068] It is also possible that the method could loop back to step S100 to re-train the global ML model. This may be useful, for example, if the system incorporating the model is to be deployed in a novel environment. For example, a model tuned to recognise domestic animals may fail to identify rare or zoo animals because these are a new class. As well as new classes, there may be a need to learn incremental class changes. For example, a model tuned to recognise dogs may fail to identify cats even though these are domestic animals. In both cases, there could be a new distillation from a teacher network to allow the global ML model (student model) to recognise previously unknown classes (e.g. the new classes or the incremental classes) in the scenes. Such a new distillation to discover new classes will make the system (e.g. the vacuum cleaner) more resilient to environment changes and may allow a successful deployment in new and diverse environments.

[069] A novel environment may also cause old classes to appear different because of a domain shift in the environment. For example, presence of fog can modify the appearance of objects in a scene or scenes appear different due to different light levels (e.g. day or night). In this case, there could be a new distillation from a teacher network to allow the global ML model (student model) to recognise known classes in a different domain. In other words, there may be generalization to a different domain or there may be an incremental learning of a new domain. More generally domain shifts may be caused by illumination (day / night), occlusion, blur (camera focus / dust), viewpoint, object pose, and so on. By distilling information from a new teacher network, the student model is personalised based on the domains and classes which are likely to be seen in the environment in which it is deployed. The personalisation on device will then complement this pre-training.

[070] Figure 2 is a schematic diagram of a cross-architecture system 200 for training an example global ML model 220 in the form of an artificial neural network which can be deployed for example on a user device for processing images. The global ML model 220 may also be termed a student network. A second or teacher ML model 204, also in the form of an artificial neural network, transfers information to the global ML model 220 (or to at least one module of the global ML model) using a cross-architecture knowledge transfer approach via an auxiliary feature space. The new technique may thus be termed AuXFT. The cross-architecture system 200 comprises two main blocks: the global network 204 which is to be deployed on a user device and a teacher network 202 which is typically significantly larger than the global network 204. The two networks are also not typically compatible. The teacher network may also be termed an oracle. During training, an input 208, e.g. an input image, is input into both networks. At inference time on the user device, the teacher network 202 is not used, in other words, the teacher network 202 is typically discarded before transferring the global ML model to the user device. A feature translation block 206 allows interfacing between the two incompatible architectures so that we can transfer the knowledge of the large network 202 into the smaller network 204.

[071] Cross-Architecture Knowledge Distillation has emerged in recent years as a generalization of hint-based knowledge distillation approaches (which assume that the architectures’ topology matches). The objective is to distill transformer architectures into more efficient alternatives {e.g. , CNNs), since their impressive performance comes at a significant compute cost. The most recent work on the topic is “One-for-AII: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation,” by Hao et al published in Proc, of NIPS, 2024 where distillation on outputs and feature spaces of models is applied simultaneously, mimicking shortcut-based architectures. This paper focuses on achieving the highest accuracy on a task shared by teacher and student models, via KL-divergence-based losses to distill teacher logits into the student’s output space and projected features. Our objective is different since we consider the reconstruction of the teacher’s feature space as an auxiliary task, which we want to solve without impacting the performance of the main task. Therefore, we use losses (which may still be KL losses as shown) more geared toward reconstruction and apply them to an auxiliary, translated space.

[072] The student network may be selected to have an architecture, e.g. a neural network, which makes it suitable for performing a primary task, e.g. object detection and classification. The teacher network may be selected to have an architecture, e.g. a neural network, which makes it suitable for performing a secondary task, e.g. refining the classification generated in the primary task.

[073] As described in more detail below, the feature translator block comprises a channel differential module which adapts the dimensionality of the detected features 212 which are extracted by the detector 230 of the global network 220 and a spatial differential module which adapts the resolution of the detected features 212. Once the features have been translated into auxiliary features 216 as described in more detail below, a few shot learning module 210 can use few shot learning approaches on the features. In other words, the student network (which includes the FSL module) is trained to perform the secondary task to mimic an output from the teacher network.

[074] The Few Shot Learning task was first introduced in “Learning from one example through shared densities on transforms,” by Miller et al in Proc, of CVPR in 2000 and it has been extensively investigated in the past years. The seminal work “Prototypical networks for fewshot learning,” by Snell et al published in Proc, of NIPS, 2017 sparked a variety of approaches which are based on a prototypical classification approach where class-representative vectors (i.e., prototypes) are stored in memory and used to classify an unknown query vector. Recently, FSL has seen impressive development thanks to its potential in a variety of tasks beyond image classification, especially for dense scene understanding. Few shot learning is a specific type of training which uses a few (e.g. fewer than 5, perhaps only 1) training item.

[075] Model architectures which can be used as the large network include transformer-based architectures such as DINOv2 described in “DINOv2: Learning robust visual features without supervision,” by Oquab et al published in arXiv:2304.07193, 2023. Examples of these architectures have 21 million, 86 million, 300 million or 1100 million parameters. Another big network is a convolutional based one such as DenseNet BC which has 25 million parameters and which is described for example in Densely connected convolutional networks by Huang et al published in Proceedings of the IEEE conference on computer vision and pattern recognition in 2017. Other examples are hybrid models such as SWIN-t having examples with 29 million, 50 million, 88 million or 197 million parameters or conformer having examples with 31 million and 119 million parameters. This is described for example in “Swin transformer: Hierarchical vision transformer using shifted windows,” by Liu et al in Proc, of ICCV Vision, 2021. Model architectures which can be used in the smaller network include transformer based architectures such as mobileViT having 2.6 million or 5.6 million parameters and which is described for example in Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer by Mehta et al published in 2021 on arXiv. Another example is a convolutionalbased model such as mobileNet with 3.4 million parameters which is described in Searching for mobilenetvS. By Howard et al published in Proceedings of the IEEE / CVF international conference on computer vision in 2019 or YOLOV8n which is described in “Ultralytics YOLOv8,” by Jocher et al published online in 2023. This has examples with 3.2 million or 11.2 million parameters. In other words, the small network is between 10 to 50 times smaller than the large network. The actual choice of both networks depends on the target task, use case and hardware constraints.

[076] In one example shown in Figure 2, the teacher network 204 is a self-supervised learning (SSL) encoder, for example DINOv2 described in “DINOv2: Learning robust visual features without supervision,” by Oquab et al published in arXiv:2304.07193, 2023. In this example, the teacher network 204 is used to transfer knowledge to an encoder module 202 within an object detector 230 within the global model 220 (which may also be termed a student network). In object detection, the input space X c ^xn / xs can identified as the set of RGB images of size H x W; while the output space is a list of coordinates (x^y^xvyi), class-assignments (c), and classification confidence (p): = {(xo^x^yi, c,p)}k=1, c (]R4 x Cc x [0,1])^; where K is the number of predicted detections. To enable FSL, we choose an encoderdecoder object detector 220, with M = D 0 E:X ^y, where 0 is the composition operator, so that we can access the intermediate features which are generated by the encoder (which may also be termed a feature extractor). A suitable example is YOLOV8n which is described in “Ultralytics YOLOv8,” by Jocher et al published online in 2023.

[077] In other words, given an input image X e X such as that shown in Figure 3a, the encoder 202 produces a set of features X 9 {Fn F2,... F„} = E(X) which we can term the detected features 212 (or intermediate features or original features). The detected features 212 can also be used in pre-training of the global model 220 as described below. In this example the object detector 220 comprises a decoder 222. The detected features 212 are also fed to the decoder 222 and the output is at least one bounding box 224 and a coarse-level classifcation “scissors” such as that shown in Figure 3b. We can also feed the detected features to the FSL block P: 7 h c as described below to obtain the fine-grained classification (“my scissors”) such as that shown in Figure 3c. As shown in Figure 3c, the fine-grained classification replaces the coarse-level classification next to the bounding box. In other words, the output may be an edited version of the original image with the second classification shown thereon and optionally the bounding box.

[078] Figure 4 is a variation of Figure 2 showing a little more detail of the cross-architecture system, in particular the translator block 206 within the global model. The features which are also shown in Figure 2 retain the same reference number. The feature translator block 206 is used to generate the auxiliary feature space. The block 206 consists of two major components which are new: a Channel Differential module (Dc) 230 and a Spatial Differential module (Ds) 232. There is also a new component which is a detection driven feature pooling module 234. The encoder module 202 and decoder module 222 are modules which relate to the object detection module and logical flow. The large model, namely the SSL encoder 204, is used for object detector training only.

[079] The Channel Differential module (Dc) 230 and Spatial Differential module (Ds) 232 translate the original feature space F, into an intermediate space R( where all feature maps have the same resolution and channel depth. Such features are then used to produce the auxiliary features via residual chain by Ri = lj=iDs(Dc^j^ Vi = 1,2.....n. where Rf is the set of auxiliary features where i = 1 to n, F, is set of original features where j = 1 to n, n is the number of feature maps which are produced by the detector encoder, Ds is the operation by the spatial differential module and Dc is the operation by the channel differential module. This allows the high-fidelity and low-resolution estimates generated by the channel-dense features to have their high-frequency components restored by their more shallow but higher-resolution counterparts.

[080] In principle, the channel differential module 230 can be any mapping Dc from the feature space of the encoder 202 to the feature space of the same spatial resolution, but with channel depth matching the one of the teacher model 204 (i.e. oracle feature extractor / SSL encoder). That is: D ' । ) RH'x^x / 0 where lE and l0 are the number of channels of the encoder 202 of the object detector 230 and encoder of the teacher model 204 (oracle), respectively.

[081] Figures 5a and 5b show two variations of suitable channel differential modules 530, 540. Figure 5a shows a linear channel differential module 530 comprising a convolution module for each feature. Figure 5b shows a non-linear channel differential module 540 comprising two convolution modules interleaved with a ReLLI layer for each feature. As detailed below, we tested a variety of different projections before choosing a mid-range option offering a good compromise between few-shot performance and computational cost, that is, a 3x3 convolution module. It may be possible to achieve higher-performing architectures by adding more capacity in the channel differential block, for example by using the arrangement of Figure 5b but this will come at a higher computational cost.

[082] After projecting the feature vectors in the teacher’s channel space, we adapt the spatial dimensions via the spatial differential module 232, to allow computation of the residual maps. In principle, this task can be performed by any mapping where l0 is the number of channels of the encoder of the teacher network 204 (oracle) which has resolution H" x W".

[083] For efficiency, we opted for a parameter-free approach using classical resampling algorithms. Since the aspect ratio (W: H) of the object detector 230 and encoder of the teacher network 204 (oracle) features is the same, we employed an adaptive strategy to select the interpolation algorithm depending on the resizing factor p = — = — as (area, bilinear, bicubic, if p <1 — 8 if 1- 8 <p<l +8 if p >1 + 8 where 8 is a parameter that we empirically set to 0.1. More information on the choice of resizing factor finds root in “Digital signal processing: a computer-based approach" by Mitra published in McGraw-Hill Higher Education, 2001. There are three algorithms that can be selected. The first resizes the area we are downsampling an image without introducing re-sampling artifacts and the best strategy is to average the pixels from the high-resolution source. In the second algorithm; if the target resolution is close to the source resolution, then there is no need for large interpolation kernels and a simple linear interpolation is enough to reduce the complexity overhead. In the third algorithm, to upsample an image of a larger resizing factor, a more complex interpolation strategy {e.g., bicubic) is preferable to obtain more accurate results. It will be appreciated that the algorithm above is just one of many algorithms which can be used to adapt the spatial dimensions.

[084] Figure 6 shows an example of a spatial differential module 532 which comprises a resizing module for each feature. The outputs of the spatial differential module 532 are a set of residually adapted features Ri, R2, ... Rn. which are also termed auxiliary features. The spatial differential module may be considered to be parameter-free and may use traditional resizing primitives. It may be possible to achieve higher-performing architectures by changing the resize operations, for example by adding non-linear spatial differential operations (e.g. “ours non-linear” in the experiments below).

[085] Returning to Figure 4, a fundamental component of our proposed method and system (AuXFT) is the generation of the secondary output which in this example is the re-labeling of the boxes detected by the detector 230 (function labelled M). This is done using a classifier such as a prototype-based FSL 210, which requires embedding vectors as inputs. As shown only Ri is output from the spatial differential module 232 to the detection driven feature pooling (DDFP) module 234 which handles the task of converting the auxiliary feature Ri into such vectors (one for each bounding box, V = {^, v2,..., v^}). These vectors are represented in Figure 4 by a vectors list 236. The predicted boxes Nk which are output from the decoder 222 of the detector 230 are used to summarize the relevant regions of the map into a vector using spatial-aware average pooling.

[086] During FSL training, these vectors are accompanied by the original coarse class prediction c e Cc and by a fine class assignment c' e Cf provided by the user. In other words, we refine the original ( / .e., coarse-level) detection class-set with a limited amount of user-labeled personal ( / .e., fine-level) samples which are shown in Figure 4 as user input 240. At inference time, only the coarse class prediction is needed by the FSL module to predict a new fine-class assignment and the user input is not used (as schematically indicated by the crossed line). The user input and FSL module may be considered to be personalisation stage modules.

[087] Figure 7 shows a block diagram of a system 700 incorporating the cross-architecture system above. The system 700 comprises a plurality of primary user devices 720 and for ease of reference, a single primary user device 720 is shown. Each primary user device 720 may be any electronic device which utilizes computer vision. For example, each primary user device may be a smart vacuum cleaner, smart domestic appliance (such as a smart oven), smartphone, tablet or other robotic device. However, it will be appreciated that the primary user device(s) may be any suitable electronic device. The primary user device 720 comprises the standard components of such devices, including for example as shown one or more processors 722, memory 724, storage 726, an image capture device 732, and a communication module 734. The primary user device 720 may optionally comprise a display 736 for displaying information to the user and a user interface 738 such as a keyboard, voice recognition input device, touch sensitive screen or any similar component for receiving user input. There may be other standard components which are omitted from the Figure for ease of reference. The display 736 may comprise any suitable display screen, e.g. LCD, LED which may also be touch sensitive to allow user input.

[088] The communication module 734 may communicate using any suitable communication, e.g. wireless communication, hypertext transfer protocol (HTTP), message queuing telemetry transport (MQTT), a wireless mobile telecommunication protocol, radio frequency communication (RFID), near field communication (NFC), ZigBee, Thread, Bluetooth, Bluetooth LE, IPv6 over Low Power Wireless Standard (6L0WPAN), Constrained Application Protocol (CoAP) or a wired communication. The memory 724 may be any suitable form of memory, including volatile memory, such as random access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example.

[089] The image capture device 732 may be any suitable device, e.g. camera for taking input images that may then be processed as described above. At least one personalised ML model 730 (also termed an artificial intelligence (Al) model) which has been trained to produce an output is stored on the user device. The local storage 726 also locally stores a local dataset 728. The local dataset comprises a training set of images which are labelled with a classification (fine classes 727) and vectors 729 for locally training the ML model to personalise the ML model. It will be appreciated that, given the local dataset 728 is stored locally on the user device, the local dataset may remain on the user device. Thus, the data remains private and there is a reduced risk of data breach or data theft. The personalisation can also then be targeted to users who do not typically share any data.

[090] Using the communication module 734, the user device 720 may communicate with a server 702. The server 702 may also comprise standard components, including for example as shown one or more processors 704, memory 706 and a communication module 708. There may be other standard components which are omitted from the Figure for ease of reference. The server 702 may also comprise two pre-trained ML models 708: the teacher model and the global ML model including the object detector, translator and the FSL module. The global ML model may be received by the primary user device 320 for local training, as described below and the teacher model may remain on the server. The server further comprises storage 710 for storing an object detection training dataset 712 (which may also be termed a master or global training dataset). The object detection training dataset 712 may comprise a global training set of images with comprising a set of coarse classes 714 for objects within the images and a set of bounding boxes 716 surrounding each object. It will be appreciated that this global training dataset may also be stored in a database which is separate and remote from the server.

[091] The system 700 may optionally further comprise a secondary user device 740. Similar to the primary user device 720, the secondary user device 740 may comprise standard components, including for example one or more processors 742, memory 744 and a display 750. The description of these standard components in relation to the primary user device also applies to the equivalent features in the secondary user device 740, and so their description is not repeated for conciseness. The secondary user device may be used to perform command and control tasks for the primary user device. For example, the primary user device may be a robot and the secondary user device may be a phone and a user inputs commands into the phone to control the robot.

[092] The secondary user device 740 further comprises storage 746 which may be used to store a personalised ML model and a local dataset (not shown). The secondary user device may communicate with the primary user device 720 and the server 702 via the communication module 734. In this arrangement, the communication modules 734, 706 of the primary user device and the server, respectively, may be further used to communicate with any of the server, primary user device or secondary user device (where appropriate).

[093] In particular, the secondary user device 740 may communicate with the primary user device 720 to provide a training set of images (together with fine classes) to the primary user device. In this arrangement, the training set of images may be received by the primary user device 720 and may be used to locally train or update the ML model 730.

[094] In an arrangement where the secondary user device 740 communicates with the server, the secondary user device may similarly receive the pre-trained global model. The pre-trained model received by the secondary user device may be a compressed version of the pre-trained model. The processing power of the processor 742 of the secondary user device 740 may be significantly greater than the processor 722 of the primary user device 720. In this arrangement, the secondary user device may be a smartphone or tablet, and the primary user device may be a robotic vacuum cleaner, which are typically resource-constrained compared to smartphones and tablets. The secondary user device may create a fully trainable version of the pre-trained model received from the server. This may be done by, for example, dequantizing a compressed version of the pre-trained model and determining whether re-training of the pre-trained model is possible. This may be possible since modern smartphones and tablets may have sufficient processing power to fully re-train a trainable version of the model received from the server.

[095] When it is determined that it is possible to re-train the model, the secondary user device may train the received model using a local dataset provided by the user. The characteristics of the local dataset used by the secondary user device in this arrangement are equivalent to those of the local dataset 728 of the primary user device, and their description is not repeated for conciseness. Once trained, the received model may be compressed by the secondary user device and received by the primary user device. For example, a smartphone or tablet may re-train the model received from the server and send the re-trained version using the local dataset provided by the user to a robotic vacuum cleaner for enhanced personalisation.

[096] As mentioned above, the user device may be a robotic device, such as a robotic vacuum cleaner. Typically, a vacuum cleaner encounters different objects and instances of objects as the vacuum cleaner navigates a user’s home using a computer vision system housed within or mounted on the vacuum cleaner comprising an image capture device and a ML model as described in relation to the methods above. That is, the images captured by the computer vision system depict object classes which differ from those which the ML model would have been trained on. It will be appreciated that the robotic vacuum cleaner is just one example and other devices of interest are smart ovens, lawnmowers and smart fridges. In these examples, there is a need to update the model to reflect the user’s preferences, e.g. class-incremental food detection when users enter new items in the fridge or class-incremental food / dishes being cooked in the oven.

[097] As an illustrative example, the vacuum cleaner may receive a voiced instruction from a user, such as “Clean close to my coffee table.” The present techniques advantageously enable a way for the robotic vacuum cleaner to identify the user’s personal coffee table and to control the robotic vacuum cleaner based on the identification of the user’s coffee table i.e. by instructing the vacuum cleaner to clean around the user’s coffee table. If the present techniques are not used, the vacuum cleaner may not be able to recognise the piece of furniture and thus cannot execute the command. This will leave the user unsatisfied. It will be appreciated that the example of a user’s coffee table is provided by way of example of a personal object / class or instance-level class, and that other personal objects may be identified by a vacuum cleaner. As another example, the user may ask the vacuum cleaner for the location of a pet by the pet’s name. Using the system described in detail below, the vacuum cleaner may be able to recognise the pet and reports an accurate answer to the user. In this way, the user is satisfied by the response from the vacuum cleaner. When the system is not implemented, the vacuum cleaner cannot answer the user and thus the user is not satisfied.

[098] The present techniques may also be implemented in a user device in the form of a fridge (also termed a home hub). For example, the user may be shopping and may send a command to the fridge (e.g. via a secondary device such as a mobile device), to ask the fridge whether there are specific items in the fridge (e.g. a bottle of a particular brand or a particular variety of fruit, vegetable or other foodstuff). Using the system described in detail below, the fridge may be able to distinguish between similar foodstuffs (e.g. distinguish between brands or particular varieties) and reports an accurate answer to the user. In this way, the user can purchase missing items while shopping and is satisfied. When the system is not implemented, the fridge cannot distinguish between similar foodstuffs and the user does not know what to buy and thus the user is not satisfied.

[099] As another illustrative example, the user device may be a smart phone (or similar device) and the user may take a picture of one or more friends. With the present techniques enabled on the phone, the user can name the people in the picture with a single tap and the phone will remember and then subsequently recognise these people. The user is likely to be more satisfied and may interact more with their phone. By contrast, if the new techniques are not enabled, the camera app on the phone may only be able to detect that there are people in the picture (e.g. detect the generic or coarse-level classification of people) and the user is not satisfied and likely to feel a lack of connection with the phone. In another use on a phone, a user may be searching for something in a cluttered scene which has been captured by the phone or in a photo stored in a gallery. With the present techniques enabled on the phone, the user can vocally ask the phone to locate a specific user object within the photo of the cluttered scene and the phone will use the techniques to identify and then highlight the requested object in the scene. The user is likely to feel satisfied with this control of the phone. By contrast, without the present techniques, the user needs to manually locate the requested object and is not likely to be satisfied.

[100] Figure 8 is a flowchart showing the steps which may be considered a planning or preparation stage. In a first step S802, a primary task is chosen (e.g. classification with a bounding box) and an appropriate network for carrying out that primary task is selected at step S804. This primary network may be termed a student network and the terms may be used interchangeably. In parallel (or sequentially in either order) at step S812, a second task is chosen (e.g. refining the classification of the primary task) and an appropriate network for carrying out that secondary task is selected at step S814. This secondary network may be termed a teacher network and the terms may be used interchangeably.

[101] At step S816, a translator block is added to the primary network so that an auxiliary feature space may be created using the translator to allow the primary network to learn from the secondary network. At step S818, a labelled dataset is obtained and used at step S820 for training the primary network to mimic the output from the teacher network. At step S822, the trained primary network is then output and this student network is able to perform both the primary and secondary tasks.

[102] As another alternative to the image processing techniques which are described previously and also below, the primary network may be a natural language processing (NLP) network (e.g. a generic GPT network) and its primary task may be to output Italian. The secondary task may be to output another language, e.g. English and thus the secondary network may be another natural language processing (NLP) network. Although both the primary and secondary networks may both be GPT models based on transformer architectures, there is still a need for the translator block to allow the student network to efficiently and automatically benefit from the knowledge of the teacher network, e.g. to learn a new language, without forgetting its knowledge. The distillation of the knowledge from the teacher network into the student network makes the system more efficient in terms of space.

[103] As another alternative to the image processing techniques which are described previously and also below, the primary network may be a generative network and its primary task may be to generate wallpaper having scenes from space. The secondary task may be to output a different wallpaper, e.g. wallpaper having earth scenes and thus the secondary network may be another generative network. Although both the primary and secondary networks may both be generative network, there is still a need for the translator block to allow the student network to efficiently and automatically benefit from the knowledge of the teacher network, e.g. to learn how to generate a new type of scene, without forgetting its knowledge. The distillation of the knowledge from the teacher network into the student network makes the system more efficient in terms of space. The distillation allows generation of new and diverse environments.

[104] Figure 9 is a flowchart showing the steps for pre-training the global model (student network) which is normally done at the server. In this example, the student network has a primary task of generating a bounding box and classification for each object in an image and a secondary task of generating an improved classification for each identified object. In a first step S900, a labelled input image is obtained from an object detection training dataset. The input space X c ^xm / x3 can identified as the set of RGB images of size (i.e. resolution) H x W. Each input image may comprise one or more objects within the image. Each input image is labelled with a first classification which may also be termed a coarse-level classification Cc for each object in the input image and a bounding box for each object in the input image. Each bounding box may be labelled as a list of coordinates, e.g. (x0,y0>xnyi) which specify the corners of the bounding box. Each input image is input to two separate neural networks.

[105] At step S902, the input image is input to a teacher neural network which may be a selfsupervised learning encoder (e.g. DINOv2) which outputs a set of teacher features 0 at step S904. This set of teacher features has a resolution defined by H" x W" and a channel depth of l0. At the same time, or sequentially, the input image is also input at step S912 to a global neural network which includes an encoder-decoder arrangement to allow access to intermediate features. For example, the global neural network may be an encoder-decoder object detector (e.g. YOLOV8n) and at step S914, the set of intermediate features T 9 {Fn F2,... F„} are output, e.g. from the encoder. This set of intermediate features has a resolution defined by H' x W' and a channel depth of lE. In other words, each of the input image, and the two sets of features have a different resolution. Furthermore, the two sets of features have a different channel depth.

[106] The next step S916 is to translate the set of intermediate features into a set of auxiliary features R( which can be used to train the global neural network, or at least part of the global neural network. In other words, we align the detector’s feature space (i.e. the intermediate feature space) with the SSL encoder’s feature space (i.e. the teacher feature space) using the auxiliary feature space. In this way, the global neural network (in particular the encoder of the detector) can learn from the teacher neural network (i.e. the SSL encoder) which is typically much larger. In the auxiliary feature space, all feature maps have the same resolution and channel depth as the teacher feature space. As explained above, the translation can be done in two stages. In a first stage, a channel differential module maps the intermediate feature space to a feature space of the same spatial resolution but with a channel depth matching the teacher channel depth (i.e. the target channel depth). That is, Dc-. ^h'*w'*1e ^'xn / 'x / 0 |n a second stage, a spatial differential module adapts the spatial dimensions. That is £^.gH,xiWx(0 , } j^H^xiv^xi0

[107] At step S906, we can then train the translator and the encoder of the global neural network using both the set of teacher features and the set of auxiliary features. In other words, we adapt the parameters of the translator and / or the encoder. For example, as is common in reconstruction tasks, we employ the sum of and norms as the distillation loss by n = y y ll Mp] - Q[p] Ik +ll RJp] - Q[p] ll2 R Z Z h" x w" i=l pEH^XW” where 0 = E0(X) are the set of teacher features (e.g. oracle features of an SSL extractor -DINOv2) having resolution H” x W”. More detail on the distillation loss may be found for example in “Efficient minimization methods of mixed 12-11 and 11-11 norms for image restoration,”by Fu et al published ins SIAM Journal on Scientific computing, pp. 1881-1902, 2006. This allows knowledge distillation losses to bring intra-class variability into the auxiliary feature space. In other words, the descriptive features produced by the oracle network are distilled in the auxiliary space through the translator block during training.

[108] At the same time as the knowledge distillation loss is being used to transfer information from the larger network to the smaller network, the set of intermediate features from the encoder are also passed to the decoder in the normal manner at step S920. The decoder then generates a bounding box and a classification for each object in the image at step S922. The encoder and decoder are then trained at step S924 to minimise a loss which may be termed a detection loss, which compares the labelled bounding box(es) with the generated bounding box(es). The detection loss may comprise of two components, one which optimizes the bounding box location and one which optimizes the class prediction. The former is similar to a regression task: the architecture predicts four numbers (the coordinates xO <x1, yO <y1) which are compared to the ground truth ones. The loss is the distance between the four points and their reference and minimizing the loss means aligning the predicted corners with the reference. The latter is standard classification: a class assignment probability distribution is proposed by the network. The distribution is modified by the loss to reflect the classassignments provided in the ground-truth labels. The predicted assignment probability can be used as a measure of confidence during inference. More details can be found for example in the article: “YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications” published on the ArXiv.

[109] Figure 10 is a flowchart showing the training steps which take place on device following the training in Figure 9. Again, for illustration, the detail relates to the image classification example. In a first step S1000, the global model which has been pre-trained as described for example in Figure 9 is received at the user device to which it has been deployed. At the next step S1002, an input image is received. In this case, the input image may be an image, e.g., a scene, which has been captured by a camera or similar component on the user device. There are no labels attached to the input image. The input image is input into the global model and the global model is used to infer information about the input image. As indicated by the two parallel tracks, the global model is used to generate two outputs: a primary output which is a bounding box and a secondary output which as explained in more detail below is used to train the FSL module within the global model. In both cases, the first step S1004 is to generate the set of intermediate features T 9 {F^ F2,... F„} from the encoder of the object detector.

[110] The generation of the primary output follows similar steps to part of the method of pretraining detailed above. At step S1006, the set of intermediate features from the encoder are passed to the decoder in the normal manner. The decoder then generates a bounding box and a coarse classification for each object in the image at step S1008. The bounding box(es) and the corresponding coarse classification(s) may be termed the primary output of the global model.

[111] The generation of the secondary output also follows similar steps to part of the method of pre-training detailed above. The next step S1010 is to translate the set of intermediate features into a set of auxiliary features R( which can be used to train a part of the global neural network. This may be achieved via residual chain by n Rj = Ds(Dc(f^ VI = 1,2.....n. i=i where R; is the set of auxiliary features, F, is set of intermediate features where j = Ito n, Ds is the operation by the spatial differential module and Dc is the operation by the channel differential module. The set of auxiliary features is then output at step S1012. There are multiple sets of auxiliary features because there are multiple sets of intermediate features in the feature space which is output from the feature extractor of the object detector. However, the auxiliary feature space will only have the same number of sets of auxiliary features as the feature space which is output from the teacher ML model. In one example, for example using an oracle SSL as the teacher ML model, there may be only one set of features. It is also noted that Rn for example when calculated using the formula above, may be expressed as the sum of all the other auxiliary features and thus will typically have the highest quality. Thus, only Ri may be used.

[112] At step S1016, the embedding vectors, one for each bounding box, V = {^, v2,..., v^}, which are to be used to train the FSL module are generated. The predicted boxes Yk which are output from the decoder of the detector are used to summarize the relevant regions of the map into a vector using spatial-aware average pooling. For example a suitable framework is set out below: *i-i y^i 1 1 y RJ^y] v Yk = (%0, y0< yn c p) 6 M(X). where x and y are coordinates, i.e. pixel locations, of the bounding boxes, c is the predicted coarse class, p is the prediction confidence (the probability of class assignment), Yk are the predicted bounding boxes which are output from the object detector and only Ri of the set of auxiliary features R( is used. It is noted that xO <x1 and yO <y1, conventionally x direction is along the image width and increases left-to-right, while y direction is along the image height and increases up-to-down.

[113] At step S1018, user input is received. The user may select one or all of the output bounding boxes and may then provide a fine class assignment c' e Cf for each selected box. The user may be in the form of a personal class for each object in a bounding box. At step S1020, the user input together with the embedding vectors and the original coarse class prediction c e Cc are used to train the FSL module. The training may be relatively simple, and occurs after feeding the image to the (frozen) detector encoder and translator blocks and obtaining the auxiliary feature set (e.g. R1). As explained above, using the predicted boxes, a list of embedding vectors (one for each detected object) may then be generated and the user interaction is then provided to select one (or more) boxes in the scene and provide an appropriate personal label for it. To reduce the computational cost of the operation, the classifier training may not use standard methods such as minimising losses or gradient backpropagation. In other words, the classifier may not contain any optimizable parameters.

[114] To mimic the limited number of samples, FSL evaluation is often performed in a cross-validation manner: in each episode T = (Ts,Tq), TsaTq = 0, a support set Ts of training samples is selected from the available samples, leaving the remaining as the validation query set Tq where the performance is evaluated. This procedure is repeated a number of times, changing the support and query set each time, to obtain the average episodic accuracy, which is a more reliable estimate of the actual system performance. When |TS| = 1 the scenario is referred to as 1-shot, when |7;| = 5 we refer to it as 5-shot.

[115] Using a FSL module in this way allows the processing burden to be split in two which is a common approach and may be termed feature-level prototypical classification. During the FSL training, the parameters of the object detector are frozen and the object detector is only used for inference. The frozen and compute-heavy feature extractor can for example be implemented in hardware, while the variable and efficient FSL classifier can be implemented in software. During the training, the FSL module learns to associate the embedding vector for the selected bounding box with the personal class provided by the user. At step S1022, the associated vector, coarse classification and fine-grained classification may be stored in an internal database on the user device. Storing the vectors with the original, coarse classification allows the creation of separate search spaces within the internal database. The internal storage may thus contain for each coarse class prediction c, a set of vectors which may be termed prototypes J>[c] = {p^ p2,...}.

[116] Figure 11 is a flowchart showing how the fine-tuned model is used on a device. At inference stage, a new input image is obtained as shown at step S1102. This could be a scene captured by a camera on the user device. Like in Figure 10, the global model is used to generate two outputs: a primary output which is a bounding box and a secondary output which as explained in more detail below is used by the FSL module to refine the primary output. As before, at step S1104, the first step in generating both outputs is to output a set of intermediate features is output from the encoder.

[117] The generation of the primary output is the same as the part of the method of fine-tuning detailed above because in both the fine-tuning and inference, the object detector part of the global model is frozen and only used for inference. At step S1106, the set of intermediate features from the encoder are passed to the decoder in the normal manner. The decoder then generates a bounding box and a coarse classification for each object in the image at step S1108.

[118] The generation of the secondary output also follows similar steps to part of the method of pre-training detailed above. The next step S1110 is to translate the set of intermediate features into a set of auxiliary features R( and to output this set of auxiliary features at step S1112. The bounding box(es) and the set of auxiliary features are used to generate the embedding vector(s) for each bounding box at step S1116 as described above.

[119] For each embedding vector, as shown at step S1118, any vectors which are stored in the internal database and which have a coarse classification which corresponds to the classification output for the associated bounding box are identified. At step S1120, the generated embedding vector is compared with and all identified vectors (which may be termed internal prototypes), for example a distance between a query vector and each identified vector(s) may be computed. In other words, our FSL is implemented in a conditional fashion, that is, we limit the search space of the fine-level class on the basis of the coarse class predicted by the detector (rather than searching among all possible options). This allows our method to be about three times faster than the standard non-conditional FSL, without significant impact on the re-classification accuracy, as we observe in the ablation study. The FSL module is still based on a prototypical network, for example as described in the Snell paper mentioned above, but is tailored for efficiency, given the computational constraints of robotic applications. In general, a prototypical network is trained by providing a series of labeled vectors, which are used to build internal class prototypes.

[120] At step S1122, the closest matching vector in the small search space within the database is found. At step S1124, there is a determination as to whether there is a fine-grained class associated with the closest matching vector and if so, at step S1126, the query vector is classified by choosing the fine-grained class which is associated with this closest matching vector (i.e. closest prototype). Another difference compared to standard FSL modules is the introduction of an additional vector in each of the conditional sets, which we use as a fallback when a query does not match any fine class. If the query vector matches this additional vector, there is no associated fine-grained classification. This because the additional vector is the centroid of the set and will match the query vectors that are not similar enough to any of the prototypes.

[121] At step S1128, in this case a new class for the query vector is computed and output. This addition requires the FSL to be robust and for this reason, we employed multiple distances and a majority voting approach for the classification. Given a set of distance functions 2) = {d^ d2,...}, a query vector q, its coarse class prediction c and a set of prototypes = {Pi, P2< ■■■}, we can compute a new class assignment as follows: 1 V fl r n 1 d = -—- >softmax]—----r; Vi = 1,..., IJ’MI L W(q,Pi) LJIJ c' = argmaxi=1.....|j>[c]|d[j] Effectively, we compute an assignment probability distribution from the inverse of the distance between the query vector and the prototypes for each function (for the results we used cosine, and ^2 distances, see above). We then average over the distances and select as the fine class the index corresponding to the maximum.

[122] The method shown in Figure 11 applies to a specific use case where the primary task is to generate a classification and optionally a bounding box for each object in an image and the secondary task is generation of a more fine-grained and user specific classification. These can then be used to generate an updated image and / or to control the user device as explained above. Other uses can be envisaged because the general principles are more widely applicable. Examples of other primary and secondary tasks include segmentation masks and descriptive features. The key, general steps of the inference method of Figure 11 may be summarised as receiving an input, which may be an image or another input such as text or speech. The input is processed by the student network (e.g. the object detector) and the intermediate features (which may also be termed original features) are output from an encoder of the student network to the decoder of the student network which does the primary task, e.g. generation of bounding box and / or classification of an object in an image. The intermediate features are also changed into auxiliary features by the translator and the auxiliary features are passed to a secondary task decoder (e.g. the FSL classifier). The secondary task decoder then generates the secondary task output which is the user-specific classification in the example of Figure 11.

[123] We validate AuXFT on four benchmarks against some baseline architectures and compute-heavy upper bounds (we ignore FM-based strategies since their cost makes them unviable on-device). Before considering the results in detail, we note that existing approaches typically fall short in learning both discriminative (for OD) and descriptive (for personalization) features. Standard OD systems such as the one described in the Ultralytics YOLOv8 paper may suffer from the Neural Collapse (NC) phenomenon. In other words, features collapse to their centre mean (centroid) which makes them incompatible with few-shot learners which require rich within-class variance. More precisely, the cross-entropy (CE) objective at pretraining stage causes the models’ features to collapse around class centroids, losing intraclass variance and hence their usefulness for the personalization stage. More descriptive features can be learned by models pre-trained via self-supervised learning (SSL) losses instead of CE, making these models especially suited as feature extractors for FSL modules. Nonetheless, as detailed above, distilling knowledge from an SSL pre-trained model {e.g., DINO), into an OD system entails several compatibility challenges and can affect features for a downstream OD task. Personalization of large semantic segmentation Foundation Models has previously been explored but such approaches require multiple architectures and, therefore, are not suitable for resource-constrained applications.

[124] Returning to the validation, we choose two metrics for the evaluation, both averaged over 100 episodes, as is common practise in FSL strategies: mAP50-95 (mean Average Precision 50-95) and retrieval accuracy. The first metric: mAP50-95 is the average of the loU-thresholded mAPs (mean average precisions) from mAP50 to mAP95 (in steps of 5), each measuring the ratio of correctly predicted boxes with at least N% loll. Such a metric is described in more detail, for example in “Microsoft coco: Common objects in context,” Lin et al published in 2015. The second metric: retrieval accuracy instead measures the fraction of boxes correctly classified, regardless of localization. We remark that both are computed on the fine-grained class set, which on average, contains 5 times more classes than the coarse set in the considered datasets.

[125] The implementation details include training using images (394,619 in total) with resolution 672x672px, using an optimizer (such as decribes in Adam: A method for stochastic optimization” by Kingma published in 2017) for 50 epochs adopting a cosine annealing with linear warmup (1000 iterations) scheduler for both learning rate (max 1 x 10-3) and weight decay (max 1 x 10-4). We follow the same policy as in the YOLOv8 paper loading weights of a detector pre-trained on the COCO dataset using an exponential moving average on the weights for a more stable training evolution. Maximum VRAM and inference time were measured on an NVIDIA 1080Ti GPU.

[126] For both our method and the benchmarks, we pre-train all detectors on a subset of the Openimages dataset with 31 coarse classes, covering the ones present in the personalization datasets, personalization classes absent from Openimages were discarded. The dataset is described in “The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale,” by Kuznetsova et al published in JCV, 2020. The results are then generated using the following datasets. Firstly, the PerSeg dataset which is described in “Personalize segment anything model with one shot,” by Zhang et al and published in arXiv:2305.03048, 2023. This dataset contains 212 images in 39 instance-level classes (13 coarse). To the best of our knowledge, it is the only dataset proposed for few-shot personalized dense tasks. PerSeg was originally proposed for semantic segmentation, so we converted the annotations for object detection (OD). Some instances have a limited number of samples (as low as 4), precluding the 5-shot scenario and the coarse-to-fine mapping is unbalanced, that is, some classes have only one personal example, while others have up to eight.

[127] Another dataset is the POD (Personalized Object Detection) dataset which is an inhouse benchmark of everyday objects in indoor scenes {e.g., glasses, shoes, bottles, etc.) developed to solve the limitations of PerSeg. We capture a total of 5 training images for each of the 45 instances considered (3 for each of the 15 coarse classes) for a total of 225 images. In the validation set, we capture images both in light and dark conditions. We acquired 3 validation images per instance: alone in the scene, mixed with the other same-class instances, and mixed with instances from different classes (for a total of 150 images).

[128] Another dataset is the CubWorld dataset which is described in “Teaching icub to recognize objects using deep convolutional neural networks,” by Pasquale et al published in Proc, of MUS. This dataset focuses on robotic views and has been acquired using video sequences. We select 15 samples per sequence (31 fine classes, 6 coarse) in the FSL split. The bounding boxes have noticeable padding around the objects making them incompatible with the pre-training. Therefore, we fine-tuned the decoder before training the FSL and used the retrieval accuracy instead of OD metrics.

[129] The final dataset is the CORe50 dataset which is described in “Core50: a new dataset and benchmark for continuous object recognition,” by Lomonaco et al published in Proc, of ACRL, 2017. This dataset provides a variety of object instances captured in variablebackground scenes (10 classes, 5 instances per class). We selected a frame with a valid bounding box every thirty from the source videos. The boxes were generated algorithmically and are squared with significant padding around the object. Therefore, we fine-tune the detector and use the accuracy in this dataset as well. To align with the other domains, we set up the episodes on this dataset to have a single scene inside them.

[130] The quantitative results for one- and five-shot learning for each dataset are reported in Figure 12a, together with the average throughput time (ms / im), maximum VRAM usage (MB), and model size (MB). The results are grouped into five groups. A first baseline (or standard) group comprises two different non-distilled YOLOv8n techniques. A second standard group comprises an inverse technique as described in “Task-oriented feature distillation,” by Zhang et al published in Proc, of NIPS, pp. 14 759-14 771, 2020 and an “avgStd” technique as described in “Online continual learning for robust indoor object recognition,” by Michieli et al in Proc, of IROS, 2023, pp. 3849-3856. Both are similar to hint-based knowledge distillation. A third group of results shows four implementations of our approach which is described above in which Dc is changed. A fourth group of results are generarated using two oracle upper bounds and jointly use the object detector and the SSL pretrained extractor.

[131] A final group of results reports a global upper limit, which is the performance of the detector on the coarse class-set, and the relative performance of our method when compared to the Yolo Only (last feat.) baseline and the Oracle (single fwd.) upper bound. The difference between the two baselines lies in how the vectors input to the FSL are generated: in last feat, we use YOLO’s last feature map only, while in concat. we use all three feature maps and concatenate the results. In the single fwd. oracle setup we use the FSL in the same way as our approach described above, but substitute Ri with 0. In the other scenario, multiple fwd., we use the detector predictions to cut the original RGB image, rather than the feature maps, these sections are then fed into the oracle network to extract a vector embedding. Note that this strategy needs as many forward passes as the number of detected boxes.

[132] Figure 12a shows how our strategy significantly improves performance over the baseline strategies and competitors using the unmodified feature space, regardless of the actual channel differential block implementation. In summary, the detector used has a good trade off between computational cost and performance in absolute terms. The mAP achieved on the training dataset (31 challenging classes) is 35%. Figure 12b shows some of the results in Figure 12a in relative terms with respect to the single forward oracle performance. This is the upper limit score attainable by our knowledge translator and transfer approach. With the proposed implementation of the translation block, we can achieve 80% of the attainable gain, using only about 30% of the inference time and 15% of VRAM of the oracle dual backbone solution.

[133] In more detail, compared to the baseline, AuXFT gains an average of 10.5 mAP in PerSeg, 9.7 mAP in POD 1-shot scenario, and 9.4 in POD 5-shot. This performance is a very close match with the oracle upper bound, where we achieve a consistent 80% relative performance with less than a 1 / 3 of the inference time and VRAM usage. Conversely, due to the scenario mismatch involved in iCubWorld and CORe50, the gains in retrieval accuracy are less impressive in absolute terms, but much more significant when compared to the upper bound. In these datasets, we achieve an astonishing 87.7% of the oracle performance. Another interesting point regards the efficiency of our approach: with just a small increase of 0.3 ms / im in throughput and 65MB of VRAM compared to the baseline, AuXFT 1x1 lin. as well as 3x3 lin. increase the mAP by about 10 points on PerSeg and POD. Given that VRAM and inference time of the two approaches are nearly identical, we choose the 3x3 Hn. as our reference implementation.

[134] We remark that the performance in POD is lower than PerSeg due to the inherently more difficult setup. Indeed, POD has more classes and contains dark scenes. As reported in Figure 12a the global upper limit is 70.0 mAP, but the number rises to 79.0 when only bright scenes are considered (and it decreases to 61.5 for dark scenes only). This difference is reflected in the personalization as well, where our approach can achieve 33.9 (46.9) in bright scenes, on 1- (5-) shot, respectively. In dark scenes the performance is lower, falling to 31.0 (35.0).

[135] To further explore our approach we report some additional experiments in Tables 1 and 2. In Table 1, we study the results when dataset-specific pretraining is employed, that is, the Openimages subset is selected to match exactly the deployment class set. In Table 2, we explore the performance of our approach when coarse-level fallback is enabled in the FSL.

[136] Table 1: Results when pretraining class set matches the deployment set. Average mAP50-95 over 100 episodes (T). Setup PerSeg (13 / 39) POD (15 / 45) 1 -shot 1 -shot 5-shot Yolo Only (last feat.) 34.8±3.0 22.1±2.2 32.0 Yolo Only (concat.) 38.2±2.9 25.6±2.4 30.3 AuXFT (3x3 lin.) 49.5±3.2 33.0±2.7 40.6 Oracle (single fwd.) 60.4±3.2 42.3±2.8 48.2 Oracle (multiple fwd.) 61.2±3.2 47.6±2.9 61.9 AuXFT Rei. Baseline 142.2% 149.3% 126.9% AuXFT Rei. Oracle 82.0% 78.0% 84.2%

[137] Table 2: Results when both fine and coarse classes are predicted in FSL. Average mAP50-95 over 100 episodes (T). Setup PerSeg (13 / 39) POD (15 / 45) 1 -shot 1 -shot 5-shot Yolo Only (last feat.) 19.9±2.7 12.5±1.9 19.3 Yolo Only (concat.) 22.4±2.8 14.1 ±2.3 23.3 AuXFT (3x3 lin.) 34.5±3.2 22.9±2.7 34.1 Oracle (single fwd.) 46.7±3.0 26.9±2.6 38.5 Oracle (multiple fwd.) 49.9±2.9 33.6±2.9 50.1 AuXFT Rei. Baseline 173.4% 183.2% 176.7% AuXFT Rei. Oracle 73.8% 85.1% 88.6%

[138] Table 1 shows that using dataset-specific checkpoints for our approach gains on average 1.3 mAP compared to the unified pretraining results in Figure Aa. This confirms the viability of pretraining the detector once for all considered scenarios. Even more, in the baseline and oracle runs the improvement is inconsistent: in the PerSeg dataset using a restricted class set leads to an average performance decrease of 2.4 mAP, conversely, in POD, we see an improvement of 1.4. This is probably due to the different number of personal instances per coarse class in the two datasets.

[139] The results in Table 2 confirm the effectiveness of our strategy, even in the challenging scenario where the FSL predictions belong to C, which contains 51 and 60 classes, for PerSeg and POD, respectively. In general, the experiments are consistent with all previous cases showing how our approach bridges the gap between the baseline and oracle architectures. Interestingly, in this scenario, the relative gain compared to the baseline is much more significant, with an average of 177.8%. Comparing with the oracle we see a different behavior: in POD the performance increases (compared to results in Figure Aa) by an average of 7.4%, while in PerSeg we see a loss of 5%. We suppose this depends on the unbalanced nature of PerSeg’s class set, which disproportionally penalizes our method when the coarse classes are enabled in the FSL. We remind the reader that PerSeg has multiple classes with only one instance as described when introducing the datasets. Ablation Study

[140] In this section we analyze in more detail some of the design choices made for the proposed architecture. In Table 3, we focus on the distance functions involved in the FSL, showing how their combined use leads to a more stable performance. In Table 4 we show the effect of changing the FSL, comparing our conditional majority voting strategy to other protonet-based strategies. In table 5, we show the performance of the FSL when different residual maps R, are used (as opposed to Ri). For this study, we also generated some qualitative results visually showing the improvement in reconstruction. We generated a colorized version of the oracle (0) and auxiliary (Rf) feature spaces, together with the original input image (augmented according to the training policy) and the pointwise cosine similarity between Ri and 0. The color is generated according to a non-linear embedding which is fit on the oracle features at training time. The channel-dense features into [0,1]3 (RGB space) are mapped using a 1x1 convolution with sigmoid activation function, the embedding is then expanded to the original space with another 1x1 convolution and a reconstruction loss between the original and reconstructed features is minimized.

[141] Table 3: Effect of different distances in the FSL accuracy. Best in bold, second best underlined. cos mAP (T) Coarse / Coarse / 7 7 48.3±3.3 34.6±3.2 7 7 48.6±3.2 33.8±3.1 7 7 48.9±3.3 34.2±3.2 7 48.4±3.3 34.6±3.2 48.8±3.4 34.5±3.2

[142] Table 4: Comparisons between different Few Shot Learners. Model mAP (T) Time [ms / im] (i) Protonet-cos [2] 50.8±3.2 144.8±6.7 Protonet-^ [2] 51.0±3.5 96.1±5.9 SimpleShot [6] 50.8±3.2 94.1±5.8 Ours 48.8±3.4 32.6±5.1

[143] Table 5: Results on residual chain shortcuts. Features mAP (T) Partial I (R3) 48.4±3.5 Partial II (R2) 48.7±3.4 Full (RJ 48.8±3.4

[144] For the analysis in Table 3, we consider three distance functions: cosine, and / 2. The table shows how different distance functions are better depending on whether the coarse classes are considered or not (cosine is best when they are enabled, ^2 otherwise). For this reason, we employ all three distances together, achieving a good compromise (second-best performance in both cases).

[145] We compare the proposed FSL strategy with state-of-the-art competitors in Table 4, measuring both personalization accuracy and time complexity. The table shows how our conditional approach leads to a slight decrease in performance (2.0 mAP compared to the best strategy), but improves the computational cost by more than 3 x. Such a trade-off is fundamental for robotic applications where many vision tasks require real-time performance.

[146] Figure 13 plots the score against expansion rate of the ground truth (GT) padding size for non-square and expanded square boxes. Figure 13 shows how sensitive the mAP is to expansion rate. The results justify the choice of retrieval accuracy made for iCubWorld and CORe50, given that the metric is much more resilient to changes in box size. For further confirmation, we measured the mAP of iCubWorld and obtained 20.9. This is very close to the expected value from the figure (i.e., around 22.5).

[147] The final ablation study we report refers to the residual chain of the auxiliary features: we investigated the performance attained by our method when the partial reconstructions were used instead of Ri. The results shown in Table 5 confirm the improvements brought by the chain: using only R„ (n = 3 in our setup) results in a total of 48.4 mAP which increases to 48.7 when two maps are merged. The best performance is improved again when Ri is also used, slightly raising to 48.8.

[148] In this work, we presented AuXFT a Few-Shot Instance-Level Personalized Object Detection architecture, which translates the features of a CNN-based detector into an auxiliary feature space where cross-architecture knowledge translation is possible. We have designed a framework for personalizing coarse-level detection to instance-level classes using a prototypical-based FSL that receives input vectors generated by our detection-driven feature pooling. We have validated the performance of our architecture on four different personalized object detection datasets, achieving significant gains in all of them. Construction of the auxiliary feature space is highly nontrivial due to the intrinsic architectural differences between the models of interest: in our case, a convolutional network (e.g., YOLOV8n]), and a vision transformer (e.g., DINOv2). The two architectures have different numbers of feature maps, channel depths, and resolution layers. As explained above, to align the feature spaces of the models, we proposed a translator block which exploits two differential modules. The effectiveness of our approach for 1-shot and 5-shot tasks on 3 public datasets (PerSeg, iCubWorld, and CORe50). AuXFT shows significant improvements in all scenarios.

[149] Different detectors and oracle extractors may be used in the future to verify the generalizability of the strategy. We also plan to test new mapping functions in the channel differential block and whether some parametric resampling strategies (such as transposed convolutions) can improve performance when added to the spatial differential module.

[150] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

24 12 241. A computer-implemented method for classifying, on a user device, an image using a machine learning, ML, model, the method comprising:5 receiving an image;obtaining, using an object detector of the ML model, a first classification for at least one object in the image;generating, using a feature extractor of object detector, a set of intermediate features for the image;10 converting, using a translator of the ML model, the set of intermediate features into a set of auxiliary features, wherein the set of auxiliary features is defined within an auxiliary feature space which has a different spatial resolution to an intermediate feature space for the set of intermediate features;generating, using a pooling module of the ML model, an embedding vector which 15 encodes at least some of the set of auxiliary features; andidentifying, using a classifier of the ML model, a second classification for the at least one object in the image byselecting, from a storage on the user device, a plurality of stored vectors each of which have a first classification and a second classification wherein the first classification 20 of each selected vector is the same as the first classification obtained using the object detector;comparing the generated embedding vector to the selected plurality of stored vectors;identifying the stored vector which is most similar to the generated embedding25 vector; andassigning the second classification of the identified stored vector to the at least one object.

2. The method of claim 1, wherein converting each set of intermediate features comprises30 changing, using a channel differential module, a channel depth by mapping the intermediate feature space to the auxiliary feature space so that a channel depth of the intermediate feature space matches the channel depth of the auxiliary feature space.

3. The method of claim 1 or claim 2, further comprising35 obtaining, using the object detector, a bounding box around each object for which a first classification was obtained.24 12 244. The method of claim 3, wherein generating an embedding vector comprises using each bounding box to identify features within the set of intermediate features which are to be 5 summarised into the embedding vector.

5. The method of any one of the preceding claims, wherein the first classification represents a class of objects and for each class of objects, there is at least one second classification representing a user-specific instance within each class of objects.

106. The method of any one of the preceding claims, further comprising controlling the user device based on the identified classification.

7. The method of any one of the preceding claims, wherein prior to receiving the image,15 the method comprises locally training a pre-trained ML model to personalise the ML model, wherein the local training comprises:receiving a training image;obtaining, using the object detector of the pre-trained ML model, a bounding box around at least one object in the training image and a first classification for the at least one object in 20 the bounding box;generating, using a feature extractor of the object detector, a set of intermediate features for the training image;converting, using a translator of the pre-trained ML model, the set of intermediate features into the set of auxiliary features;25 generating, using a pooling module of the pre-trained ML model, an embedding vector for each bounding box by encoding the set of intermediate features within the bounding box;receiving user input in the form of a selection of a bounding box and a second classification for the selected bounding box; andtraining the classifier of the pre-trained ML model to personalise the ML model using the 30 received user input, wherein parameters of the object detector are frozen during training of the classifier.

8. A computer-implemented method for locally training a pre-trained ML model to personalise the ML model, wherein the local training comprises:35 receiving, at a user device, a pre-trained ML model comprising an object detector, a translator, a pooling module and a classifier;24 12 24receiving a training image;obtaining, using the object detector, a bounding box around at least one object in the training image and a first classification for the at least one object in each bounding box;generating, using a feature extractor of the object detector, a set of intermediate features 5 for the training image;converting, using the translator, the set of intermediate features into a set of auxiliary features, wherein the set of auxiliary features is defined within an auxiliary feature space which has a different spatial resolution to an intermediate feature space for the set of intermediate features;10 generating, using the pooling module, an embedding vector for each bounding box by encoding the set of auxiliary features within the bounding box;receiving user input in the form of a selection of a bounding box and a second classification for the selected bounding box; andtraining the classifier using the received user input to personalise the ML model, wherein 15 parameters of the object detector are frozen during training of the classifier.

9. The method of claim 7 or claim 8, wherein training the classifier comprisesteaching the classifier to associate the embedding vector for the selected bounding box with the second classification provided by the user input.2010. The method of claim 9, further comprisingstoring, in internal storage on the user device, each embedded vector with its associated first and second classifications.25 11. The method of claim 10, further comprisingderiving an additional vector for each first classification from all the stored vectors having the same first classification; andstoring the derived additional vector in the internal storage with the stored vectors having the same first classification.3012. The method of claim 11, further comprisingcomputing a new second classification for the derived additional vector by using a distance in embedding space between the derived additional vector and each of the other stored vectors having the same first classification and35 storing the new second classification in the internal storage with the derived additional vector.24 12 2413. The method of any one of claims 7 to 12, further comprising pre-training the ML model prior to deployment on a user device, wherein pre-training comprises using a teacher ML model to teach the ML model.

514. The method of claim 13, wherein the pre-training comprises receiving a set of labelled training inputs, each of which is an image labelled with a bounding box and a first classification for at least one object in the image;for each labelled training input:10 generating, using the teacher ML model, a set of teacher features for the labelledtraining input, wherein the set of teacher features is defined in a teacher feature space which has a first resolution;generating, using the feature extractor of the object detector, the set of intermediate features for the labelled training input;15 converting, using the translator, the set of intermediate features into a set ofauxiliary features, wherein the auxiliary feature space has the same resolution as the first resolution;generating, using the pooling module, an embedding vector for each bounding box by encoding the set of auxiliary features within the bounding box; and20 obtaining, using the object detector, a bounding box around at least one object inthe training image and a first classification for the at least one object in each bounding box; andtraining the ML model bytraining the feature extractor and the translator using the sets of auxiliary features 25 and the sets of teacher features; andtraining the object detector using the set of labelled input data and the obtained bounding boxes and first classifications for each object within each labelled input data.

15. A computer-implemented method for pre-training a student ML model using a teacher 30 ML model, wherein the pre-training comprises:receiving a set of labelled training inputs;for each labelled training input:generating, using the teacher ML model, a set of teacher features for the labelled training input image, wherein the set of teacher features is defined in a teacher feature 35 space which has a first resolution;24 12 24generating, using a feature extractor of the student M L model, a set of intermediate features for the labelled training input, wherein the set of intermediate features is defined in an intermediate feature space which has a second resolution which is different to the first resolution;5 converting, using a translator in the student ML model, the set of intermediatefeatures into a set of auxiliary features, wherein the set of auxiliary features is defined in an auxiliary feature space which has the same resolution as the first resolution;generating, using a pooling module in the student ML model, an embedding vector which encodes at least part of the set of auxiliary features; and10 obtaining, using a primary network within the student ML model, a primary outputfor the labelled training input; and training the student ML model bytraining the feature extractor and the translator using the sets of auxiliary features and the sets of teacher features; and15 training the primary network using the set of labelled input data and the obtainedprimary outputs.

16. The method of claim 15, whereineach training input is an image which is labelled with a bounding box and a first 20 classification for at least one object in the image,generating an embedding vector comprises encoding the set of auxiliary features within the bounding box;obtaining a primary output comprises obtaining, using an object detector, a bounding box around at least one object in the training image and a first classification for the at least 25 one object in each bounding box; andtraining the primary network comprises training the object detector.

17. The method of claim 14 or claim 16, wherein training the feature extractor of the object detector comprises adjusting parameters within the feature extractor of the object detector and 30 the translator to minimise a loss between the set of auxiliary features and the set of teacher features.

18. The method of claim 16 or claim 17, wherein the training the object detector comprises adjusting parameters within the object detector to minimise a loss between the obtained 35 bounding box and first classification and the labelled bounding box and first classification.

19. A user device comprising:a processor coupled to memory, wherein the processor is configured to carry out the method of any one of claims 1 to 14.5 20. A user device according to claim 19 in the form of a vacuum cleaner, lawnmower, mobile device, smart oven or smart fridge.

21. A system comprisinga server and10 a user device according to claim 19 or claim 20, wherein the user device is connected to the server; andwherein the server comprises a processor which is configured to carry out the method of any one of claims 13 to 18.15 22. A non-transitory data carrier carrying code which, when implemented on a processor on a user device, causes the user device to carry out the method of any of claims 1 to 18.24 12 24

Images