Method and system for classifying images

Abstract

Method for classifying an image using a machine learning (ML) model, comprising: extracting features from an input image to generate a feature vector; detecting and categorising an object in the image

Description

Field

[001] Embodiments of the present techniques relate to a method and a system for classifying images, in particular for processing images to control a device, e.g. a home appliance. The images may be processed by a machine learning model which has been personalised by training, for example using few-shot instance-level personalisation. Background

[002] In the last decades, vision models have outperformed human-level accuracy on many recognition benchmarks. Deep learning has produced outstanding results by training all parameters of large models via offline updates over large batches of abundant samples available all at once.

[003] The rise of specialized robots (e.g., indoor domestic and service robots) has driven the recent advancements in automatic scene analyses. However, vision models deployed on robotic agents experience new classes (e.g., new objects) and domains (e.g., new rooms) specific to users at test time; and classical models cannot be easily personalized without suffering from catastrophic forgetting. Hence, Continual Learning (CL) paradigm has emerged to address such adaptation-preservation challenge.

[004] At the same time, robotic agents are equipped with low computation resources and no storage availability due to hardware and privacy constraints. Therefore, online CL (OCL) methods have gained attention to learn from a non-repeated stream of data and tasks. In particular, OCL has been recently investigated for low-resource embedded devices with no data storage available. However, no extensive study exists at the time of writing. Furthermore, when learning new classes, users will only provide a few labelled samples, as the labelling operation is time-consuming and tedious. That is, few-shot (FS) learning should be paired with OCL (FS-OCL).

[005] After learning new concepts, robotic agents move around the environment (e.g., the user home) and discover the same or other instances of newly learned objects in different conditions. Therefore, vision models should be robust to test-time distortion of objects (e.g., different poses, viewpoints, etc) and / or environment (e.g., different illumination, room, etc). For example, a user may want to update the robotic agent to recognize its own pet. The user will likely provide a few images of the pet, e.g., lying on the living room carpet; however, the agent should identify the same pet in other poses (e.g., standing in front of a door) and in other domains (such as a corridor or kitchen).

[006] Moreover, smart devices are starting to be ubiquitous in everyday life and users are demanding increased personalization of vision systems to localize and identify personal instances of objects {e.g , my dog rather than dog) from a few-shot dataset only. Despite outstanding results of deep networks on classical label-abundant benchmarks {e.g , those of the latest YOLOv8 model for standard object detection), they struggle to maintain within-class variability to represent different instances rather than object categories only.

[007] The present applicant has identified the need for an improved technique for personalising machine learning models for image processing. Summary

[008] In a first approach of the present techniques, there is provided a computer-implemented method for classifying, on a user device, an image using a machine learning, ML, model which comprises a feature extractor, a pooling module and a classifier, the method comprising: receiving an input image; generating, using the feature extractor of the ML model, a feature vector for the input image; creating, using the pooling module, a statistical vector comprising at least one statistical moment from the feature vector; concatenating, using the pooling module, the statistical vector to the feature vector to create a concatenated vector; and identifying, using the classifier, a classification for the input image. Identifying the classification is done by comparing the concatenated vector to a plurality of prototype vectors which are stored on the user device, wherein each prototype vector is a statistical representation of a specific classification; identifying, the prototype vector which is most similar to the concatenated vector; and assigning the classification of the identified prototype vector to the input image.

[009] An output based on the identified classification may be generated and may comprise controlling the user device based on the identified classification. The user device may be a robotic device, such as an indoor domestic and service robot like a vacuum cleaner. Merely as an example, the robotic device may receive an instruction to clean close to an object having a specific classification (e.g. clean in front of a specific furniture they have in their home) and when the identified classification corresponds to the specific classification, the robotic device may begin cleaning. More generally, the image processing can be used to provide precise cleaning without harm to a customer’s objects. For example, an image can be processed to assign a classification of a particular object(s) within the image and the correct approach can be used when getting close to the detected object(s). For example, some items such as glass cups (which could be broken) or wires (which could interfere with the robotic device) may be completely avoided and other non-delicate objects (such as backpacks, tables, chairs, etc.) could be closely cleaned around. Other use cases include the user asking the robotic device to monitor where a specific pet (e.g. petA, or petB) is and check their health or to stay away from a specific pet (e.g. petA) when cleaning because they are easily scared by the robotic device. Another use case is to allow personalized cleaning, for example by recognising a type of dirt on the floor and selecting a level of vacuum power to use to clean the particular classification of dirt. Alternatively, the robotic device can select to use a different mode, e.g. mop mode, based on the classification of the dirt in the image.

[010] Splitting the ML model into a feature extractor, a pooling module and a classifier may be considered to be using a modular approach and provides a computationally efficient framework for on-device deployment. In addition to deployment on the user device, the ML model may also be deployed on a connected user device, e.g. a mobile or edge device, such as a smart phone or digital television.

[011] The at least one statistical moment (which may also be termed a statistic and the terms may be used interchangeably) may include average, variance, skewness, and kurtosis. Other high-order statistics may also be used. In other words, the at least one statistical moment may include any combination of nth-order central statistical moments. In mathematics, the moments are a well-known term and the moments of a function may be defined as quantitative measures related to the shape and characteristics of the function’s graph. The statistical vector may comprise all the statistical moments or a subset of the statistical moments. The concatenated vector may be considered to be an enriched distribution of features and may reduce forgetting when applying continual learning. Extraction and usage of high order statistics from the input image (current data) may thus help the ML model to gain extra, useful clues from otherwise scarce input data to improve recognition and robustness.

[012] A user of the user device typically wants personalised object recognition services (e.g. the ability to add new classes to the model or to personalize the users’ instances of objects) on the user device. This may be achieved by using the plurality of prototype vectors which may be termed a search space. Each prototype vector is a prototypical representation of an input sample and the term prototype or prototype model may be used interchangeably with prototype vector. The method may further comprise personalising the search space to a user of the user device, for example by updating the plurality of prototypes vectors which are stored on the user device. The updating may be done in response to an input from a user, e.g. to add a new classification. The updating may be done continuously to generate user data which can then be stored and used for personalisation. The prototype vector may be generated using the concatenated vectors for each image having the particular classification. Thus, the prototype vector may be considered to be user data in the form of feature-level statistics.

[013] As noted above, the prototype vectors are stored on the user device (i.e. locally stored) and may be used to update (train) the classifier. In this way, personalisation of the ML model may be achieved without sharing data with a central server to preserve data privacy. The user device may have a relatively small storage and thus, the prototype vectors are preferably lightweight and thus require less storage than the images and / or the model. By using a prototype which is a statistical representation of each classification, there is also no need to store private replay user data (e.g. the images themselves) on the user device. The use of such prototype vectors (feature-level statistics) may thus result in efficiencies may be with respect to storage and / or memory requirements as well as training and / or inference speed.

[014] For example, each prototype vector may be a mean vector of a Gaussian model or multivariate normal model. Thus, Gaussian parameters rather than a fully connected output layer may be included in the ML model. Each prototype vector may comprise a mean feature vector for each classification and optionally a covariance matrix shared across the classifications may also be generated. By generating both the prototype vector in the form of a mean vector and shared covariance matrix, we may consider that we are building a prototype model. The shared covariance matrix may be equal to the identity matrix. The mean feature vector for a particular classification may be obtained by creating concatenated vectors for each image having the particular classification and averaging the created concatenated vectors.

[015] Each prototype vector may be derived using any suitable probability distribution. For example, Laplacian, Cauchy and other distributions may be used and the parameters of the prototype vector adjusted accordingly. For example, the Laplacian distribution is characterized by two values (p and b). Their meaning can change, for example p may be termed a location parameter and b may be termed the diversity. The probability density function of the Laplace distribution is similar to the normal distribution but the Laplace density is expressed in terms of absolute difference from the mean rather than squared distance from the mean as with the normal distribution. The statistical moments which are used in the statistical vector may be adapted to match the distribution chosen.

[016] Updating the plurality of prototypes may comprise updating an existing prototype or creating a new prototype. Updating an existing prototype may comprise receiving at least one new image which is labelled with a classification corresponding to the existing prototype and adjusting the existing prototype based on the at least one received image. Adjusting the existing prototype may comprise creating a concatenated vector for the at least one new image and updating the mean feature vector for the classification based on the concatenated vector for the at least one new image, for example using: ^yn ’ m7n + Z« mv <------- yn tv +1 yn tv yn «- tv yn +1. where z„ is the concatenated vector, myn on the right-hand side of the equation is the mean feature vector of the existing prototype and on the left-hand side of the equation is the updated mean feature vector and tyn is an index or counter representing the number of samples used to compute the mean feature vector for each classification and yn is a label corresponding to the classification of the new image, where the index n represents an index for the input sample. Here, creating the concatenated vector may be done using the pooling module. In other words, the created concatenated vector of the new image may be used together with the existing prototype vector to calculate the mean feature vector which is output as the updated prototype.

[017] Creating a new prototype may comprise receiving multiple images each of which are labelled with a new classification and calculating the mean feature vector for the new classification from the concatenated vectors for the multiple images. When a covariance matrix is shared across the classifications, the shared covariance matrix may also be updated when updating or creating the mean feature vector. The shared covariance may be updated to generate an updated covariance matrix based on an existing covariance matrix, for example by: _nZn + 8n _n(zn-myn)(zn-myn)T ^n+l । । i ' n + 1 n + 1 where is the existing covariance matrix, Sn+1 is the updated covariance matrix and n, z„ and myn have the definitions provided above.

[018] The updated mean feature vector and updated covariance described above may be termed “running” feature-level statistics given that they are updated in an “online” fashion for personalisation of the ML model. In this manner, the ML model advantageously continually learns how to classify classes and domains not seen during its training.

[019] The input image may be normalized or standardized before generating the feature vector, for example using an input-level data inverse normalization module. Normalizing is used to mean adjusting data to be in the range [0,1] and standardizing is used to mean adjusting the data so that the mean is equal to zero and the standard deviation is 1. Objects and domains change appearance overtime, depending on illumination (day / night), occlusion, blur (camera focus / dust), viewpoint, object pose, etc. Users want robust personalized object recognition services that work in variable conditions of the objects (different pose or appearance) and environment (illumination changes, level of clutter, etc.). For example, when robotic devices such as vacuum cleaners are used in users’ homes, the robotic devices need to understand and interact with the surroundings. They need to understand unseen classes (e.g. users’ pets, shoe organizers or other belongings), unseen domains (e.g. users’ home or spaces within the home) and react to test-time variability due to time-varying factors (e.g. light, occlusion and clutter). The ML model thus needs to identify objects belonging to old classes in new domains, contexts and environments (different rooms, different houses and / or different environmental conditions). Normalizing an input image may help reduce the domain gap, i.e., to bring the input data into a similar domain as the data which was used to train the ML model. Explicit teaching of the system by the user may also help with personalized results as described below.

[020] Normalizing or standardizing the input image may comprise obtaining statistics for the input image; comparing the obtained statistics with stored statistics; and adjusting the input image based on any differences between the obtained statistics and the stored statistics. The use of normalization or standardisation will depend on the image statistics which are used. The stored statistics may be updated with the obtained statistics to provide “running” inputlevel statistics. The stored statistics and the obtained statistics may be any suitable image statistics of an input image, for example a colour histogram, white point, colour jitter parameters and so on. The stored statistics may be historical data and may be obtained from some or all previous input images which have been received, including any input images which were used to generate the prototype models and / or train the ML model. The stored data is only the image statistics and not the full images (neither the input image or previous images), thus is a data efficient and low-footprint method. The efficiency may be with respect to storage and / or memory requirements as well as training and / or inference speed. These image statistics may also be generated during the training process.

[021] Identifying a classification for the input image may comprise identifying a class, e.g. an object category, for the input image. Each object category such as ball, bottle etc. may comprise different types or instances of the objects each of which has an instance-level category such as balh (e.g. tennis ball), balk (e.g. football) and bottlei (e.g. John’s bottle), bottle2 (e.g. my bottle). In other words, an instance-level category is a sub-category or subclass for a specific type or instance of an object within the object category. It is helpful to further classify these objects. Thus, identifying a classification for the input image may alternatively comprise identifying a sub-class, e.g. an instance-level category. When identifying a sub-class, the method may comprise determining, using an object detector, an object category (class) for at least one detected object in the input image; selecting prototype vectors from the plurality of prototype vectors which have the determined object category; comparing the concatenated vector for the input image only to each selected prototype model to identify the prototype vector which is most similar to the concatenated vector; and assigning the instance-level category (sub-class) of the identified prototype vector to the input image. In other words, the search space of prototype vectors has been simplified to only within the instances whose object category matches the object category assigned to the input image, e.g. once the object is identified as a ball to only consider whether the input image is a tennis ball or a football. More than one object may be detected in the image and thus multiple object categories may be assigned to the input image. Such a pipeline may be thought of as an Object-conditioned Bag of Instances since generic category level output (e.g. ball) is converted to specific personal-level output (e.g. football) via conditional nearest prototype selection.

[022] The object detector may also be termed a detection head. The object detector may detect at least one object in the input image and may determine a location within the input image for the at least one detected object. Creating the concatenated vector may use the determined location, e.g. by creating a concatenated vector for the portion of the image at the location. In this arrangement, the feature extractor may be termed an object detection backbone or encoder and the terms may be used interchangeable.

[023] According to a second aspect of the present techniques, there may be provided a computer-implemented method for personalising a machine learning, ML, model, the method comprising: inputting an image into an object detector to detect at least one object within the input image; determining, using the object detector, an initial class label for the at least one detected object; determining, using the object detector, a location within the image for the at least one detected object; generating a feature vector using the determined location; inputting the generated feature vector and the initial class label to a learner module; identifying, using the learner module, an updated class label for the at least one detected object within a search space containing only instances having an object category which matches the initial class label. In other words, the second aspect may also be considered as a method for classifying, on a user device, an image using a machine learning, ML, model which comprises a feature extractor, an object detector, a pooling module and a classifier, the method comprising: receiving an input image; determining, using the object detector, an object category for at least one detected object in the input image; generating, using the feature extractor of the ML model, a feature vector for the input image; creating, using the pooling module, a statistical vector comprising at least one statistical moment which is extracted from the feature vector; concatenating, using the pooling module, the statistical vector with the feature vector to create a concatenated vector; identifying, using the classifier, a classification for the input image by selecting prototype vectors from the plurality of prototype vectors which have the determined object category and at least one instance-level category, wherein an instance-level category is a sub-category for a specific instance of an object within the object category; comparing the concatenated vector for the input image only to each selected prototype vectors to identify the prototype vector which is most similar to the concatenated vector; and assigning the instancelevel category of the identified prototype vector to the input image. The ML model of the second aspect may further comprise an object detector, as described above.

[024] The location may be defined by bounding box coordinates. The method may further comprise masking regions within the image outside the location of the at least one detected object, e.g. outside the bounding box coordinates. Generating the concatenated vector may comprise extracting, using the pooling module (which may also be termed a reduction module), the at least one statistical moment from a feature vector which is generated from the portion of the image at the location, e.g. within the bounding box where used. The method may further comprise rescaling coordinates of the location to match a dimension of the feature vector.

[025] The classifier may also be termed a learner module and may be a prototype based fewshot learner (PFSL). As explained above, the learner module and / or the search space over which it searches may be termed an object-conditioned bag of instances (OBol). When personalising the training, the object conditioned element may be frozen and the bag of instances may be trainable. The search space comprises only a few-shot labelled user data. Using the techniques described above, the generic category level output (e.g. the initial class label or object category) is converted to a specific personal-level output (e.g. the updated class label or instance-level category) via nearest prototype selection. Thus according to another aspect, we describe a method of online continual learning for robust indoor object recognition which incorporates the object-conditioned bag of instances approach described above.

[026] As an addition to, or as an alternative to, the personalisation using the search space, the method may comprise receiving a pre-trained ML model from a server and locally training the ML model to personalise the model. The local training may be done on the user device itself or on a second user device which is connectable to the user device. This local training may use continual learning and complement the personalisation of the search space described above. The user device(s) may have relatively small processing power when compared to a server and thus may not be able to train the whole model. Thus, during the local training, the feature extractor may be frozen and only the classifier is trainable, for example using continual learning. By using the statistical moments, an enriched distribution is used in the training which helps to reduce the risk of catastrophic forgetting. In other words, there is a data-efficient framework which is set-up for few-shot learning. New classifications may be learned from as few as five samples.

[027] Accordingly, the method may further comprise locally training the ML model to personalise the ML model. The local training may be done prior to receiving an input image for classification. The local training may also be updated as explained above. The local training comprises: receiving a training set of images each of which are labelled with at least one classification (class and / or sub-class); generating, using the feature extractor of the ML model, a feature vector for the input image or a portion of the input image; creating, using the pooling module, a statistical vector comprising at least one statistical moment which is extracted from the feature vector; adding, using the pooling module, the statistical vector to the feature vector to create a concatenated vector; and computing a prototype vector for a specific classification using the concatenated vectors for each training image labelled with the specific classification. In other words, the classifier has been trained using the computed prototype models by updating the parameters of the prototype vector and when used the shared covariance matrix. In other words, we may be considered to be updating the parameters of the multivariate normal model.

[028] According to a third aspect of the present techniques, there may be provided a computer-implemented method for locally training the ML model to personalise the ML model, wherein local training comprises: receiving a training set of images comprising a plurality of images each of which is labelled with a classification; generating, using the feature extractor of the ML model, a feature vector for the input image or a portion of the input image (when using an object detector); creating, using the pooling module, a statistical vector comprising at least one statistical moment which is extracted from the feature vector; concatenating, using the pooling module, the statistical vector to the feature vector to create a concatenated vector; and computing a prototype vector for a specific classification using the concatenated vectors for each training image labelled with the specific classification. Image statistics may be generated for each image in the training set and the generated statistics may be stored as stored statistics for normalization or standardization of an input image.

[029] The features relating to the generation of an output based on the identified classification and controlling a user device based on the identified classification described above in relation to the first aspect also apply to the second and third aspects, and their description is not repeated for conciseness.

[030] According to another aspect, there is provided a user device which is adapted for carrying out the methods described above. The user device may be part of a system which is adapted for personalising a machine learning, ML, model as described above. The system may comprise: a server; a primary user device (or first user device) connected to the server, the primary user device comprising: a processor coupled to memory, wherein the processor is configured to: receive a pre-trained ML model from the server and locally train the ML model to personalise the model according to the third aspect of the present techniques described above. The system may further comprise a secondary user device (or second user device) which comprises a processor coupled to memory, wherein the processor is configured to: receive a pre-trained ML model from the server or the primary user device; locally train the ML model to personalise the model and then send the locally trained ML model to the primary user device for use on the primary user device. Such an arrangement may be useful when the primary user device is not able to train the model itself.

[031] In summary, the solution described above achieves on-device continual learning without catastrophic forgetting and without storing samples.

[032] In each approach, the user device (or client device) may be a constrained-resource device but typically has the minimum hardware capabilities to use a trained neural network / ML model. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, a vacuum cleaner, an image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge). It will be understood that this is a non-exhaustive and non-limiting list of example client devices. For some of these devices, e.g. a vacuum cleaner, there may not be the minimum hardware capabilities to perform the local training and this may be done on a second user device, e.g. a smartphone, tablet, laptop, computer or computing device which has more capability including a user interface for a user to input labelled training data. The second user device may be local to the first user device (e.g. within the same household) whereby the training may be described as local training. The second user device may be connectable to the first user device to transfer the locally trained ML model.

[033] In a related approach of the present techniques, there is provided a second user device such as a personal computer, a laptop, a smartphone or tablet computer and / or other mobile device which is used for training / personalisation and a more resource constrained device, such as a robotic device or refrigerator which is configured to implement the image processing as described above. Similarly, in a related approach of the present techniques, there is provided a server which is configured to implement the steps carried out by the server as described above. Similarly, in a related approach of the present techniques, there is provided a system comprising a server which is connected to multiple user devices and the server and user devices cooperate to implement the steps described above.

[034] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[035] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[036] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[037] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[038] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[039] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[040] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[041] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[042] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[043] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[044] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), deep Q-networks, and transformer or visual transformer networks.

[045] The learning algorithm is a method for training a predetermined target device (for example, a user device or client device) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semisupervised learning, or reinforcement learning. Brief description of the drawings

[046] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[047] Figure 1 is a flowchart showing example steps of how a machine learning ML model may be personalised and used;

[048] Figure 2a is a schematic diagram of an example ML model which can be trained to process images;

[049] Figure 2b is a schematic diagram showing how a ML model such as that shown in Figure 2a can be locally trained to personalise the ML model;

[050] Figures 3a to 3c plot the three statistical moments (average, variance and skewness) for clean and augmented data;

[051] Figure 4 is a flowchart of example steps in a method for locally training a ML model to personalise the ML model for image processing;

[052] Figure 5a is a flowchart showing example steps in a method for creating a training set of images;

[053] Figure 5b is a flowchart showing steps of an alternative method for creating a training set of images to the method shown in Figure 5a;

[054] Figure 5c is a flowchart showing steps of an alternative method for creating a training set of images to the methods shown in Figures 5a and 5b;

[055] Figures 5d to 5f are illustrative screenshots which may be presented during the implementation of Figure 5a;

[056] Figure 6a is a schematic diagram of an example ML model for processing images;

[057] Figure 6b is a schematic diagram showing an example of how a ML model such as that shown in Figure 6a can be used to process input data;

[058] Figure 7a is a flowchart showing example steps of how a machine learning ML may be used to process an image;

[059] Figure 7b is a flowchart showing example steps of how an input image may be normalised;

[060] Figure 7c is a flowchart showing example steps of how a classification may be identified using a classifier of a ML model;

[061] Figure 8 is a schematic diagram of an example ML model for processing images for instance-level classification;

[062] Figure 9 is a flowchart showing the steps of the method for training an ML model such as that shown in Figure 8;

[063] Figures 10a to 10d illustrate the intermediate outputs at steps in the method of Figure 9;

[064] Figure 11 is a flowchart showing the steps of using the trained ML model such as that shown in Figure 8;

[065] Figure 12 is a schematic illustration of a system using the methods described above;

[066] Figure 13a is an example of a user device which can use the methods described above;

[067] Figure 13b is a flowchart for known training of a vacuum cleaner recognition model (VCRM) which can be used on the device of Figure 13a;

[068] Figure 13c is a flowchart for the present techniques for training of a vacuum cleaner recognition model (VCRM) which can be used on the device of Figure 13a;

[069] Figure 13d outlines how the steps of updating the VCRM on the vacuum cleaner;

[070] Figures 13e and 13f show the steps for updating the VCRM on the vacuum cleaner or a secondary user device respectively;

[071] Figure 14 is a table showing the accuracy of the present techniques and several known techniques on same-domain OpenLORIS data on 16 backbones and 10 OCL baselines;

[072] Figure 15 is a table showing accuracy of the present techniques and several known techniques on real other-domain few-shot datasets on ResNets and ViTs backbones;

[073] Figure 16 shows the per-step accuracy of ResNet-152 on the OpenLORIS-small training dataset;

[074] Figure 17a is a table showing results of experiments performed on OpenLORIS using the best CNN;

[075] Figure 17b is a table showing results of experiments performed on OpenLORIS using the best transformer;

[076] Figure 17c is a table showing the accuracy on other-domain data with controlled augmentations on ResNet-152 on the few-shot benchmark;

[077] Figure 17d is a table showing the accuracy of multiple pooling mechanisms on OpenLORIS-small and SLDA;

[078] Figure 17e is a table showing results of additional metrics on the OpenLORIS-small and ResNet-152;

[079] Figure 18a shows same domain accuracy and per-object accuracy for three object-conditioned bag of instances, OBols, methods via ProtoNet;

[080] Figure 18b shows other domain accuracy and per-object accuracy for three OBols methods via ProtoNet; and

[081] Figure 19 shows accuracy at variable shots comparing two known methods with corresponding adaptations using the present techniques. Detailed description of the drawings

[082] Broadly speaking, embodiments of the present techniques provide methods, systems and apparatuses for training and using ML models to process images. For example, the training may personalise a ML model by training a classifier of the ML model using prototype models computed from a training set of images labelled with a classification. The ML model may be used to generate an output based on a classification identified from an input image. For example, the output may comprise instructions to control the operation of a robot storing the ML model. The classification is identified by comparing a concatenated vector created based on the input image to a plurality of prototype models and by identifying the prototype model which is most similar to the concatenated vector.

[083] Vision systems mounted on home robots need to interact with unseen classes in changing environments. Robots have limited computational resources, labelled data and storage capability. These requirements pose some unique challenges: models should adapt without forgetting past knowledge in a data- and parameter-efficient way. The problem may be characterized as few-shot (FS) online continual learning (OCL), where robotic agents learn from a non-repeated stream of few-shot data updating only a few model parameters. Additionally, such models experience variable conditions at test time, where objects may appear in different poses {e.g , horizontal or vertical) and environments {e.g , day or night).

[084] Figure 1 is a flowchart showing example steps of how a machine learning ML may be personalised and used. At step S100, a global model is pre-trained. The global model may be pre-trained (or trained, and the terms will be used interchangeably herein) on a central server (which may be termed a factory). Typically, the global model is trained on a large-scale global training dataset (general purpose dataset) on a pre-defined set of classes and a limited set of acquisition domains. Merely as examples, the pre-defined classes may include standard objects such as a sofa, a vase and a table and chairs. The limited set of domains may include rooms within a house, e.g. a kitchen and living room. Although there can be many pre-defined classes and many domains (for example thousands of samples), it is impossible to cover all existing objects and domains during deployment, new objects may appear quickly and different scenes will be encountered.

[085] Model pre-training requires large computational power (for example hundreds of GPUs / TPUs) to update the whole model via large batches. That is, the parameters of the whole model are updated during training. Large storage is also required for the large batch sizes. Furthermore, as described above, the global model is exposed to a pre-defined set of classes with limited domains during its training. The global classes and domains seen by the global model during training may be different to the classes and domains seen by the model at test or inference time, despite the global training dataset typically being very large (the terms test time and inference time will be used interchangeably herein). Consequently, the accuracy of the global model on novel, unseen classes and domains not seen during training may be lower than expected.

[086] At step S102, the global model is deployed from the central server for personalization to the user environment. The global model may be deployed to at least one user device. In other words, the model may be embedded into user devices, e.g. home robots, and shipped to users. Prior to deployment, the pre-trained model may be compressed to reduce the data transfer burden associated with deploying the model. The or each user device may be any electronic device which utilizes computer vision. For example, the or each user device may be a smart vacuum cleaner, smart domestic appliance (such as a smart oven, e.g. to recognise new dishes or smart fridge, e.g. to recognise new items in the fridge), smartphone, tablet, laptop or other robotic device. However, it will be appreciated that the user device(s) may be any suitable electronic device. Once deployed to the user device(s), the global model is personalised to generate a personalised model. The global model may be personalised based on the environment of the user device(s). For example, in cases where the user device(s) is a robot operating in the home of a user, users customize the recognition model to discover novel classes (e.g. lamp, glass, chair, toy) in their target domain / environment (e.g., user home). That is, the model is personalised based on the domains and classes in the user’s home which are typically different to the global classes and domains seen during training. For example, a user’s kitchen or a user’s bedroom may differ from the kitchens and associated objects / classes used to train the ML model.

[087] Different users typically have very different homes and objects, depending for example on regional area (e.g. certain products only exist in certain regions or may look different in different regions - compare a classical Korean table with a modern Italian table), level of wealth (e.g. rich people have larger houses with more expensive objects, for example pool tables and golf bags), personal taste (e.g. certain people may prefer certain furniture) and the number of people / animals living in the house (e.g. babies, adults and pets require different objects and user’s pets are unique and visually different for one another). In other words, object classes can have different appearances in different regions or in different houses. Similarly domains can have different appearances in different regions, for example a traditional Japanese house looks very different to a modern US house, due to different levels of wealth or different tastes. With our proposed system, we can personalize any massive Al model to the final user so that there is an enhanced user experience and the user is more engaged with the technology.

[088] User devices such as robots do not have large storage due to privacy and storage constraints, and cannot update the whole model due to computational limitations. Large batches are also unavailable, since no storage and low RAM is assumed. Furthermore, users do not label many samples as it is a time-consuming operation. Therefore, the ML model is personalised by locally training the ML model on a few labelled samples which may be provided by the user. In contrast to the training of the model on the central server, there is no update of the parameters of the whole model during the personalisation. Instead, only a subset of the model parameters are updated due to the constrained resources of the user device. Moreover, the batch size is reduced compared to the batch size used to train the backbone network. The batch size is typically 1, since the user device cannot store large amounts of data and there may be additional privacy constraints. In other words, the user device should use the sample as it is acquired. In other words, the model undergoes continual learning, CL, whereby the classes and domains used to train the model are incremented; CL is said to be both class- and domain-incremental. Personalisation in this way requires a small storage, RAM and computation capability.

[089] The training problem may be characterised as few-shot (FS) online continual learning (OCL), where the device learn from a non-repeated stream (i.e. online) of few-shot data and update only a few model parameters. The personalisation may be termed a Robust Online Continual Learning method (RobOCLe). As explained in more detail below, this new method constructs an enriched feature space computing high order statistical moments from the embedded features of samples; and computes similarity between high order statistics of the samples on the enriched feature space, and predicts their class labels. The robustness of existing and the proposed CL models to train / test augmentations is also evaluated in various cases. As shown below different moments allow RobOCLe to capture different properties of deformations, providing higher robustness with no decrease of inference speed. As explained below, we can also bring robust on-device personalisation to the devices (e.g. robots) in a user’s home. The personalisation can be parameter-efficient personalisation via continually learning new classes over time and online learning may be used to avoid storing any data sample.

[090] At step S104, the personalised model is used in the user environment. For example, the personalised model may be used to navigate the movement of a robot in a user’s home. The robot navigates the new environment and discovers recently-added classes in unseen domains and conditions, e.g., under variable illumination, occlusion, blurring and geometric distortions. Steps S102 and S104 may be alternated at any time. In other words, as the robot navigates a user’s home, the robot encounters new classes and domains which the robot must accurately classify / recognize to effectively navigate the user environment. Naturally, the characteristics of the images used by the user device to navigate will also vary. For example, over the course of the day, the illumination of the images will vary from the morning, where there is an abundance of natural light, to night time where the images will be darker. Additionally, the new classes encountered by the user device may be hidden or occluded by other objects in the user environment. For example, a soft toy encountered by the user device may be partially obscured by another object in the image. That is, the occlusion of the images may vary. Other examples of varying image characteristics are blur (where the images are not in focus) and geometrical distortion (where the new classes are oriented at different angles within an image). Thus, there may be considerable inference-time variability due to timevarying factors. Consequently, there is a need for personalised object recognition models which are robust to inference-time variability.

[091] Figure 2a is a schematic diagram of an example ML model 200 in the form of an artificial neural network which can be deployed on a user device for processing images and can be trained by the new Robust Online Continual Learning method (RobOCLe). The model 200 comprises 3 main blocks: a feature extractor 6(-) 202, a pooling scheme or module P(-) 204 and a classifier F(-) 206 which together categorize samples or an input image 208 from their extracted features to generate an output 210. The overall model may be expressed as F(P(6(xnk))) and is incrementally trained via supervised OCL updates over subsequent tasks to generate predictions ynk, where xn k is the n-th input sample, n e [Nk], of the Zc-th task, k e [*]■

[092] Following traditional transfer learning setups, the feature extractor 202 is a backbone network {e.g , ResNets - K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,”in CVPR, 2016, pp. 770-778) pre-trained on a server on public benchmark training datasets, with large batch sizes over many epochs updating the whole model (step S100 of Fig. 1). When 6(-) is embedded in a user device, such as robotic agents, it experiences new domains {e.g , user environments) compared to training ones.

[093] As noted above, users want to personalize models to recognize personal objects. Therefore, as explained in more detail below, during training the novel classifier F(-) adapts the general features extracted by 6(-) to the class set of particular users (step S102 in Fig. 1). Users show a few labeled samples of the new classes to recognize, and the devices do not have storage or large computation resources. Therefore, F(-) is trained using online updates, seeing a non-repeated stream of few-shot labelled samples using a frozen feature extractor 6(-). In the experiments described below, the most challenging case of a single class being learned per task is considered.

[094] Figure 2b is a schematic diagram showing an example of how a ML model similar to that shown in Figure 2a can be locally trained to personalise the ML model. A training dataset 212 comprising a plurality of labelled data items is provided as input to the backbone feature extractor network 202. The training dataset is few-shot labelled data and may for example be a plurality of labelled images (e.g. around 1-10 images, or any suitable number of images), where each image is labelled with the same classification. The classification is a user-specific classification which the ML model would typically not have been exposed to during the pretraining in step S100 of Figure 1. That is, the classification may be understood as a new classification (or class, and the terms will be used interchangeably herein). The few-shot labelled data is in the form (x,y), where x denotes the input data and y denotes the label.

[095] Each input 208 from the training dataset is received by the backbone feature extractor network 202 to generate a feature map or feature vector (the terms will be used interchangeably herein) for each data item in the training dataset. The dimensionality of the feature vector is generally lower than the dimensionality of the few-shot labelled data provided as input by the feature extractor 202. The parameters of the backbone feature extractor 202 are frozen and are not updated during the local training. The backbone feature extractor 202 may be any suitable feature extractor, such as MobileNets-V3 (A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, and H. Adam, “Searching for MobileNetv3,” in ICCV, 2019, pp. 1314-132) or ResNets (K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770-778).

[096] The features from the feature extractor network 202 are received by the pooling module 204 which extracts at least one statistical moment from the feature vector. Pooling has been one of the key enablers for deep learning models mapping high dimensional features into a summary of active features. Multiple schemes have been proposed to extract the most useful information from the input data. The pooling mechanism of the present techniques is compared with the following schemes: average pooling (AVG) (Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” IEEE, 1998); max pooling (MAX) (M. Riesenhuber and T. Poggio, “Hierarchical models of object recognition in cortex,” Nature NN, vol. 2, no. 11, pp. 1019-1025, 1999); a linear combination of AVG and MAX (MIX) (Q. Zhou, Z. Qu, and C. Cao, “Mixed pooling and richer attention feature fusion for crack detection,” PRL, vol. 145, pp. 96-102, 2021); a dropout-inspired probabilistic selection of activations based on a multinomial distribution fit (STOCHASTIC) (D. Zeiler and R. Fergus, “Stochastic pooling for regularization of deep convolutional neural networks,’’ ICLR, 2013); a concatenation of top-k% of features to discover information spread over the spatial map (RAP) (S. Bera and V. K. Shrivastava, “Effect of pooling strategy on convolutional neural network for classification of hyperspectral remote sensing images,’’ IET IP, vol. 14, no. 3, pp. 480-486, 2020); a concatenation of AVG and MAX pooling (AVGMAX) (J. Monteiro, M. J. Alam, and T. Falk, “On the performance of timepooling strategies for end-to-end spoken language identification,’’ in LREC, 2020, pp. 3566-3572); an average of the p-th order non-central moment (Lp-norm) (J. Feng, B. Ni, Q. Tian, and S. Yan, “Geometric Ip-norm feature pooling for image classification,” in CVPR, 2011); and iSQRT-COV (P. Li, J. Xie, Q. Wang, and Z. Gao, “Towards faster training of global covariance pooling networks by iterative matrix square root normalization,” in CVPR, 2018, pp. 947-955), which computes second-order covariance pooling via iterative matrix square root normalization. The present techniques consider high order central moments (II. Michieli, P. P. Parada, and M. Ozay, “Online continual learning in keyword spotting for low-resource devices via pooling high-order temporal statistics,” INTERSPEECH, 2023) to discover robust features from limited input data.

[097] High order pooling. A pooling scheme is generally inserted to summarize the input information and reduce the feature map size so that the classifier F(-) can be applied over the pre-trained backbone 6(-) during training. In this way at test time (i.e. at inference), the user device should recognize personal classes in a variety of conditions (e.g., illumination, occlusion, blur, pose). Therefore, the overall OCL procedure should be robust to such factors of variations at test time. With this aim in mind, the pooling PQ) is performed, employing higher order statistical moments to increase the amount of clues extracted from the input samples. In particular, PQ) computes and concatenates the first R statistical moments from the output of 6(-). More formally, P(g) = 11 g-g Eg[{g-g)2^ (1) where g = G(x.nk) e j^hxwxd where anc| w are spatial sizes of features and d is the number of channels, £$[-] is the expectation overg is the empirical mean, and 11 (.) denotes the concatenation operation. That is, P(g) e ]RR d concatenates the first R moments of g and such moments characterize the distribution £ as well as capturing rich spatial statistics to improve recognition accuracy.

[098] As noted above, the pooling module summarizes the input information and reduces the size of the feature vector. The at least one statistical moment may include average, variance, skewness, and kurtosis. In other words, the at least one statistical moment may include any combination of nth-order central statistical moments. Figures 3a to 3c show the distribution of the first three statistical moments in z„. clw'. Wasserstein distance. Figure 3a shows the distribution of the first statistical moment, namely average (AVG), of features extracted from clean versus, augmented samples. Similarly, Figures 3b and 3c show the distribution of the second and third statistical moments (variance and skewness) of features extracted from clean versus augmented samples. Figure 3a shows that AVG has more variability (higher Wasserstein distance) than higher moments when changing domain. As further shown in the results below, high order moments increase robustness of OCL models and their invariance to augmentations on both same-domain and other-domain test data ( / .e., when models receive test data in other conditions than at train time).

[099] Other high-order statistics may also be used. The concatenated vector may be considered to be an enriched distribution of features and may reduce forgetting when applying continual learning. Extraction and usage of high order statistics from the input image (current data) may thus help the ML model to gain extra, useful clues from otherwise scarce input data to improve recognition accuracy and robustness. Additionally, the pooling module adds the at least one statistical moment to the feature vector to create a concatenated vector. The concatenated vector may be considered to be a vector of enriched features. Returning to Figure 2b, these moments (enriched features) are then fed to the classifier 206.

[100] Similarity estimation. The classifier F(-) 206 should be representative of the new classes and extremely lightweight for on-device deployment. A prototype model which may also be termed running feature-level statistics is computed for a specific classification using the concatenated vectors for each data item labelled with the specific classification. Two example suitable methods are to extend the Nearest Classifier Mean (NCM) and the Linear Discriminant Analysis (LDA) to support the streaming setup (T. L. Hayes and C. Kanan, “Lifelong machine learning with deep streaming linear discriminant analysis," in CVPRW, 2020) and the high order pooling. Streaming NCM and LDA are denoted by NCM and SLDA.

[101] The method which extends the NCM may be termed RobOCLeNCM and the method which extends the LDA may be termed RobOCLe SLDA. Both methods compute a running mean feature vector per class ( / .e , the c-th class prototype vector mc e ]RR d, Vc e C) each with an associated counter denoting the number of samples employed to compute each average value (tc) using mv , yn.k hn.k^Vn.k^^ tv , +1 yn,k tv tv + 1. yn,k yn.k (2) where mynk is the running mean vector and tynk is the associated counter, ynk is the label for a new data vector xnk, and znk is the embedding of the new data vector using znk = P(G(xn,k)) e ^'d.

[102] The second method RobOCLe SLDA computes one channel-wise covariance matrix of features shared across classes (2 e ]Rdxd) that is updated online via the techniques described in Dasgupta and D. Hsu, “On-line estimation with the multivariate Gaussian distribution," in ICCLT. Springer, 2007, pp. 278-292. The implementation from T. L. Hayes and C. Kanan is used to update the S and compute predictions. For example, running covariance may be computed from + 8„ n(zn - mv )(z„ - mv )T y _ fl fl p _ \ ri yn7K n yn' ^n+1 ~ , i ' °n — „ i i ' W n + 1 n + 1 where (to simplify notation, the task index k is removed) is the covariance matrix, myi is the running mean feature vector and z„ is the embedding vector of the new data.

[103] RobOCLe NCM is a special case of RobOCLe SLDA where S is equal to the identity matrix. For inference on new samples, RobOCLe NCM assigns the label of the nearest prototype according to its F2 distance on the enriched feature space. The original NCM has shown to be a simple yet effective baseline in CL. During inference, RobOCLe SLDA runs inference on a new sample (to simplify notation, the task index k is removed) xn by F(P(G(x„))) = Wz„ + b, where W e ]RlelxR d and b e ]R|e|, where | -1 is the set cardinality. Rows of W are computed by wc = AmJ, and elements of b are computed by bc = -0.5(mc • Amc) = (mc • wc) with the shrinking approximation A = [(1 — e)S + el]-1 with parameter 6 = 10-4. RobOCLeSLDA assigns to a new sample the label of the closest Gaussian model in the feature space defined using the running class means and shared 2.

[104] The prototype models (e.g. running class mean vectors and optionally running covariance matrices - also termed feature-level statistics) are stored on the user device in a database 216. The stored feature-level statistics help to achieve efficient personalisation of the ML model on the user device. The feature-level statistics capture rich information on the few-shot labelled data provided as input which is personal to the user. Advantageously, the feature-level statistics also occupy a smaller proportion of the storage of the device compared to (say) storing a high-resolution image in its entirety. Thus, a significant amount of information can be extracted from the personal few-shot labelled data provided as input and saved on the user device in a way that is resource efficient. The stored feature-level statistics also help to alleviate the problem of catastrophic forgetting, since the stored information can be re-used and updated at test-time or for further training of the model.

[105] The classifier 206 is trained using the computed prototype models. The classifier obtains a prediction for the classification of each data item of the training dataset by comparing the concatenated vector to the computed prototype models and identifying the prototype model which is most similar to the concatenated vector. A loss function may be computed using the labels and the obtained predictions.

[106] The model may be calibrated using a calibration module 220 to make the obtained predictions more reliable. Generally speaking, the obtained predictions for the classifications described above may have an associated confidence or confidence score i.e. probability of correctness. For example, given 100 predicted classifications each with an associated confidence score of 0.8, it would be expected that 80 should be correctly classified. However, the number of correct classifications may be different to the number expected based on the confidence values. In other words, there may be a discrepancy between the accuracy and confidence of the model. Therefore, the model may be calibrated to align the obtained predictions with a confidence score. The aim of the calibration is to reduce uncertainty and to promote representation invariance via normalization of predictions on deformed test sets. The model may be calibrated using any suitable technique, such as temperature scaling (C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, “On Calibration of Modern Neural Networks" published on ArXiv). Typically, model calibration introduces at least one new parameter to modify the confidence scores of the obtained predictions. For example, temperature scaling works by introducing a new ‘temperature’ parameter and performing a softmax operation using the temperature parameter to modify the confidence predictions. Other calibration techniques operate in similar ways, but where the implementation of the new parameters and their corresponding functionalities varies in each technique.

[107] Input-level image statistics may also be computed from the training dataset 212, for example a colour histogram, white point, colour jitter parameters and so on. The input-level statistics may also be saved and stored on the user device in a database 214. Collectively, the stored feature-level and input-level statistics may be termed stored statistics and may be stored in common database 218. The stored image statistics may be used as part of the normalization of input data at inference time. The stored statistics may be considered to be a lower-dimensional representation of the input data, and thus occupy a small proportion of the user device storage once stored (for example the input level statistics may be less than 0.2Mb). The stored statistics also provide enhanced user privacy since only statistical information is stored on the device, rather than the input images themselves.

[108] Figure 4 is a flowchart of example steps in a method for locally training a ML model such as the model shown in Figures 2a and 2b to personalise the ML model for image processing. At step S400, the method comprises receiving a training set of images each of which are labelled with a classification. The training set of images may be a plurality of images with labels provided by the user, as described below.

[109] At step S402, the method comprises generating, using the feature extractor of the ML model, a feature vector for the input image. The feature extractor may also be considered a backbone feature extractor wherein the parameters are frozen during the training. For example, the feature extractor may be a neural network such as MobileNets-V3 or ResNets (cited above), or any suitable feature extractor.

[110] At step S404, the method comprises extracting, using the pooling module, at least one statistical moment from the feature vector. The at least one statistical moment may include average, variance, skewness, and kurtosis, and other high-order statistical moments may also be used. The method further comprises creating, using the pooling module, a statistical vector comprising the at least one statistical moment and concatenating this statistical vector to the feature vector to create a concatenated vector at step S406.

[111] At step S408, the method comprises computing, using the classifier, a prototype model for a specific classification using the concatenated vectors for each training image labelled with the specific classification. At step S410, the classifier is trained using the computed prototype models.

[112] The training is done using continual learning. Continual Learning has many definitions. Classical CL assumes data released at incremental steps and multiple training epochs performed with large batch sizes updating the whole net. Online CL (OCL) assumes, instead, that data comes as a non-repeated stream of small-batch samples. OCL methods are mainly based on replay due to the inherent difficulty to prevent forgetting. Recently, OCL for embedded devices further restricts the scope to parameter-efficient updates with unitary batch size due to low-resource devices. In parallel, few-shot CL (FS-CL and FS-OCL) train a model to recognize new classes based on few labelled data only, via fast and efficient model updates with minimal human effort. Among all these definitions, three main scenarios have been considered: 1) class-incremental (Cl) where new classes are introduced to models over time; 2) domain-incremental (DI) where the same problem is learned in different contexts (e.g., with a fixed set of classes); and 3) task-incremental (Tl), where new distinct tasks are presented to models.

[113] The challenges of continual learning include catastrophic forgetting. When learning multiple problems sequentially, ML models tend to forget old problems that they have not experienced fora long time. Additionally, as explained above, user will not label large amounts of data and thus there is a need to learn from few-shot data. The models also need to trained, and hence updated, on low-resource user devices such as robots. Finally, the models need to be robust to test-time variations (e.g. light, occlusion, geometric modifications, blur, clutter, etc.). The current CL methods are mainly focussed on classical CL which assumes a large availability of data, storage availability to replay data points several times (non-online setups), fine tuning of the whole network parameters and computationally expensive techniques.

[114] The present techniques focus on the most practical and less-explored FS-OCL for low-resource devices and few-shot data in the Cl setup, with domain changes happening at both fine-tuning and testing stages. OCL, FSCL and FSOCL methods generally assume replay data, violating the important assumption that no storage is guaranteed. Therefore, the present techniques rely on the baselines introduced in T. L. Hayes and C. Kanan (“Online Continual Learning for Embedded Devices," CoLLAs-2022,2022). Additionally, the robustness of agents at test time is evaluated.

[115] In our setup, a model learns from streamed examples, seen one at a time with no repetitions, since many applications cannot store samples due to the continuous nature of the data stream and the limited time and storage to process it. Over time, the learner sees new classes but no task identifier is provided, since the agent should infer the class regardless of the specific task information. On resource-constrained devices, CL methods should be both data and computation efficient: they should learn from few-shot labelled data and adhere to strict memory and time constraints. Finally, agents will operate in uncontrolled environments, therefore, CL methods should be robust to (severe) test-time corruptions.

[116] Running in parallel to the steps which lead to training, optionally, at step S40, the method may comprise extracting input-level image statistics from the image and saving and storing the extracted input-level statistics and prototype in memory at step S42. The saved extracted input-level statistics may be used for normalizing the input image during inference-time operation of the model. The normalization of an input image during inference-time advantageously improves the robustness of the model to inference-time variability. Additionally or alternatively, the saved extracted input-level statistics may be used to augment the (original) training set of images. In particular, the training set of images may be augmented partially, to augment a subset of the training set of images, or may be augmented in its entirety. An augmented training set of images may be generated as a result of the augmentation. Thus, the training set of images may be expanded to include the original training set of images and an augmented training set of images.

[117] Figure 5a is a flowchart showing example steps in a method for creating a training set of images. In some cases, the user device on which the ML model is to be used will comprise the necessary functionality to perform the steps of the method. However, it will be appreciated that some user devices on which the ML model is to be used may not have this functionality (e.g. a vacuum cleaner). In such an arrangement, a second user device such as a user’s smartphone may perform some or all of the steps of Figure 5a. At step S500, the method comprises capturing at least one image depicting a new class which is of interest to the user device in which the ML model is to be used. The image may be captured by the primary user device (e.g. the vacuum cleaner) and sent to the second user device (e.g. the phone) or may be captured directly by the phone. At step S502, the method comprises receiving a label for the new class from the user, for example by an input into a user interface, such as a touchscreen in the example of the smartphone above. The user may supply a label to be received by the user device. For example, the user may provide a label “this is my pet Bruno” for their pet dog to the device.

[118] At step S504, the method comprises optionally searching a local set of images for relevant images and determining whether any relevant local images are found at step S506. Local images determined to be relevant may be images with similar characteristics to the captured image(s). For example, a local set of images (e.g. a gallery) on the user device may be searched for images depicting dogs similar to the user’s pet dog. When it is determined that relevant local images have been found, these images may be fetched from the local storage and presented on a user interface to the user so that the user can confirm that the presented images from the determined images do indeed depict the user’s pet dog. In other words, the user may be prompted to identify any of the presented images which show the pet dog of interest at step S508. By using such a search, more samples may be retrieved for the training but this is not compulsory.

[119] Alternatively, when it is determined that no relevant local images are found, the user may be optionally prompted to take more images depicting the new class at step S510, e.g. by selecting images from their gallery. All images, the original image together with the local images identified confirmed as depicting the new class or the new images, are stored on the user device at step S512 together with their associated class label. The stored data is fewshot labelled data.

[120] Optionally at step S50, the method may further comprises storing all images with at least one instance-level label in addition to the classification label described above. An instancelevel label corresponds to an instance-level category within an object category (i.e. classification). In the example of an image depicting a user’s dog, such instance-level category may include the breed of the dog, such as “Labrador”, the name of the dog, such as “Bruno” or the age / stage of life of the dog, such as “1”. In other words, the stored images may be stored with a label corresponding to the object(s) category depicted in an image and also with one or more labels corresponding to the specific instance-level categories of any object(s) depicted in the image. Local images stored with instance-level category labels may be used to identify instance-level categories when identifying a classification for an input image, as described below.

[121] In Figure 5a, the training set focuses on a classification label for a particular object but as set out above, the domain in which the user device operates may also be important. Figure 5b is a flowchart showing steps of a method for creating a training set of images which focuses on domain. As in Figure 5a, the steps of Figure 5b may be carried out on the user device which is to use the ML model or on a secondary user device (e.g. a smartphone) when the user device (e.g. vacuum cleaner) which is to use the ML model does not have the required functionality. At step S520, the method comprises capturing at least one image depicting a new domain and receiving a label for the new domain from the user at step S522. For example, the user may capture an image depicting the user’s kitchen, and the user may label the domain of the image as “my kitchen”. Optionally, at step S52, the method may further comprise prompting or guiding the user to capture additional images of the new domain to generate a set of images of the new domain, for example which cover the space in its entirety. The user may be prompted to capture an image of the domain from e.g. a different angle, position, perspective (merely by way of example, and it will be appreciated that the additional images may be captured in other suitable ways).

[122] The method further comprises identifying objects within the images at step S524. This may be done automatically using any suitable technique. Using an interface, the user may be presented with a set of objects and their proposed labels at step S526. For example, in the example of the new domain being the user’s kitchen described above, the labels may be “my fridge”, “my kettle”, “my toaster” based on typical objects found in a user’s kitchen. These objects and labels may be identified automatically. Optionally the user is able to correct the label on the user interface. Once the classes of the identified object are confirmed by the user, the set of images together with the new domain label and any object (class) labels are stored on the user device at step S528. The stored data is few-shot labelled data.

[123] In each of Figures 5a and 5b, the user initiates the creation of new training data. Figure 5c is a flowchart showing steps of an alternative method for creating a training set of images to the methods shown in Figures 5a and 5b. In Figure 5c, the user device automatically prompts the user to consider creating new training data when the user takes a photo. The method is thus initiated by the user capturing at least one image at step S540. For example, the image may be a mirror-selfie image taken by a user wearing a particular type of t-shirt, such as a t-shirt pertaining to one of their favourite musical groups or bands. At step S542, the user device automatically analyses the images to identify any objects within the images and to compare them to any learnt user’s preferences. In this example, the user may have taken many mirror- selfies with his band t-shirts (for example around a dozen images) and these may have been stored with object-level class corresponding to “t-shirt” and the instancelevel classes corresponding to “band t-shirts”. In other words, a plurality of instance-level categories may be constructed based on the instances of specific types of objects depicted in the images, such as [band t shirty band t shirt2}. Thus, in the analysis, the user device recognises that the user is wearing a band t-shirt in the new image and the new image shows a new instance (i.e. new t-shirt) of the object-level class (band t-shirt). It will be appreciated that the user preference may be identified based on any suitable number of images.

[124] At step S544, the method comprises prompting the user to capture a classification for the object identified in the image. At step S544, the prompt may be an icon on the user interface and the user may choose to select the icon, e.g. by clicking on the icon. When the user selects the icon, at step S546, the user may be presented with a first user interface which shows the object level classification, e.g. shirt and asked to confirm if they wish to store the new image with this label. If the user selects to save the image with the object level classification, the user may be presented with a second user interface to confirm the instance level classification, e.g. band shirt. At step S548, the new image is stored with each confirmed classification. Similarly to the method described in relation to Figure 5a, the stored images may be used to identify instance-level categories when identifying a classification for an input image, as described below. In other words, the label for each image can be in two formats. A first label is a classification which may be obtained by using a pre-trained object detector to get an object category and then confirmed by the user as explained above. The user may then be asked to provide the instance-level category only. Alternatively, we can ask the user to provide both a bounding box around an object and an object category. The stored data is fewshot labelled data.

[125] Figures 5d to 5f are schematic illustrations of a user interface on a second user device at some of the steps in Figure 5a. Figure 5d illustrates step S502 where a user supplies the label “This is my pet Bruno” by an input to a user interface of a smartphone. Steps S504 to S508 are schematically illustrated in Figure 5e where a local set of images has been searched for images depicting dogs similar to the user’s pet dog and where relevant local images have been found and presented on a user interface to the user. The user has been prompted to identify any of the presented images which show the pet dog “Bruno”, illustrated with a “tick” mark. Step S512 and optional step S50 are schematically illustrated in Figure 5f, which shows a user interface to allow a user to input an associated class label and at least one instancelevel label (i.e. “Breed: Labrador” or “Age: 1”) for an input image.

[126] Figure 6a is a schematic diagram of an example ML model 600 in the form of an artificial neural network which can be deployed on a user device for processing images and can be trained by the new Robust Online Continual Learning method (RobOCLe) as described above. The model 600 comprises 5 main blocks: a normalization block 630, a feature extractor 6(-) 602, a pooling scheme or module P( ) 604, a classifier F(-) 606 and an optional calibration module 620. All the modules are novel except for the feature extractor 602 and this is indicated by the bold outline for the blocks. At inference or test time, the model receives an input image 608 and categorises the input image from its extracted features to generate an output prediction (e.g. classification) 610. The normalization module 630 is connected to a database of input-level statistics and the classifier 606 is connected to a database of feature-level statistics.

[127] Figure 6b is a schematic diagram showing an example of how the ML model of Figure 6a can be used. The input image(s) 608 may be stored in a database 612 of unlabelled input data. Each input image 608 may be normalized, for example using an input-level data inverse normalization module 630. Objects and domains change appearance over time, depending on illumination (day / night), occlusion, blur (camera focus / dust), viewpoint, object pose, etc. Users want robust personalized object recognition services that work in variable conditions of the objects (different pose or appearance) and environment (illumination changes, level of clutter, etc.). The ML model needs to identify objects belonging to old classes in new domains, contexts and environments (different rooms, different houses and / or different environmental conditions). As described in more detail below, normalizing an input image may help reduce the domain gap, i.e., to bring the input data into a similar domain as the data which was used to train the ML model.

[128] Input-level image statistics may also be computed from the input data and stored in a running input-level statistics database 614. Examples of statistics include colour histogram, white point, and colour jitter parameters. As described in more detail below, these computed input-level statistics together with the stored statistics are used in the normalization process. The newly computed statistics are added to the storage to update the storage.

[129] Once the input data is in the same domain input as the corresponding training data, the normalized input data is then input to the feature extractor 602. The extracted features will also be in the same domain as the training data. These extracted features are the input as a feature vector to the pooling module 604 which as described above extracts at least one statistical moment from the feature vector and adds the at least one statistical moment to the feature vector to create a concatenated vector of enriched features. These enriched features are input to the classifier 606 which predicts or identifies a classification for each input data item by comparing the concatenated vector to a plurality of prototype models which are stored on the user device in the running feature-level statistics database 616. The statistics in this database may comprise mean and covariance of the enriched features.The classifier 606 identifies the prototype model which is most similar to the input concatenated vector. The most similar data item is assigned the classification of the identified prototype model.

[130] Figure 7a is a flowchart showing example steps of how a machine learning ML model such as that shown in Figures 6a and 6b may be used to process an image. Many of the steps are the same as those used to train the model. At step S700, the method comprises receiving an input image. The input image may be received by capturing an image using an image capture device (e.g. a camera) of the user device. The image may depict a single object to be classified. Alternatively, the image may depict two or more objects, each of which are to be assigned a classification. It will be appreciated that this alternative may also be considered as classifying the image, despite two or more objects each being classified separately.

[131] The input image may optionally be normalised at step S70 before generating the feature vector at step S702. That is, the received image may be normalised and then provided as input to the feature extractor of the ML model. As noted above, the feature extractor may be any suitable feature extractor for computer vision tasks.

[132] The method further comprises extracting, using the pooling module, at least one statistical moment from the feature vector at step S704 and adding, using the pooling module, the at least one statistical moment to the feature vector to create a concatenated vector at step S706. The adding may be done by creating a statistical vector which comprises the statistical moment(s) and concatenating this with the feature vector. The pooling module may create the concatenated vector through a concatenation operation, such as Eq. (1) above. Merely as an example, the at least one statistical moment may be the first three statistical moments and the concatenation operation may concatenate the first three statistical moments.

[133] At step S710, the method further comprises identifying, using the classifier, a classification for the input image. The method further comprises generating an output based on the identified classification at step S712. Generating an output based on the identified classification may comprise controlling the user device based on the identified classification at step S72. The controlling may be based on instructions generated based on the identified classification. The user device may be a robotic device, such as a robotic vacuum cleaner. Merely as an example, the robotic device may receive an instruction to clean close to an object having a specific classification (e.g. my_object) and when the identified classification corresponds to the specific classification, the robotic device may begin cleaning. In this example, instructions may be generated based on my_object being classified (i.e. found by the vacuum cleaner) and the robotic vacuum cleaner may be controlled to clean close to my_object.

[134] As an alternative example, a robotic vacuum cleaner may receive instructions to maintain its distance from the user’s pet, to reduce the stress of the user’s pet. In this case, generating an output comprises controlling the robotic vacuum cleaner to stay away from the user’s pet. A robotic vacuum cleaner may be controlled in any number of similar and nonlimiting ways. For example, the robotic device may be controlled to monitor a user’s pet(s), to clean close to specific objects such as a backpack or to clean close to a specific object which may be delicate and therefore to adapt the method of cleaning based on how delicate the specific object is.

[135] The generating an output may further comprise providing feedback to the user at step S74. In the example of a robotic vacuum cleaner instructed to clean close to a specific object, the feedback may comprise a statement that the vacuum cleaner was unable to navigate to and find the specific object. In another example where a vacuum cleaner is instructed to monitor a user’s pet, another example of providing feedback to the user is showing or sending a picture of the pet to the user once the pet is found. Other examples of providing feedback to the user are sending an image(s) of a surface before and / or after cleaning, sending the location (e.g. a “pin” location or GPS coordinates) of the pet on a map once found or sending the location of a cleaned surface on a map to the user.

[136] Figure 7b is a flowchart showing example steps of how an input image may be normalised or standardized. Normalizing or standardizing the input image may comprise obtaining image statistics for the input image at step S720. The normalizing further comprises comparing the obtained image statistics with stored image statistics at step S722 and adjusting the input image based on any differences between the obtained statistics and the stored statistics at step S724. The stored statistics may be updated with the obtained statistics to provide “running” input-level image statistics at step S726. The stored statistics and the obtained statistics may be any suitable statistics of an input image, for example a colour histogram, white point, colour jitter parameters and so on. Normalisation or standardisation is selected based on the type of statistic. The two terms are generally used interchangeably but they are different. However, they refer to the same kind of operation: reporting the input to a common ground so it is easier to process because the ground is always the same across consecutive images.

[137] The stored statistics may be historical data and may be obtained from some or all previous input images which have been received, including any input images which were used to generate the prototype models and / or train the ML model. The stored data is only the statistics and not the full images (neither the input image or previous images), thus is a data efficient and low-footprint method. The efficiency may be with respect to storage and / or memory requirements as well as training and / or inference speed.

[138] The technique of Figure 7b may be termed Test-Time Training (TTT) and is a special case of Unsupervised Domain Adaptation (UDA) where a model trained on a source domain is adapted to an unlabelled target domain without accessing source data. Many approaches design novel loss functions to train the model and reduce domain shift, requiring access to the full target set.

[139] Test-Time (i.e. inference-time) Performance is heavily influenced by the domain gap with respect to the training set. Test-Time Augmentation (TTA) approaches use data augmentation at test time to obtain greater robustness, improved accuracy or estimates of uncertainty. TTA pools predictions from several transformed versions of a given test input to obtain a smoothed prediction. In particular, current techniques obtain the predictions over augmented views (e.g., cropped, rotated, etc.) of input samples and calculated an average. Current techniques also average embeddings obtained over augmented views of the input samples or propose a learnable aggregation of TTAs. Additionally, a selection policy for TTAs may be learned or images may be enhanced to make them recognition-friendly.

[140] Recently, current techniques also perform TTT on single images coupling TTA with TTT to improve test-time robustness. Continual TTT has also been recently considered. The present techniques are linked to current techniques but with some key differences: • The test set is not assumed to be available all at once, as in practical applications where an agent encounters new samples and domains over time. • Difference in scope: TTA boosts model performance at test time aggregating multiple predictions and TTT updated the model at test time. The aim of the present techniques is to propose an OCL method with increased robustness over variable test sets. • Multiple TTAs of the same input sample are not computed, but rather a deployment scenario is mimicked where robots experience variable domains with no clear delimitation. • Rich information is extracted from inputs, to reduce the need for TTA / TTT and maintain same inference time (one inference step per sample is used, in contrast to TTA / TTT).

[141] Figure 7c is a flowchart showing example steps of how a classification may be identified using a classifierof a ML model. The classification is identified by comparing the concatenated vector to a plurality of prototype vectors (or models - the terms may be used interchangeably) which are stored on the user device, wherein each prototype model is a statistical representation of a different classification in a first step S740.

[142] At step S742, the identification comprises identifying the prototype model which is most similar to the concatenated vector. The prototype model which is most similar to the concatenated vector may be determined based on any suitable metric. For example, the metric may be a distance (i.e. distance metric) in feature space between the concatenated vector and each prototype model in the plurality of prototype models. Merely as an example, the distance in feature space may be the l2 (or, equivalently, the Z2-norm) distance. The l2 distance is also commonly referred to as a Euclidean distance between vectors in a vector space (the feature space and concatenated vectors described here are examples of such spaces and vectors).

[143] At step S744, the method comprises assigning the classification of the identified prototype model to the input image. Thus, the classifier advantageously, identifies a classification for the input image based on a comparison between a concatenated vector corresponding to enriched features of the input image and a plurality of prototype models. This provides a robust and efficient way of classifying input images which depict new, unseen classifications.

[144] As noted above, users are also demanding instance-level personalized detection of vision systems mounted on such devices. For example, vacuum cleaners can now monitor the behavior of users’ specific pets, and stay away from those specific pets that are mostly scared by the robot’s noise. Nonetheless, users do not provide many labeled examples, being a time-consuming operation. Therefore, a new task of few-shot instance-level personalization of object detection models to detect and recognize personal instances of objects (e.g , dog i and dog 2 rather than just dog) is introduced. The limited availability of the data distinguishes our task from curret techniques, which assume large availability of labelled data and finetune (FT) the models through computationally expensive updates. However, FT-based methods inevitably fail when few-shot samples are provided.

[145] As noted above, there is a desire to identify instance-level categories for an input image. For example, an object category may be {ball}. The object category may be divided or classified further into instance-level categories {ball^ ball2}. For example ballr may be a specific ball preferred by e.g. the child of a user and ball2 may be a specific ball preferred by the user’s dog. Moreover, an input image may depict instance-level categories across different domains (e.g. different image backgrounds). That is, as a user device operates over time, there may be a sequence of images for each of the ballr and ball2 instance-level categories, wherein each image in the sequence depicts the instance-level category in a different domain. For example, a first image in a sequence may depict ballr with a concrete wall background and a third image in the sequence may depict ball2 with a grass lawn as a background.

[146] Few-Shot Personalized Instance Recognition. The present techniques are aimed at personalizing generic object detection models to recognize objects by a set of instance-level labels. Figure 8 shows how the ML model of Figure 2a may be adapted for training to provide instance-level categorisation. Figure 8 is a schematic diagram of an example ML model 800 for processing images for instance-level classification. Similar to the model shown in Figure 2a, the model 800 comprises a feature extractor 802, a pooling module 804 (which may also be termed a reduction module and a classifier 806. The feature extractor 802 may be termed an object detection backbone or encoder and the terms may be used interchangeably. The model 800 further comprises an object detector 802 which may also be termed a detection head. The object detector may detect at least one object in the input image and may determine a location within the input image for the at least one detected object. Together the feature extractor and object detector may be considered to form an object detection model.

[147] As an example a generic object detection model Mo ( e.g , YOLOv8 - described in “YOLOv8 by ultralytics,” https: / / github.com / ultralytics / , Accessed: 2023-08-20) is provided. Without loss of generality, it is assumed that Mo = DO°EO can be decomposed into an encoder Eo (also termed a feature extractor 802) and a detection head Do (also termed an object detector 830). The detection head Do detects an object within an image and defines a bounding box for the detected image. In this example, the generic object detection model Mo has been trained on a labelled object-level dataset To = {xOik,yOik}k°=1, whose labels yok e Co belong to an object-level class set Co {e.g , Co = {ball, bottle}). The present techniques target personalization of the classification ability of Mo to recognize a set of personal classes given few-shot labelled samples Tt = {xt^yt^Ey where No » Nt and labels yik e Ct belong to an instance-level class set Ct ( e.g , Ct = {ball^bal^bottle!}).

[148] The present techniques focus on the detector’s classification part (i.e. the classifier 806) and the localization part is not updated. In other words, it is assumed that there exists an instance-to-objectfunction f (•) mapping each class c e to a label in Co, i.e, f: -> Co (e.g, f (ball-J = ball), therefore there is no need to update the detector’s regression part. The classifier 806 comprises two modules: an object module 840 which is the part of the classifier which identifies the object-level classification and an instance module 842 which identifies the instance-level classification. As explained below, during training, the parameters of the object module 840 are frozen and only the parameters of the instance module 842 are updated.

[149] The model also comprises a down-sampling module 832 which reduces the dimensions of the feature vector to correspond to the dimensions of the bounding box for the identified object. There is also a mask module 834 which discards regions in the input image which are outside the bounding box.

[150] Figure 8 is a schematic diagram illustrating how a generic object detector (e.g, YOLOv8) is adapted to detect personal instances via the present techniques. A lightweight module that can be integrated into any object detection network is proposed. The present techniques are based on three key components: (i) an object detection network (i.e. the feature extractor 802 and object detector 830), with (ii) a multi-order statistical augmentation of embeddings (e.g. the pooling module 804) for (iii) instance-level recognition via an OBol (part of the classifier 806). The model in Figure 8 may be termed an Object Conditioned Bag of Instance (OBol) to personalize object detectors pre-trained on To on the server side.

[151] Figure 9 shows the steps of the method for training an object detection network using the OBol techniques. The model Mo (e.g. YOLOv8) may be trained on any appropriate training dataset, such as MSCOCO (Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'ar, and C Lawrence Zitnick, “Microsoft COCO: Common objectsin context,” in ECCV. Springer, 2014, pp. 740-755). To is made of subsets of the generic Open-lmages-V7 (OIV7) dataset. To comprises samples from OIV7 of the same object-level classes as in the personal dataset 7); i.e , such that f (•) exists. The model Mo is then adapted to recognize personal user-specific object instances.

[152] At step S800, the method comprises receiving a training set of images which are labelled with both an object-level classification and an instance-level classification. An example input image is shown in Figure 10a. Alternatively, only the instance-level classification may be provided and this instance-level classification may be used to determine the object-level classification. In other words, the training set of images may be a plurality of images with labels provided by the user, for example captured using the techniques described in relation to Figures 5a to 5c. Given a sample from Th (xik,yik) e Th where xi:k e nV7*14'*3 is an RGB input image with size H x W and yik the associated personal instance label, xik is passed through the model (i.e. through the feature extractor and object detector) to obtain the objectlevel predicted label {y0 k) and a bounding box around the identified object at step S842. An example of a bounding box is shown on Figure 10b. The coordinates H x W of the bounding box are determined. For the sake of simplicity, it is assumed that only one object is present in each image, but the same rationale applies in presence of multiple objects seamlessly

[153] At the same time as the bounding box is determined, a D -dimensional feature vector eik = E0(xi k) e nV7'*14"*7’ is determined at step S902. An example of such a feature vector is shown in Figure 10c. The predicted coordinates are rescaled by H / H' = W / W' to match the low-resolution spatial dimensions H' x W' of the feature vector. The rescaling may be done in the downsampling module. A binary mask Si k is then built at step S906 to discard regions outside the bounding box, for example using the mask module. An example of such a mask is shown in Figure 10d. It is applied at step S908 via the Hadamard product obtaining a feature vector which may be termed a masked vector e'i k = Si k O ei k.

[154] This masked feature vector is then passed through a reduction operation. In order to characterize the instance-level distribution from the few input samples, the first R statistical moments are extracted from the masked feature vector at step S910 and concatenated to form vik = concatCmi,..., at step S912. These steps are the same as described above with concat(-) being the concatenation operation and mn the n-th order central statistical moment. The difference is that the concatenated vector is computed over the features in e'ik corresponding to non-zero entries in Si k.

[155] Finally, the concatenated vector vik together with the object-level classification from step S940 and the original label for the instance-level classification is used as the input to training the classifier at step S914, for example using prototype based few shot learning (PFSL). PFSLs are used to identify instance-level classes on a metric space spanned by multi-order statistics. The prototypes may be calculated as described above. The search for representations of personal objects is conditioned only within the instances whose object category matches yok, thus simplifying the search of the correct nearest instance-level prototype.

[156] Prototypes-based Few-Shot Learners (PFSLs), such as those described in Jake Snell, Kevin Swersky, and Richard Zemel, “Prototypical networks for few-shot learning,” NeurlPS, 2017 and an Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens Van Der Maaten, “Simpleshot: Revisiting nearest-neighbor classification for few-shot learning”, arXiv: 1911.04623, 2019. In short, PFSLs learn a metric space in which classification is performed by computing distances to prototypical representations of each class.

[157] In this context, PFSLs are extended to support object-class conditioned search, and these approaches are termed Object-conditioned Bag of Instances (OBol), since they contain instance-level prototypes. The present techniques enriche any OBol method by augmenting localized encoder embeddings (EEs) of the input object via multi-order statistics to construct a richer metric space, where instance-specific patterns are separable. Augmented EEs (AEEs) are computed via a reduction module similar to current techniques for pooling schemes to characterize the distribution of the specific instances from the few-shot labelled data. Current techniques also apply ensemble learning on multi-order features learned separately; however, their focus is neither personalized instance recognition nor object detection, and they require gradient-based training. A backpropagation-free approach, instead, could be especially useful where dynamic compilers are not available for the target hardware. The OBols described herein with AEEs significantly increase model personalization, alleviating neural collapse, i.e, a state at which within-class variability of hidden layer outputs is completely lost due to the object-level optimization objective. The contributions of the present techniques can be summarised as: • A novel task of few-shot personalization of object detectors to recognize instances of objects is proposed; • PFSLs are extended via object-level conditioning (OBols); • A multi-order feature space is designed where personal instances can be separated via a backpropagation-free metric learning on few-shot labelled user data only; and

[158] Thus, an Object-conditioned Bag of Instances (OBol) is constructed based on multiorder statistics of extracted features, where generic object detection models are extended to search and identify personal instances from the OBol’s metric space, without need for backpropagation. The overall pipeline can be thought of as an OBol since generic categorylevel output is converted to specific personal-level output via conditional nearest prototype selection. Thus, the present techniques may be considered a backpropagation-free Object-conditioned Bag of Instances (OBol) approach with augmented embeddings by multi-order statistics.

[159] The present techniques are fully compatible with the key requirement of continually learning new instances overtime; whenever a user presents a new instance to be recognized, new instance-level prototypes can be included in the OBol at any time with no accuracy degradation with respect to the case where all instances are available from the beginning of the adaptation process.

[160] Figure 11 is a flowchart showing examples steps in a method for identifying an instancelevel category. Optionally, at step S70, the received image may be normalised as described above to reduce the domain gap. In a first step, an unlabelled input image is received at step S1110. A feature vector is generated at step S1102, for example using the steps of downscaling and masking as described above. Generating the feature vector may use the determined location of an object(s) within the input image. The location may be defined by bounding box coordinates. The method may further comprise masking regions within the image outside the location of the at least one detected object, e.g. outside the bounding box coordinates. The method may further comprise rescaling coordinates of the location to match a dimension of the feature vector.

[161] Statistical moments are extracted at step S1104 and added to a vector (which may be termed a statistical vector) and a concatenated vector is created for the input image at step S1106. Extracting and concatenating may be done using the pooling module (which may also be termed a reduction module). The extracting of the at least one statistical moment may be over the features within a portion of the image at the determined location, e.g. within the bounding box where used.

[162] In parallel, or before or after, the creation of the concatenated vector, using an object detector, an object category (class) for at least one detected object in the input image is determined at step S1120 and the prototype models which have the determined object category are selected from the stored plurality of prototype models at step S1122. At step S1108, the concatenated vector for the input image is compared only to each selected prototype model to identify the prototype model which is most similar to the concatenated vector. The instance-level category (sub-class) of the identified prototype model is then assigned to the input image (step S1110). In other words, the search space of prototype models has been simplified to only within the instances whose object category matches the object category assigned to the input image. More than one object may be detected in the image and thus multiple object categories may be assigned to the input image. Such a pipeline may be thought of as an Object-conditioned Bag of Instances since generic category level output is converted to specific personal-level output via conditional nearest prototype selection.

[163] The classifier may also be termed a learner module and may be a prototype based fewshot learner (PFSL). As explained above, the learner module and / or the search space over which it searches may be termed an object-conditioned bag of instances (OBol). When personalising the training, the object conditioned element may be frozen and the bag of instances may be trainable. The search space comprises only a few-shot labelled user data. Using the techniques described above, the generic category level output (e.g. the initial class label or object category) is converted to a specific personal-level output (e.g. the updated class label or instance-level category) via nearest prototype selection.

[164] Figure 12 shows a block diagram of a system 300 for performing the methods described above. The system 300 comprises a plurality of primary user devices 320 and for ease of reference, a single primary user device 320 is shown. Each primary user device 320 may be any electronic device which utilizes computer vision. For example, each primary user device may be a smart vacuum cleaner, smart domestic appliance (such as a smart oven), smartphone, tablet or other robotic device. However, it will be appreciated that the primary user device(s) may be any suitable electronic device. The primary user device 320 comprises the standard components of such devices, including for example as shown one or more processors 322, memory 324, storage 326, an image capture device 332, and a communication module 334. The primary user device 320 may optionally comprise a display 336 for displaying information to the user and a user interface 338 such as a keyboard, voice recognition input device, touch sensitive screen or any similar component for receiving user input. There may be other standard components which are omitted from the Figure for ease of reference. The display 336 may comprise any suitable display screen, e.g. LCD, LED which may also be touch sensitive to allow user input.

[165] The communication module 334 may communicate using any suitable communication, e.g. wireless communication, hypertext transfer protocol (HTTP), message queuing telemetry transport (MQTT), a wireless mobile telecommunication protocol, radio frequency communication (RFID), near field communication (NFC), ZigBee™, Thread™, Bluetooth™, Bluetooth™ LE, IPv6 over Low Power Wireless Standard (6L0WPAN), Constrained Application Protocol (CoAP) or a wired communication. The memory 324 may be any suitable form of memory, including volatile memory, such as random access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example.

[166] The image capture device 332 may be any suitable device, e.g. camera for taking input images that may then be processed as described above. The local storage 326 also stores at least one personalised ML model 330 (also termed an artificial intelligence (Al) model) which have been trained to produce an output. The personalised ML model may be at least one of the models 100, 200 described above, or any combination of the models 100, 200.

[167] The storage 326 also locally stores a local dataset 328 comprising a plurality of prototype models which may be used to update (train) the classifier, as described above. The local dataset may further comprise a training set of images which are labelled with a classification for locally training the ML model to personalise the ML model. It will be appreciated that, given the local dataset 328 is stored locally on the user device, the local dataset may remain on the user device. Thus, the data remains private and there is a reduced risk of data breach or data theft. The personalisation can also then be targeted to users who do not typically share any data.

[168] Using the communication module 334, the user device 320 may communicate with a server 302. The server 302 may also comprise standard components, including for example as shown one or more processors 304, memory 306 and a communication module 308. There may be other standard components which are omitted from the Figure for ease of reference. The server 302 may also comprise a pre-trained ML model 308. The pre-trained ML model 308 may be received by the primary user device 320 for local training, as described above. The server further comprises storage 310 for storing a global training dataset 312. The global training dataset 312 may comprise a global training set of images with comprising a set of global classes 314 and a set of global domains 316. As described above, the global classes and domains of the global training dataset 312 are typically different to those encountered by the primary user device 320 during operation of the device.

[169] The system 300 may further comprises a secondary user device 340. Similarly to the primary user device 320, the secondary user device 340 may comprise standard components, including for example one or more processors 342, memory 344 and a display 350. The description of these standard components in relation to the primary user device also applies to the equivalent features in the secondary user device 340, and so their description is not repeated for conciseness.

[170] The secondary user device 340 further comprises storage 346 which may be used to store a personalised ML model and a local dataset (not shown). The secondary user device may communicate with the primary user device 320 and the server 302 via the communication module 334. In this arrangement, the communication modules 334, 306 of the primary user device and the server, respectively, may be further used to communicate with any of the server, primary user device or secondary user device (where appropriate).

[171] In particular, the secondary user device 340 may communicate with the primary user device 320 to provide a training set of images to the primary user device. The training set of images may be obtained through any suitable method, such as those described in relation to Figures 5a to 5c. In this arrangement, the training set of images may be received by the primary user device 320 and may be used to locally train or update the ML model 330.

[172] In an arrangement where the secondary user device 340 communicates with the server, the secondary user device may similarly receive the pre-trained model. The pre-trained model received by the secondary user device may be a compressed version of the pre-trained model. The processing power of the processor 342 of the secondary user device 340 may be significantly greater than the processor 322 of the primary user device 320. In this arrangement, the secondary user device may be a smartphone or tablet, and the primary user device may be a robotic vacuum cleaner, which are typically resource-constrained compared to smartphones and tablets. The secondary user device may create a fully trainable version of the pre-trained model received from the server. This may be done by, for example, dequantizing a compressed version of the pre-trained model and determining whether re-training of the pre-trained model is possible. This may be possible since modern smartphones and tablets may have sufficient processing power to fully re-train a trainable version of the model received from the server.

[173] When it is determined that it is possible to re-train the model, the secondary user device may train the received model using a local dataset provided by the user. The characteristics of the local dataset used by the secondary user device in this arrangement are equivalent to those of the local dataset 328 of the primary user device, and their description is not repeated for conciseness. Once trained, the received model may be compressed by the secondary user device and received by the primary user device. For example, a smartphone or tablet may re-train the model received from the server and send the re-trained version using the local dataset provided by the user to a robotic vacuum cleaner for enhanced personalisation.

[174] Figure 13a shows an example user device for performing the methods described above. As mentioned above, the user device may be a robotic device, such as a robotic vacuum cleaner. Typically, a vacuum cleaner encounters different objects and instances of objects as the vacuum cleaner navigates a user’s home using a computer vision system housed within or mounted on the vacuum cleaner comprising an image capture device and a ML model as described in relation to the methods above. That is, the images captured by the computer vision system depict object classes and domains which differ from those which the ML model would have been trained on. It will be appreciated that the robotic vacuum cleaner is just one example and other devices of interest are smart ovens and smart fridges. In both of these examples, there is a need to update the model to reflect the user’s preferences, e.g. class-incremental food detection when users enter new items in the fridge or class-incremental food / dishes being cooked in the oven.

[175] As an illustrative example, the vacuum cleaner may receive a voiced instruction from a user, such as “Clean close to my coffee table.” The present techniques advantageously enable a way for the robotic vacuum cleaner to identify the user’s personal coffee table and to control the robotic vacuum cleaner based on the identification of the user’s coffee table i.e. by instructing the vacuum cleaner to clean around the user’s coffee table. It will be appreciated that the example of a user’s coffee table is provided by way of example of a personal object / class or instance-level class, and that other personal objects may be identified by a vacuum cleaner.

[176] Figure 13b is a flowchart for training a vacuum cleaner recognition model (VCRM) and deploying on multiple vacuum cleaners which uses the prior art techniques rather than the techniques proposed above. The VCRM is trained at step S1300 on a server using a large dataset for indoor object detection. The dataset comprising classes C and with data from domain D. At step S1302, the VCRM is typically compressed so that it can be deployed to the vacuum cleaners (VC) which typically have lower processing capability. Two such deployments are illustrated with a first user (User_A) buying a first vacuum cleaner (VC_A) and a second user (User_B) bying a second vacuum cleaner (VC_B). Both the vacuum cleaners have been trained as described above.

[177] The first user is using the first vacuum cleaner in their home and thus as illustrated at step S1304, the first user is using the first vacuum cleaner on data from a first domain (D_A). Similarly, the second user is using the second vacuum cleaner in their home and thus as illustrated at step S1314, the second user is using the second vacuum cleaner on data from a second domain (D_B). The compressed VCRM which is stored on the first and second vacuum cleaners will recognise objects in the first and second domains (D_A and D_B) with lower accuracy that the object recognition which was achieved on the training data for the training domain D.

[178] At step S1306, the first user would like the first vacuum cleaner to recognise new classes C_A for example a class labelled backpack as well as classes for a first cat Cat_A and a second cat Cat_B. Similarly, the second user would like the second vacuum cleaner to recognise new classes C_B, for example classes for two different dogs, e.g. Dog_A and Dog_B. Unlike with the methods proposed above, there is no update capability for the compressed VCRM once it is deployed to the vacuum cleaners. Thus, the first vacuum cleaner cannot be updated to recognise the new class C_A and similarly, the second vacuum cleaner cannot be updated to recognise the new class C_B.

[179] Figure 13c is a flowchart showing how the techniques described above ensure that the VCRM deployed on the vacuum cleaner is adapted for the user’s domain and any new classes. As in Figure 13b, the newly proposed VCRM is trained at step S1310 on a server using a large dataset for indoor object detection. The dataset comprising classes C and with data from domain D. At step S1312, the VCRM is typically compressed so that it can be deployed to the vacuum cleaners (VC) which typically have lower processing capability.

[180] The first user is using the first vacuum cleaner in their home and thus as illustrated at step S1314, the first user is using the first vacuum cleaner on data from a first domain (D_A). As shown at step S1316, the VCRM on the first vacuum cleaner automatically adapts to the data from the first domain to reduce the domain gap between the original domain D and the first domain D_A. In other words, the first user does not need to take any further actions because labels are not needed to close the domain gap. As explained above, there are various techniques which are used to reduce the domain gap, including input-level data normalization and denormalization, output level model calibration, model tuning (e.g. minimizing entropy) and the use of feature-1 eve I enriched space via high order statistics. Some of these techniques are employed at training (e.g. model tuning and calibration), some are employed at inference (e.g. input-level data normalization) and other may be employed during both training and inference (e.g. high order statistics). The automatic adaptation means that the first vacuum cleaner has good accuracy on the first domain to detect classes C.

[181] At step S1324, the first user would like the first vacuum cleaner to recognise new classes, backpack, Cat_A and Cat_B. The VCRM on the first vacuum cleaner is updated, as described above, and outlined in the following Figures, to recognise the new classes. The updated model on the first vacuum cleaner now has good accuracy on the union of classes C and C_A on the first domain D_A. At step S1328, the user can now personalise the services provided by the first vacuum cleaner thanks to the updated VCRM. For example, the user can ask the first vacuum cleaner to stay away from the first cat with label Cat_A when cleaning to reduce the pet’s stress. The user can ask the first vacuum cleaner to monitor both the first and second cats (Cat_A and Cat_B) when the first user is not at home. The first vacuum cleaner can be instructed to clean in front of the specific objects defined by the new classes, e.g. backpack. The user can also specify that an object is delicate and the first vacuum cleaner should stay away from it so that cleaning can be achieved without harming any objects. As a final example, the first vacuum cleaner can be instructed to carry out specific types of cleaning when debris with a specific class is detected.

[182] Figure 13d outlines how the steps of updating the VCRM on the first vacuum cleaner may be carried out using either the vacuum cleaner itself or a connected second user device such as a smart phone. In a first step S1330, the vacuum cleaner receives a command from the user showing the vacuum cleaner where to acquire and take some photos. At step S1332, the vacuum cleaner acquires some a small dataset of photos P_A which will be objects taken in the first domain D_A. A vacuum cleaner does not typically have a user interface and thus the photos are sent to a second user device at step S1334 and are downloaded by the second user device at step S1336. It will be appreciated that the photos could also be captured directly by the second user device. The photos on the second user device are then labelled at step S1338 using the user interface of the second user device. This creates a small labelled dataset P_A for the new classes C_A which are in the first domain D_A. This labelled dataset can be used to train (i.e. update) the VCRM using the vacuum cleaner, when this has sufficient processing power to do so, or using the phone.

[183] At step S1340, there is a determination as to whether the vacuum cleaner or the phone is to update the model. This determination is shown as a decision for the phone but it will be appreciated that it can be done on either the phone or the vacuum cleaner. When the VCRM is to be updated on the vacuum cleaner, the labels are sent from the phone to the vacuum cleaner and received at the vacuum cleaner at step S1342. The model can then be updated on the vacuum cleaner at step S1344. When the model is to be updated on the phone, the model is updated at step S1346 and then sent to the vacuum cleaner at step S1348. In both cases, the updated model is used to replace the previously stored VCRM on the vacuum cleaner at step S1350. The first vacuum cleaner now has good accuracy to detect the new classes C_A on the first domain D_A and maintains good accuracy to detect the original classes C on the original domain D.

[184] Figure 13e shows an example method for updating the VCRM on the vacuum cleaner itself. In a first step S1350, there is a determination as to whether there are new images to use fortraining purposes. If there are no such images (i.e. no such dataset P_A), the method keeps looping until there are new images. When there are new images, there is an optional step S1352 to request the labels from the phone. This is necessary when the labels have not been created and stored on the vacuum cleaner with the new dataset. This may also be necessary when the vacuum cleaner is not able to interact with a user to obtain the labels. At step S1354 the phone sends the labels to the vacuum cleaner and these are received at step S1356.

[185] There is a determination at step S1358 to see if the vacuum cleaner is idle, i.e not in use. If the vacuum cleaner is being used, the method loops back to the determination step to await the vacuum cleaner being idle. When the vacuum cleaner is idle, there is an optional determination at step S1360 as to whether the full VCRM can be trained on the vacuum cleaner. Typically, the vacuum cleaner is not able to train the full VCRM. At step S1362, most of the parameters of the VCRM is frozen, for example the parameters of the backbone of the feature extractor and / or object detector are frozen as well as the part of the model relates to classify objects when using OBol. At step S1364, the high-order feature level statistics are extracted from the images in the training dataset to create the concatenated vectors as described above. The parts of the VCRM which are not frozen, particularly the whole or part of the classifier model, are then updated at step S1366. Any suitable known technique for updating the model may be used, e.g. minimising loss functions. The updated full model (i.e. updated part together with frozen part) is used to replace the previously stored VCRM on the vacuum cleaner at step S1368. The vacuum cleaner can then use the new model straight away.

[186] Figure 13f shows an example method for updating the VCRM on the phone. As in Figure 13d, the method is triggered by a first step S1370 which is a a determination as to whether there are new images to use for training purposes. In this example, this determination is done on the phone. If there are no such images (i.e. no such dataset P_A), the method keeps looping until there are new images. When there are new images, there is a determination at step S1372 to see if the vacuum cleaner is idle. If the vacuum cleaner is being used, the method loops back to the determination step to await the vacuum cleaner being idle. When the vacuum cleaner is idle, there is a determination at step S1374 to see if the phone is idle (or at least will have enough bandwidth to run the update). If not, there is a loop back until the phone is idle.

[187] Once the phone is ready to run the update, the phone requests at step S1376 the latest model (VCRM) from the vacuum cleaner and optionally the new data, e.g. P_A (where this has been captured by the vacuum cleaner and not yet transferred to the phone). The vacuum cleaner sends the requested information and this is received at the phone at step S1378. It will be appreciated that if the phone captures the new data, there is no need for these steps.

[188] There is a determination at step S1382 as to whether the full VCRM is to be trained on the phone. As explained above, the proposed methods provide accurate results when only part of the model is updated. However, the phone may have sufficient power and bandwidth to update the entire model and thus there is a determination as to whether this is to be done. If so, the whole model is updated at step S1384. Any suitable technique may be used, e.g. gradients may be computed so that the whole network can be updated.

[189] Typically, the phone will not train the full VCRM. Thus, similar to the steps in Figure 13d, at step S1386, most of the parameters of the VCRM is frozen. At step S1388, the high-order feature level statistics are extracted from the images in the training dataset to create the concatenated vectors as described above. The parts of the VCRM which are not frozen, particularly the whole or part of the classifier model, are then updated at step S1390.

[190] There is then an optional step at S1392 of compressing the updated model (whether the updated full model or the full model with the updated part together with frozen part) and then using this at step S1394 to replace the previously stored VCRM on the vacuum cleaner. The vacuum cleaner can then use the new model straight away.

[191] A series of experiments for validating the methods described above will now be described. In particular, experiments for validating the methods described in relation to Figure 7a for processing images using the general model as trained in Figure 4 together with experiments for validating the (OBol) method described in relation to Figure 9 for processing images and generating an output based on an instance-level classification.

[192] Firstly, results of experiments for validating the method described in relation to Figure 7a for processing images and generating an output based on a classification will be described by way of reference to Figures 14a to 17.

[193] OCL Baselines. For comparison with the method described above for updating the classifier F(-) in Figure 2a, we use several OCL methods which are used to update a classifer, mostly focusing on data-free approaches. • Fine-Tuning (FT) updates a fully-connected output layer using SGD and CE loss with no mechanisms to prevent forgetting. • Online Perceptron (PRCPT) keeps one weight vector for each class, initialized to the first sample of the class. After that, when the model misclassifies a new sample, the class weight vector and the weight vector of the misclassified class with the highest score are updated. During inference, a label is assigned by taking the arg max over the dot product between weights and input vector. • Streaming One-vs-Rest (SOvR) measures how close a new input is to a class mean vector while also considering its distance to examples from other classes (T. L. Hayes and C. Kanan), which is reminiscent of SVMs. FT, PRCPT and SOvR are described in “Online Continual Learning for Embedded Devices” by T. L. Hayes and C. Kanan published in CoLLAs-2022. • Streaming Gaussian Naive Bayes (SNB) estimates a running variance vector per class (Welford, “Note on a method for calculating corrected sums of squares and products," Technometrics, vol. 4, no. 3, pp. 419-420, 1962.) (i.e , diagonal covariance matrices assuming independent features). It requires significantly less memory than SODA. • Online Centroid-Based Concept Learning (CBCL) extends NCM with multiple class prototypes (A. Ayub and A. R. Wagner, “Cognitively-inspired model for incremental learning using a few examples," in CVPRW, 2020, pp. 222-223). At inference, CBCL searches for the weighted nearest neighbor, where class weights are inversely proportional to the number of samples seen so far for that class. The default parameters suggested in F-SIOL-310: “A Robotic Dataset and Benchmark for Few Shot Incremental Object Learning,” in ICRA. IEEE, 2021. are used, i.e , a distance threshold of 17, a nearest neighbor value of 1, and a maximum buffer size of 44 prototypes. No clear difference compared to NCM was noticed. • SQDA extends Streaminng Linear Discriminant Analysis (SLDA) by estimating one covariance matrix for each class ( / .e , Quadratic). The drawbacks are increased memory consumption and need for many samples per class to estimate reliable covariance matrices. SQDA is described in “Online linear and quadratic discriminant analysis with adaptive forgetting for streaming classification” by Anagnostopoulos et al published in SADM in 201. • Online iCaRL maintains a class-balanced memory buffer (T. L. Hayes and C. Kanan, “Lifelong...”) and randomly replaces an example from the most represented class with a new sample when the buffer is full. During training, it randomly draws examples from the buffer, combines them with the new sample and makes a single update. Similar to other methods, linear output layer is trained. While effective, it requires storing sensitive samples on devices. A buffer size of 1K samples (iCarl) or of 2 samples per class (iCarl-2pc) is used.

[194] Datasets. The present techniques are analysed on F-SIOL-310 (cited above) and OpenLORIS(-Objects) (Q. She, F. Feng, X. Hao, Q. Yang, C. Lan, V. Lomonaco, X. Shi, Z. Wang, Y. Guo, Y. Zhang, F. Qiao, and R. H. M. Chan, “OpenLORIS-Object: A robotic vision dataset and benchmark for lifelong deep learning,” in ICRA. IEEE, 2020, pp. 4767-4773), which are robot-acquired datasets specifically designed for CL and FS, and their train and test sets have different properties. Merely as examples, training and test data from each of the three datasets may be obtained and may comprise images of toys, forks and so on. The table below summarizes example differences between train and test sets. We consider a class-IID setup, where all samples are ordered by class and shuffled within each class. Database Different objects in test data Different conditions in test data (real - no augmentation) Different conditions in training data (after mix augmentation) Few shot data OpenLORIS No Partial Yes No OpenLORIS (small) Yes Partial Yes Partial F-SIOL-310 Yes No Yes Yes

[195] OpenLORIS collects 121 instances including 40 categories of daily necessities objects under 20 scenes from variable camera-object distance / angle. The dataset comprises four environmental factors: illumination variability during recording, occlusion percentage of the objects, object pixel size in each frame (i.e , scale of objects), and clutter of the scene. Each factor has three levels of intensity, yielding around 550,000 total RGB samples. Two splits are considered: instance and instance-small.

[196] The instance split (referred to as OpenLORIS) presents videos ( / .e , image frames from the same video) of shuffled object instances to learners one at a time. For example, a learner experiences a video of toy#1, then a video of hat#3, then a video of toy#2, and so on. The learner experiences temporally-correlated data with repeating classes. This split uses the original train-val division provided by the authors.

[197] The instance-small split (OpenLORIS-small) evaluates low-shot out-of-domain generalization ability of models. When learning a new class, frames from a single video of an object instance are shown to the learner. When testing, the model is evaluated on all test data containing the classes seen so far. Test data contains same object instance seen under different domains ( / .e , factors) as well as other object instances from the same classes seen under different domains. Challenges of this setup are: low-shot labelled learning from correlated data, generalization to different object instances of same class, generalization to unseen domains. This setup is often encountered in practice. For example, given a new pet, users provide a few labels and a classifier should recognize the pet in different domains and conditions.

[198] F-SIOL-310 is specifically designed for Few-Shot Incremental Object Learning. This dataset is employed to further evaluate the ability of models to generalize from extremely small labelled data. It contains 310 instances of 22 household categories with no object instances overlapping between train and test sets. Two splits are considerd, respectively with 5 and 10 shots ( / .e , 5 or 10 labelled training samples).

[199] Objects, Conditions &Augmentations. The considered datasets span a wide range of setups with variable factors. OpenLORIS uses the whole dataset and the same objects are shown in train and test sets. Images are acquired under some different conditions (illumination, occlusion, scale, clutter). However, co-existence of conditions is limited. OpenLORIS-small restricts the set of samples and has only one instance of objects in training. F-SIOL-310 uses 5 or 10 training samples, and has different objects between train and test sets. However, objects are acquired with similar conditions (same background, viewpoint / scale, no occlusion, no clutter).

[200] To cope with the limited variability of existing datasets, we introduce some augmentations at either train or test splits. These synthetic data augmentations are employed to analyse different conditions in which objects could be found. For example, illumination changes via color jittering with parameters chosen uniformly in the following ranges: brightness in [0.5,1.5]. contrast in [0.5,1], saturation in [0.5,1.5], and hue in [-0.1,0.1] may be used. Another alternative augmentation is image blur {e.g , due to dust on camera or wrong focus) which is introduced via Gaussian noise with 11x11 kernel and standard deviation (std) in [0.1,0.5]. Geometrical changes may be introduced via a combination of a) affine transformation keeping image center invariant, with degree in [-30,30], translate shift in [-0.2,0.2] for both horizontal and vertical axes, scale in [0.8,1.2], and shear in [-0.1,0.1]); b) random perspective with probability 0.2 and distortion amount 0.2, c-d) horizontal and vertical flip with probability 0.5 and 0.3, respectively. A combination of all of the augmentations of described above may also be used. To ensure fair comparisons among OCL methods, all samples are augmented with the same parameters in each set of experiments.

[201] All the results are averaged across 5 class orderings, although std is not reported to improve readability. Offline training upper bounds are not considered, because the classifier F(-) is different for each approach.

[202] Metrics. Both plasticity and forgetting are evaluated in comparison of OCL methods. All metrics are computed on test sets at the end of training phases. Accuracy is computed (Acc %, T) to assess the final model performance across all classes. Often, the room-aware relative gain is reported, RARG, (AR, %) of model 2 with accuracy Acc2 compared to a model i with Acci with respect to the available room of improvement, defined by AR = (Acc2 -Acc1) / (100 - AccJ. In addition, (1) backward transfer (BwT, T), which tracks the influence that learning a new task has on the preceding tasks’ performance, to measure stability; (2) forgetting (Forg, I), which averages difference of class-wise Acc at the last step and the best previous class-wise Acc; (3) plasticity (Pla, T) as the average Acc achieved on each task evaluated right after learning that task; (4) training time (TTime [min]); and (5) frames per second (FPS) in inference are also measured. Evaluation is carried out using an NVIDIA GeForce RTX 2080Ti GPU supported by an Intel(R) Xeon(R) Gold 5218R CPU@2.10GHz with 80 cores and 252GB RAM. Nonetheless, relative considerations remain the same on hardware mounted on robotic agents. All results are averaged across 5 runs with different seeds: only the mean value is shown since the standard deviation is negligible (< 0.16 in all cases). The present techniques are evaluated in multiple scenarios

[203] Same-domain data (OpenLORIS). In the first analyses, 16 backbones of variable sizes are considered. CNN-based backbones such as the 2x MobileNets-V3 (A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, and H. Adam, “Searching for MobileNetv3,” in ICCV, 2019, pp. 1314-132), 2x EfficientNets (M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in ICML, 2019, pp. 61, and 5x ResNets (K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770-778)) are considered. Transformer-based backbones such as 3x Swins (Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in ICCV, 2021, pp. 10012-10022) and 4x ViTs (Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Geliy, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words:Transformers for image recognition at scale,” in ICLR, 2021) are considered. These are commonly used baselines and allow the examination of variability of results depending on model size and learned visual features. All architectures have been pre-trained on the ImageNet dataset (A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Communications of the ACM, vol. 60, no. 6, pp. 84-90, 2017.).

[204] Figure 14 is a table showing the accuracy of the proposed techniques RobOCLe (SLDA) when compared to 9 OCL baslines on the same-domain OpenLORIS data on the 16 backbones described above. Most naive OCL methods ( / .e, PRCPT, SNB, SOvR) are unable to outperform FT, since, in this case, data is sufficient and correlated, with no severe change of domains / objects at test time. Replay data ( / .e, iCaRL) shows benefits, however, contradicts the important OCL assumption on data availability: robots should not store data due to privacy and storage constraints. SLDA outperforms all other competitors thanks to the Gaussian modeling of features, while its extension (SODA) can achieve competitive performance only on certain backbones: we argue that this is because SODA cannot create reliable class-wise correlation matrices, especially for larger networks.

[205] The proposed technique - RobOCLe - brings significant improvements with respect to the best prior approach (SLDA). RobOCLe reaches an outstanding average accuracy of 99.21% with a room-aware relative gain (AR) of 65.8% over it. This first analysis justifies the robustness of RobOCLe to multiple backbones on same-domain test data when sufficient training data is available. This experimental case is the main benchmark employed in current techniques. However, it may often not be the case in practise, as discussed in what follows.

[206] Real other-domain few-shot data (OpenLORIS-small, F-SIOL-310). When deploying robots to users and updating their Al models to recognize new objects, two simultaneous problems are typically encountered: 1) users do not label a large amount of data as it is a tedious time-consuming operation, and 2) training and testing sets have different conditions in terms of, e.g , object pose and background environment. For example, users show some pictures of their pet lying on the living room carpet during daytime, while at a later stage, the dog’s pose and the environment will change.

[207] Figure 15 is a table showing the accuracy on real other-domain few-shot datasets on ResNets and ViTs backbones. In particular, the challenging scenario of FS-OCL with a different domain (other-domain) at test time is analysed. Three benchmarks are considered based on OpenLORIS-small and F-SIOL-310 (with 5 and 10 shots). The evaluation of the various backbones is restricted to ResNets and ViTs which are the most popular ON Ns and Transformers at the time of writing (also, their effectiveness is confirmed in Figure 15a).

[208] Also in this case, the known FT and naive OCL methods show low accuracy. SODA achieves competitive performance only in certain cases. iCaRL improves significantly classical FT, however, it shows limited results. It is observed that when data is scarce, NCM reduces the gap with respect to SLDA. In some cases {e.g , ResNets on OpenLORIS-small) NCM outperforms SLDA. This may reflect two properties: 1) SLDA optimizes a larger number of parameters (prototypes and covariance matrix) compared to NCM (prototypes matrix only); and 2) feature dimensionality of ResNets ( e.g 2048 for RN152) is higher than that of ViTs ( e.g 1024 for ViT-L32), thus increasing the size of the covariance matrix. Therefore, on fewshot data, both NCM and SLDA can achieve competitive results.

[209] The present techniques (i.e. RobOCLe) improve both NCM and SLDA in almost every backbone and dataset, with just 2 exceptions out of 42 scenarios. The average gain AR ranges from 1.8% to 32.6%, with the highest gains shown in the F-SIOL-310 dataset using lower labelled samples. Remarkably, the gains on the other-domain datasets reported in this paragraph are even larger than the gains reported on same-domain data on OpenLORIS in Figure 14. Overall, these analyses certify the robustness of the present techniques to both low-shot labelled training samples (outperforming competitors with as little as 5-shots) and datasets (improving on both the OpenLORIS-small and F-SIOL-310 with different objects and / or conditions seen at test time).

[210] Figure 16 shows the change in per-step accuracy against incremental steps for each of four methods using a backbone - RN152 on the databset OpenLORIS-small. The proposed techniques - RobOCLe (NCM or SLDA) consistently outperforms the baselines at every step, making it particularly suitable for real-world applications where the number of classes is not defined a priori. Figure 16 shows that average gains of RobOCLe over NCM and SLDA are 5.81% and 1.20%, respectively.

[211] Controlled augmentations on other-domain data (OpenLORIS, OpenLORIS-small, F-SIOL-310). It was anticipated that the existing datasets are limited in having either partially different conditions at test time (OpenLORIS) or no different conditions at test time (F-SIOL-310). To cope with this limitation, further augmentations at train and test time are applied as described above.

[212] Figure 17a is a table showing results of experiments performed on OpenLORIS using the best CNN (RN152) and Figure 17b is a table showing results of experiments performed on OpenLORIS the best transformer (ViT-L16). In both tables, the two best OCL methods (NCM and SLDA) and the corresponding current techniques are compared with variable train and test augmentations. For each sub-table, train augmentations are reported on the rows and test augmentation is reported on the columns, and the average other-domain accuracy is computed (Avg OD). For each group of results (RobOCLe vs. baseline), the RARG of RobOCLe is reported in terms of Avg OD. On each scenario, RobOCLe shows superior results on severe and controlled synthetic augmentations when applied in training and / or in testing stages. Employing either color, geometric or noise augmentation generally improves Avg OD. Their combination improves Avg OD when using SLDA only, as it has more capacity than NCM to capture and disentangle input-level domain variations. When non-clean same augmentation is applied to both train and test sets, instead, lower accuracy is obtained compared to the original accuracy obtained using clean train / test sets. This is due to the tasks becoming much harder as, in the OCL setup, the agent experiences data only once. Therefore, convergence of models to learn robust representations is hindered. Also in this case, it is confirmed how RobOCLe is more effective than the baseline counterpart, especially on other-domain setups.

[213] Figure 17c is a table showing the accuracy on other-domain data with controlled augmentations on RN152 on the few-shot benchmark. In particular, RN152 is selected, and the effect of controlled augmentations on the few-shot datasets using SLDA is analysed. For each dataset and for each training augmentation, the present techniques robustly outperform the baseline by a large margin on both same-domain and other-domain data.

[214] Ablation studies. So far, we presented evaluation of results on the basis of 1) mostly replay-free OCL methods, and 2) accuracy have been presented. Here, the analyses are complemented.

[215] Figure 17d is a table showing the accuracy of multiple pooling mechanisms on OpenLORIS-small and SLDA. Here, Ay represents the average feature size multiplier compared to AVG. It is observed that [CLS] pooling performs slightly worse than AVG, due to the limited input data. Concatenating the top-k% features (RAP) increases the feature size but shows lower accuracy compared to the proposed method: using larger feature size is not enough to improve accuracy and robustness. All other approaches do not achieve competitive results. The present techniques slightly increase the feature size, however, it brings robust recognition results.

[216] Figure 17e is a table showing results of additional metrics on the OpenLORIS-small and ResNet152. TTime represents training time [min] and FPS represents frames per sec at test time. As described above, the multiple metrics characterize performance of CL agents. The present techniques improve performance with respect to baseline competitors with higher Acc (overall final performance), BwT (knowledge transfer to past classes), and lower Forg (average forgetting of all classes), by finding a better trade-off between stability and Pla (which is lower than, e.g , FT). For completeness, FwT is always 0, since incremental steps contain disjoint classes. FT implements the classifiers as a fully-connected layer and shows longer TTime than NCM and SLDA, which only requires computation of distance between embedded features and class prototypes. RobOCLe adds some computational overhead which is mostly affecting TTime, while inference FPS remains practically unchanged with a small decrease of less than 0.5% in all cases. On the other hand, the slight increase of TTime is not a relevant concern, since training is done on few-shot data only.

[217] As explained above, the RobOCLe method targets robust test-time object recognition for low-resource robots with limited labelled data and computational / storage capability. The RobOCLe method promotes invariance to augmentation via high order statistical moments of the embedded features of input samples. The results above show that the RobOCLe method achieves robust recognition in a variety of scenarios, using several backbones, low-shot setups, per-step accuracy and controlled test / train augmentation on both same-domain and other-domain data.

[218] Few-shot personalised instance recognition - a series of experiments for validating the method relating to personalised instance recognition (OBols) will now be described. In these experiments, the model Mo is implemented by YOLOv8, and pre-trained on MSCOCO (cited above) and then on To which comprises samples from OIV7 (also cited above) of the same object-level classes as in the personal dataset 7); The default learning parameters for pre-training are used. Several setups are designed whereby a few samples are assigned to the training set and the remaining ones to the testing (80%) and validation (20%) sets.

[219] Datasets. CORe50 (Vincenzo Lomonaco and Davide Maltoni, “CORe50: a new dataset and benchmark for continuous object recognition,’’ in CoRL, 2017, pp. 17-26) or iCubWorld Transformations (iCWT) (Giulia Pasquale, Carlo Ciliberto, Francesca Odone, Lorenzo Rosasco, and Lorenzo Natale, “Are we done with object recognition? the icub robot’s perspective,’’ Robotics and Autonomous Systems, vol. 112, pp. 260-281, 2019) as the personalized recognition datasets. CORe50: a subset of 45 personal instances ( / .e, |Q| = 45) belonging to 9 object-level classes ( / .e , |C0| = 9), acquired over 11 variable-background sequences. iCWT: a subset of 9 object-level classes with 10 personal instances each acquired under 5 sequences with diverse affine transformations of the items is considered. On both datasets, the personalization stage is restricted to the frames being correctly labelled by YOLOv8n, maintaining a balanced number of samples per instance and per sequence

[220] Metrics. The instance recognition accuracy is computed averaged within each objectlevel class (Acc0, %) and averaged over all instances (Acc;, %) on the test set. The relative gain is defined between two methods obtaining Acc,A and Acc, 2 (Acc; 2 >Acc; J as: A = 100 • (AccC2 -Accu) / (Accu).

[221] Most of the evaluation focuses on the personal instance-level accuracy, since the present techniques do not influence in any way the object-level detection accuracy and bounding box regression. For the sake of clarity, it is considered that each input sample contains one single instance. Nonetheless, the present techniques can handle multiple instances in input samples by running the instance-level prototype search for each input object independently. Provided that the general object detection results are accurate, the results described below would not change. Unless otherwise stated, all results are reported on YOLOv8n, being the most suitable for deployed applications, and in the case of 2 instances per each object-level category.

[222] Same domain. The first scenario considers 1-Shot from All Sequences (1SAS), therefore the same domain is seen during few-shot training (one sample per each sequence) and testing (all remaining samples from all sequences). Table 1 below reports Acc; in multiple setups having a variable number of instances per object-level class. + AEE indicates that augmented encoder embeddings have also been applied as per the description above. Table 1: Acct of OBols on 1SAS (same domain). # Instances per object class 2 3 4 5 Random 50.00 33.33 25.00 20.00 FT 54.12 34.43 24.11 21.41 SimpleShot 68.84 52.69 42.44 38.46 + AEE (ours) 76.34 62.04 51.79 46.30 A + 10.9 + 17.7 +22.0 +20.4 Proto Net 68.84 54.88 47.97 43.98 + AEE (ours) 77.08 64.57 55.81 53.39 A + 12.0 + 17.7 + 16.3 +21.4

[223] First, it is observed that gradient-based fine-tuning methods (e.g , FT) are not effective and obtain comparable results to a random classifier (lower bound). OBol via PFSL methods such as SimpleShot (Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens Van Der Maaten, “Simpleshot: Revisiting nearest-neighbor classification for few-shot learning,’’ arXiv: 1911.04623, 2019) and ProtoNet (Jake Snell, Kevin Swersky, and Richard Zemel, “Prototypical networks for few-shot learning,’’ NeurlPS, 2017) show large gains compared to FT by learning a metric space from the extracted features. In both cases, the augmentation of embeddings via our multi-order statistics (AEE) boost the recognition accuracy significantly, especially in presence of multiple instances per object. Remarkably, YOLOv8n can be personalised to achieve 77.08% Acc; when detecting 18 personal instances via just a few samples and via a backpropagation-free approach, assuming that a correct object-level classification and bounding box regression are output from the detection head.

[224] Figure 18a shows same domain (i.e. average) accuracy and per-object accuracy for three OBols methods via ProtoNet. In particular, Acc; and Acc0 are shown for OBols via ProtoNet in the case of 2 instances per object. Three configurations for ProtoNet are considered: at the logits level ( / .e, the output of the last layer of the detector’s head), at the encoder embedding level ( / .e , the output of the detector’s encoder) or via our multi-statistics augmented encoder embeddings ( / .e , with features augmented via multi-order statistics). It is observed that the present techniques consistently improve or achieve comparable results on every object class.

[225] Other domain. A more realistic, yet challenging, setup is designed considering 1-Shot from 1 Sequence (1S1S) during training and all remaining samples during testing. The model experiences different domains at training time (one sample from the first sequence only) and at testing time (all remaining samples from all sequences). Table 2 below summarizes the main results. Table 2: Acq of OBols on 1S1S (other domain). # Instances per object class 2 3 4 5 Random 50.00 33.33 25.00 20.00 FT 48.12 35.98 21.74 22.53 SimpleShot 60.62 43.95 34.31 29.84 + AEE (ours) 65.46 49.62 40.52 35.33 △ +8.0 + 12.9 + 18.1 + 18.4 Proto Net 60.33 46.52 37.13 32.13 + AEE (ours) 64.37 51.39 40.70 36.77 △ +6.7 + 10.5 +9.6 + 14.4

[226] Reducing the training samples further decreased the accuracy of FT, compared to the 1SAS setup. Also SimpleShot and ProtoNet show lower accuracy, due to the fewer training samples and the domain gap of the 1S1S setup. Nonetheless, they exhibit large gains over FT. The present techniques (+AEE) show a significant improvement of the performance in every case, even in the presence of a domain shift, and especially in case of multiple instances per object category. The gain attained by the present techniques is lower than the previous one due to the difficulty in reliably matching multi-order statistics between a single input sample from a single domain and several target samples from several domains.

[227] Figure 18b shows other domain (average) accuracy and per-object accuracy for three OBols methods via ProtoNet. The three methods are the same methods as used in Figure 18a. In particular, Acc0 is reported. Similarly to the Figure 18a results, it is confirmed that the present techniques obtain robust results across most of the classes.

[228] Variable training shots. Figure 19 shows accuracy at variable shots. Samples are drawn randomly from each sequence. PN: ProtoNet, SS: SimpleShot. It is observed that OBols with our AEE improve personal recognition accuracy regardless of the number of available training samples ( / .e , shots) for both ProtoNet and SimpleShot.

[229] Other YOLOv8 sizes are evaluated in Table 3 below on both general object detection and personalized instance recognition task. Table 3: Ablation on YOLOv8 models. YOLOv8: n s m I X Size [MB] 5.9 21.4 77.0 83.6 130.4 Detection Precision 72.0 73.2 70.4 72.4 77.2 Recall 72.5 70.3 74.4 75.5 77.3 mAP50 77.7 77.6 79.3 78.2 78.7 mAP50-95 60.9 61.7 62.3 62.7 63.1 1S1S Proto Net 44.03 43.56 48.44 55.38 58.77 +AEE (ours) 48.31 48.47 56.30 56.88 60.84 A +9.72 + 11.27 + 16.23 +2.72 +3.52 1SAS Proto Net 53.92 53.86 61.66 64.16 66.60 +AEE (ours) 62.71 66.67 75.29 75.21 74.65 A + 16.31 +23.79 +22.10 + 17.23 + 12.09

[230] Larger YOLOv8 models can improve detection performance, and this correlates with the personal instance recognition accuracy. The improvement of larger YOLOv8 models comes at a cost of a significantly larger model size and slower FPS: YOLOv8x improves personal recognition by about 25% compared to YOLOv8n, while having about 22x larger size and 3.6 x slower inference. The final choice depends on the hardware specifications of target devices. General object detection results are computed on the subset from OIV7. Personal instance recognition is evaluated on the subset from CORe50; we report a single score that is the meanp(AcCi P) where p indicates the setup with p instances per object (p e {2,3,4,5}).

[231] Computational inference time of our AEE on top of the OBol with ProtoNet increases by as little as 0.8% making our method lightweight with a nearly negligible impact.

[232] Additional ablation studies to evaluate the present techniques are reported here on ProtoNet. Removing the object-level conditioning lowered Acc, in the 1SAS setup from 77.1% of the present techniques (ProtoNet + AEE) to 70.9%, showing a relative gain of about 3% compared to the baseline (68.8%). This is due to the larger search space in metric learning. Removing the mask Sik leads to more background noise regions to flow into prototype computation, and it decreases accuracy by 2.6% in the 1SAS setup, and by even more (5.1%) in the 1S1S setup, since background varies across different sequences.

[233] Another personal dataset (iCWT) is shown in Table 4 against the highest baseline ProtoNet. Our approach exhibits robust gains across all setups ranging from 18 to 90 instances. Table 4:Acci on iCWT. PN: ProtoNet # Instances per object c ass 2 3 4 5 6 7 8 9 10 1SAS PN 82.3 66.5 62.6 58.8 46.9 46.0 43.6 41.5 41.5 + AEE 85.8 75.8 71.9 67.2 57.7 55.8 52.4 49.5 47.8 A +4.4 + 13.9 + 14.9 + 14.3 +22.9 +21.5 +20.2 + 19.1 + 15.2 1S1S PN 71.7 52.9 51.7 47.2 39.5 38.7 36.1 33.4 31.1 + AEE 79.6 60.1 56.5 49.8 41.1 40.6 37.5 34.2 33.0 A + 11.1 + 13.5 +9.4 +5.5 +4.1 +5.0 +3.8 +2.4 +6.1

[234] Few-shot instance-level personalization of object detectors was introduced as described above. A new method (OBol) to personalize detection models to recognize user specific instances of object categories was proposed. OBol is a backpropagation-free metric learning approach on a multi-order statistics feature space. It is believed that this setup and the present techniques could pave the way to personal instance-level detection and could stimulate future research and applications. By relying on multi-order statistics, OBol achieves consistent superior accuracy in distinguishing different instances. In the results, 77.1% personal object recognition accuracy in case of 18 personal instances is achieved, showing about 12% relative gain over current techniques. It is hoped that the problem formulation, approach and extensive comparison with current techniques described above will encourage future works on this direction.

[235] In summary, users want personalized object recognition services (i.e., adding new classes to the model or personalize to users’ instances of objects) on device, without sharing data with central server. Thus, we want to achieve on-device continual learning (CL) avoiding forgetting and storing samples. The state of art solutions use one or more of replay data, parameter isolation, regularization and additional network modules. These lead to the disadvantages of large storage requirements, large time complexity, low results and / or higher memory and inference time. As described above our solution provides personalized scene understanding using continuously generated user data without storing any private user data on a device and with minimal user supervision.

[236] We construct an enriched distribution of features considering high-order statistics to reduce forgetting; without storing any replay private user data on a device and via a computationally-efficient framework for on-device deployment. There is a modular approach and deployment can be on mobile and edge devices, such as smartphones, tablets, smart TVs, Samsung phones and DTV. There is also a data-efficient framework for the setup of low-shot learning and the solution is for robust deployment on other-domain data (different than training data domain).

[237] Users also want robust personalized object recognition services that work in variable conditions of objects (different pose or appearance) and environment (illumination changes, level of clutter,...). The state-of-the-art Solutions tackle CL without considering robustness of CL models. The disadvantages of SOTA solutions include large decrease of accuracy when evaluated on robustness and they find a worse tradeoff between forgetting and knowledge transfer leading to a final accuracy degradation. As described above our solution can provide robust adaptation of object recognition models to new classes (to recognize via continual learning) and new domains (i.e., environments, in domain generalization).

[238] Our solution allows a device to recognise new classes, in other words learn to recognize additional classes / objects / categories over time. Similarly, our solution allows a device to recognise new domains. In other words, the device learns to recognize the same classes in different contexts / domains / environments (different room, different house, different environment conditions). The Method promotes parameter- and data- efficient continual learning and domain generalization in object recognitions via a modular approach efficient with respect to storage / memory and training / inference speed with a-priori unknown number of test classes. We use input-level statistics for inverse data normalization to bring input data in similar domain of training data; feature-level enrichment to promote invariance to deformation of samples and output-level model calibration to improve robustness of overall recognition: making predictions aligned with confidence score.

[239] Users also want efficient personalized object recognition services so that they do not impact on usability of the device. State-of-the-art Classical Solutions use several replicas of the massive Al model and / or grow new branches for new classes to recognize. The disadvantages of SOTA solutions include that they are not efficient and unfeasible on constrained devices (high memory, storage and inference time); and that new branches have been used to tackle CIL (class-incremental learning) which are not viable in our case. Our solution is personalized object recognition services with low-footprint personal Al models learned from user-specific data. In our solution, object recognition models are personalized for a specific user exploiting low-footprint storage of aggregate input-level, feature-level and output-level historical statistics used during inference to normalize the data at different levels of the model. The proposed method is efficient with respect to storage / memory requirements and training / inference speed and uses a modular approach.

[240] The key benefits of our proposed approach include two-fold personalization: The final model works best for each specific user categories and environments. There is also storage efficiency. There is parameter-efficient continual learning since massive Al model used as backbone is frozen and shared across the learning problems. There is no need for storing samples belonging to previous tasks (replay-free approach) and no need for storing sample belonging to current task (online learning). There is also minimal user supervision and computational efficiency. Unlike recent CL methods, we add only <0.45% increase in inference time. Unlike recent CL methods, we learn new classes from as little as 5 samples, making it suitable for few-shot learning. Therefore minimal user supervision is needed. There is also robustness. As illustrated in the results above the robustness and accuracy of our few-shot online CL method has been evaluated over multiple benchmark datasets (OpenLORIS, OpenLORIS-small, F-SIOL-310-5shots, F-SIOL-310-10shots), 16 backbone network architectures (both CNN-based and Transformer-based) according to several metrics (accuracy, plasticity, forgetting, time, etc) with variable test-time augmentations. Finally, there is modularity and the proposed approach can be applied on top of existing approaches.

[241] The novel scenario of parameter-efficient FS-OCL for low-resource robotic agents for robust test-time performance in variable conditions is thus introduced. Previous setups are extended to analyse and improve the robustness of the algorithms to test-time distortions without updating model weights at test time. This challenging and very practical scenario has not been addressed by current techniques, and in general, little exploration has been carried out on robustness of CL models to variable test-time conditions.

[242] The novel aspects of the present techniques for RobOCLe can be summarized as follows: • A new FS-OCL paradigm for low-resource robots is tackled, where storage is unavailable and computation power is low (i.e , most of model weights cannot be updated). • A new parameter-efficient FS-OCL method (RobOCLe) suitable for mobile robotic agents and FS data is introduced. RobOCLe extracts high order statistics from embeddings, building more reliable feature representations robust to variable domain and object conditions. RobOCLe shows consistent improvements against 10 OCL baselines on 4 benchmarks and 16 backbones. RobOCLe shows a room-aware relative accuracy gain (RARG, Sec. 4) of average 65.8% on same-domain data and 16.4% on other-domain data compared to the best-competing approach, while decreasing inference FPS by less than 0.5%. • A new evaluation paradigm to determine robustness of vision models is presented, and particularly CL models, over a suite of real and synthetic augmentations, resembling practical use cases. RobOCLe shows average RARG of 18.3% on controlled augmentations of other-domain data.

[243] We have also constructed an object-conditioned bag of instances (OBol) based on multiorder statistics of extracted features, where generic object detection models are extended to search and identify personal instances from the OBols metric space, without the need for backpropagation. By relying on multi-order statistics OBol achieves consistent superior accuracy in distinguishing different instances. OBols provide superior results on both same and other domain data (11-22% and 7-18% relative gains respectively). Overall we achieve 77.1% personal object recognition accuracy in case of 18 personal instances.

[244] Further aspects of the novel aspects of the present techniques are detailed in the following numbered clauses:

[245] Clause 1. A computer-implemented method for classifying, on a user device, an image using a machine learning, ML, model which comprises a feature extractor, a pooling module and a classifier, the method comprising: receiving an input image; generating, using the feature extractor of the ML model, a feature vector for the input image; creating, using the pooling module, a statistical vector comprising at least one statistical moment which is extracted from the feature vector; concatenating, using the pooling module, the statistical vector to the feature vector to create a concatenated vector; and identifying, using the classifier, a classification for the input image by comparing the concatenated vector to a plurality of prototype vectors which are stored on the user device, wherein each prototype vector is a statistical representation of a specific classification; identifying the prototype vector which is most similar to the concatenated vector; and assigning the classification of the identified prototype model to the input image. Clause 2 The method of clause 1, wherein the at least one statistical moment comprises at least one of average, variance, skewness and kurtosis. Clause 3 The method of clause 1 or clause 2, wherein each prototype vector is a mean feature vector of a Gaussian model for each classification. Clause 4 The method of clause 3, further comprising updating, in response to an input from a user of the user device, the plurality of prototype vectors which are stored on the user device. Clause 5 The method of clause 4, wherein updating the plurality of prototype vectors comprises updating an existing prototype vector by receiving at least one new image which is labelled with a classification corresponding to the existing prototype vector; creating, using the pooling module, a concatenated vector for the at least one new image; calculating a mean feature vector using the created concatenated vector for the at least one new image and the existing prototype vector; and outputting the calculated mean feature vector as the updated prototype vector for the classification. Clause 6The method of clause 4 or clause 5, wherein updating the plurality of prototype models comprises creating a new prototype vector by: receiving at least one new image which is labelled with a new classification; creating, using the pooling module, a concatenated vector for the at least one new image; calculating a mean feature vector from each created concatenated vector; and outputting the calculated mean feature vector as a new prototype model for the new classification. Clause 7 The method of any one of clauses 3 to 6, further comprising updating a covariance matrix of the Gaussian model which is shared across the classifications. Clause 8 The method of any one of the preceding clauses, comprising normalizing or standardizing the input image before generating the feature vector. Clause 9 The method of clause 8, wherein normalizing or standardizing the input image comprises: generating image statistics for the input image; comparing the generated image statistics with stored image statistics; and adjusting the input image based on any differences between the generated image statistics and the stored image statistics. Clause 10 The method of clause 9, further comprising updating the stored image statistics by storing the generated image statistics for the input image. Clause 11 The method of any one of the preceding clauses, wherein identifying a classification for the input image comprises identifying an object category for at least one detected object in the input image. Clause 12. The method of any one of clauses 1 to 10, wherein each object category comprises at least one instance-level category which is a sub-category for a specific instance of an object within the object category and identifying a classification for the input image comprises identifying an instance-level category within an object category. Clause 13. The method of clause 12, further comprising identifying, using the classifier, an instance-level category for the input image by determining, using an object detector, an object category for at least one detected object in the input image; selecting prototype vectors from the plurality of prototype vectors which have the determined object category and at least one instance-level category; comparing the concatenated vector for the input image only to each selected prototype vector to identify the prototype vector which is most similar to the concatenated vector; and assigning the instance-level category of the identified prototype model to the input image. Clause 14 A computer-implemented method for classifying, on a user device, an image using a machine learning, ML, model which comprises a feature extractor, an object detector, a pooling module and a classifier, the method comprising: receiving an input image; determining, using the object detector, an object category for at least one detected object in the input image; generating, using the feature extractor of the ML model, a feature vector for the input image; creating, using the pooling module, a statistical vector comprising at least one statistical moment which is extracted from the feature vector; concatenating, using the pooling module, the statistical vector with the feature vector to create a concatenated vector; identifying, using the classifier, a classification for the input image by selecting prototype vectors from the plurality of prototype vectors which have the determined object category and at least one instance-level category, wherein an instance-level category is a sub-category for a specific instance of an object within the object category; comparing the concatenated vector for the input image only to each selected prototype vectors to identify the prototype vector which is most similar to the concatenated vector; and assigning the instance-level category of the identified prototype vector to the input image. Clause 15 The method of any one of the preceding clauses, wherein the ML model further comprises an object detector and the method further comprises detecting, using the object detector, at least one object in the input image; determining, using the object detector, a location within the image of the at least one detected object; and wherein creating the concatenated vector comprises creating a concatenated vector for a portion of the image at the determined location. Clause 16The method of clause 14 or clause 15, further comprising defining the location using bounding box coordinates; and masking regions within the input image outside the location of the at least one detected object. Clause 17 The method of any one of clauses 14 to 16, comprising generating the concatenated vector by extracting, using the pooling module, the at least one statistical moment from a feature vector which is generated from a portion of the image at the determined location. Clause 18 The method of any one of the preceding clauses, wherein prior to receiving an input image, the method comprises locally training the ML model to personalise the ML model, wherein local training comprises: receiving a training set of images comprising a plurality of images each of which is labelled with a classification; generating, using the feature extractor of the ML model, a feature vector for the input image; creating, using the pooling module, a statistical vector comprising at least one statistical moment which is extracted from the feature vector; concatenating, using the pooling module, the statistical vector to the feature vector to create a concatenated vector; and computing a prototype vector for each specific classification using the concatenated vectors for each training image labelled with the specific classification. Clause 19 The method of clause 18, further comprising generating image statistics for each image in the training set; and storing the generated statistics as stored statistics for normalization or standardization of an input image. Clause 20 The method of any one of the preceding clauses, further comprising controlling the user device based on the identified classification. Clause 21 A user device comprising: a processor coupled to memory, wherein the processor is configured to carry out the methods of any one of clauses 1 to 20. Clause 22 A user device according to clause 21 in the form of a vacuum cleaner, smart oven or smart fridge. Clause 23. A system comprising a server and a primary user device which is connected to the server and which comprises a processor, wherein the processor of the primary user device is further configured to: receive a pre-trained ML model from the server; and locally train the ML model to personalise the model according to the method of clause 18 or clause 19. Clause 24 A system according to clause 23, further comprising a secondary user device comprising a processor coupled to memory, wherein the processor is configured to: receive a pre-trained ML model from the server or the primary user device; locally train the ML model to personalise the model according to the method of clause 18 or clause 19; and send the locally trained ML model to the primary user device for use on the primary user device. Clause 25 A non-transitory data carrier carrying code which, when implemented on a processor on a user device, causes the user device to carry out the method of any of clauses 1 to 20.

Claims

1. A computer-implemented method for classifying, on a user device, an image using a machine learning, ML, model which comprises a feature extractor, an object detector, a pooling module and a classifier, the method comprising:receiving an input image;determining, using the object detector, an object category for at least one detected object in the input image;generating, using the feature extractor of the ML model, a feature vector for the input image;creating, using the pooling module, a statistical vector comprising at least one statistical moment which is extracted from the feature vector;concatenating, using the pooling module, the statistical vector with the feature vector to create a concatenated vector;identifying, using the classifier, a classification for the input image byselecting, from a plurality of prototype vectors, prototype vectors which have the determined object category and at least one instance-level category, wherein an instance-level category is a sub-category for a specific instance of an object within the object category;comparing the concatenated vector for the input image only to each selected prototype vectors to identify the prototype vector which is most similar to the concatenated vector; andassigning the instance-level category of the identified prototype vector as the classification for the input image.

2. The method of claim 1, further comprising:determining, using the object detector, a location within the image of the at least one detected object; andwherein creating the concatenated vector comprises creating a concatenated vector for a portion of the image at the determined location.

3. The method of claim 2, further comprisingdefining the location using bounding box coordinates; andmasking regions within the input image outside the location of the at least one detected object.

4. The method of any one of the preceding claims, comprising generating the concatenated vector by extracting, using the pooling module, the at least one statistical moment from a feature vector which is generated from a portion of the image at the determined location.

5. The method of any one of the preceding claims, wherein the at least one statistical moment comprises at least one of average, variance, skewness and kurtosis.

6. The method of any one of the preceding claims, wherein each prototype vector is a mean feature vector of a Gaussian model for each classification.

7. The method of any one of the preceding claims, wherein the plurality of prototype vectors are stored on the user device.

8. The method of claim 7, further comprising updating, in response to an input from a user of the user device, the plurality of prototype vectors which are stored on the user device.

9. The method of claim 8, wherein updating the plurality of prototype vectors comprises updating an existing prototype vector byreceiving at least one new image which is labelled with a classification corresponding to the existing prototype vector;creating, using the pooling module, a concatenated vector for the at least one new image;calculating a mean feature vector using the created concatenated vector for the at least one new image and the existing prototype vector; andoutputting the calculated mean feature vector as the updated prototype vector for the classification.

10. The method of claim 8 or claim 9, wherein updating the plurality of prototype vectors comprises creating a new prototype vector by:receiving at least one new image which is labelled with a new classification;creating, using the pooling module, a concatenated vector for the at least one new image;calculating a mean feature vector from each created concatenated vector; andoutputting the calculated mean feature vector as a new prototype vector for the new classification.

11. The method of any one of claims 7 to 10, further comprising updating a covariance matrix of the Gaussian model which is shared across the classifications.

12. The method of any one of the preceding claims, comprising normalizing or standardizing the input image before generating the feature vector.

13. The method of claim 12, wherein normalizing or standardizing the input image comprises:generating image statistics for the input image;comparing the generated image statistics with stored image statistics; andadjusting the input image based on any differences between the generated image statistics and the stored image statistics.

14. The method of claim 13, further comprising updating the stored image statistics by storing the generated image statistics for the input image.

15. The method of any one of the preceding claims, wherein prior to receiving an input image, the method comprises locally training the ML model to personalise the ML model, wherein local training comprises:receiving a training set of images comprising a plurality of images each of which is labelled with a classification;generating, using the feature extractor of the ML model, a feature vector for the input image;creating, using the pooling module, a statistical vector comprising at least one statistical moment which is extracted from the feature vector;concatenating, using the pooling module, the statistical vector to the feature vector to create a concatenated vector; andcomputing a prototype vector for each specific classification using the concatenated vectors for each training image labelled with the specific classification.

16. The method of claim 15, further comprisinggenerating image statistics for each image in the training set; andstoring the generated statistics as stored statistics for normalization or standardization of an input image.

17. The method of any one of the preceding claims, further comprising controlling the user device based on the identified classification.

18. A user device comprising:a processor coupled to memory, wherein the processor is configured to carry out the methods of any one of claims 1 to 17.

19. A user device according to claim 18 in the form of a vacuum cleaner, smart oven or smart fridge.

20. A system comprisinga server anda primary user device which is connected to the server and which comprises a processor coupled to memory, wherein the processor of the primary user device is further configured to carry out the methods of any one of claims 1 to 17.

21. A system according to claim 20, further comprising a secondary user device comprising a processor coupled to memory, wherein the processor is configured to:receive a pre-trained ML model from the server or the primary user device;locally train the ML model to personalise the model according to the method of claim 15 or claim 16; andsend the locally trained ML model to the primary user device for use on the primary user device.

22. A non-transitory data carrier carrying code which, when implemented on a processor on a user device, causes the user device to carry out the method of any of claims 1 to 17.T +44(0)30 0300 2000A

Images