Electronic device, system, and method

Abstract

An electronic device comprising circuitry configured to update weights of a of a pre-trained AI model (51, ƒ(.;θ)) based on a set of sparse modifications (52) to obtain an updated AI model (56, (I)), and to receive input data (53) which is used as input to the updated AI model (56, (I)).

Description

[0001] ELECTRONIC DEVICE, SYSTEM, AND METHOD TECHNICAL FIELD The present disclosure generally pertains to the field of Artificial Intelligence (AI), in particular to methods, devices and systems that relate to fine-tuning or calibration of pre-trained AI models. TECHNICAL BACKGROUND Artificial intelligence (AI) refers to the intelligence of computer software. AI technology relies on machine learning algorithms such as neural network technology. AI is widely used throughout industry, government and science. Applications are advanced web search engines, recommendation systems, understanding human speech, self-driving cars, and the like. Out-of-distribution (OOD) in artificial intelligence is a term that refers to the situation when a machine learning model encounters an input data that is different from the distribution of the data that it was trained on. The OOD problem is the challenge of detecting and handling data that is outside the distribution of the training data. OOD data can cause the model to make incorrect predictions with high confidence, which can be problematic for safety-critical applications. Therefore, it is important to detect OOD data and handle them appropriately. There are various methods for OOD detection, such as classification-based, density-based, and distance-based ones. Although there exist techniques that address the OOD problem, it is generally desirable to provide improved techniques that address the OOD problem. SUMMARY According to a first aspect the disclosure provides an electronic device comprising circuitry configured to update weights of a pre-trained AI model based on a set of sparse modifications to obtain an updated AI model, and to receive input data which is used as input to the updated AI model. According to a further aspect, the disclosure provides a computer-implemented method comprising updating weights of a pre-trained AI model based on a set of sparse modifications to obtain an updated AI model, and to receive input data which is used as input to the updated AI model. According to a further aspect, the disclosure provides a computer-implemented method of training an AI model, the method comprising determining, based on a pre-trained AI model a set of sparse modifications. According to a further aspect, the disclosure provides an electronic device comprising circuitry configured to train an AI model, the circuitry being configured for determining, based on a pre- trained AI model a set of sparse modifications. Further aspects are set forth in the dependent claims, the drawings and the following description. BRIEF DESCRIPTION OF THE DRAWINGS Embodiments are explained by way of example with respect to the accompanying drawings, in which: Fig.1 schematically describes a supervised learning setting related to a vision language model comprising an image encoder and a text encoder; Fig.2 describes a schematic overview of SAFT and / or calibration of an AI model; Fig.3 shows an exemplifying implementation of the SAFT process; Fig.4 schematically shows a graphic representation of a fine-tuned model obtained via the sparse update; Fig.5 schematically shows a fine-tuned AI model obtained by the processes above in the inference phase; Fig.6 illustrates a network including a server providing a SAFT update to an electronic device for updating an AI model based on sparse updating as described in Fig.5; and Fig.7 illustrates an electronic device for implementing the SAFT process and sparse updating based on a SAFT update. DETAILED DESCRIPTION OF EMBODIMENTS Before a detailed description of the embodiments under reference of Fig.1 is given, general explanations are made. Some embodiments pertain to an electronic device comprising circuitry configured to update weights of a pre-trained AI model based on a set of sparse modifications to obtain an updated AI model, and to receive input data which is used as input to the updated AI model. The update of the pre-trained AI model with the set of sparse modifications may for example provide an adaptation of the foundation AI model that results in an enhanced performance, contributing to an improved user experience. The update of the pre-trained AI model with the set of sparse modifications may efficiently adapt the pre-trained AI model with minimal alterations to the pre-trained (foundation) model. A pre-trained AI model may for example be any algorithm or program that has been trained on a set of data to recognize certain patterns or make certain decisions without further human intervention. The pre-trained AI model may for example apply different algorithms to relevant data inputs to achieve the tasks, or output, they have been programmed for. The electronic device may for example be any type of computing device or system, e.g. a terminal computer, a smartphone, a tablet or other mobile device, such as a laptop or a wearable device, such as smart glasses, a smart watch, headphones, earphones etc. The electronic device may also relate to a server device, or a cloud service. Circuitry may for example include a processor, a memory (RAM, ROM or the like), a storage, input means (mouse, keyboard, camera, etc.), output means (display (e.g. liquid crystal, (organic) light emitting diode, etc.), loudspeakers, etc., a (wireless) interface, etc., as it is generally known for electronic devices (computers, smartphones, etc.). Moreover, it may include sensors for sensing still image or video image data (image sensor, camera sensor, video sensor, etc.), for sensing a fingerprint, for sensing environmental parameters (e.g. radar, humidity, light, temperature), etc. According to some embodiments, the pre-trained AI model is implemented as a neural network and the set of sparse modifications comprises weights defining parameters of the neural network. The set of sparse modifications may for example be called a sparse update. The set of sparse modifications on which the adaptation process relies may for example be efficiently stored in a data storage, minimizing resource requirements. The update of the pre-trained AI model with the set of sparse modifications may prevent the update from forgetting general knowledge in the pre-trained AI model. The set of sparse modifications may be configured to adapt the pre-trained AI model with regard to out-of-distribution generalization. The set of sparse modifications may be configured for fine-tuning the pre-trained AI model to a new domain or data distribution. For example, the set of sparse modifications may be configured to fine-tune the pre-trained AI model to a specific usage scenario. For example, in image recognition a model trained on data from one dataset or task may need to generalize to data from other datasets or tasks, where the image quality, resolution, background, or object categories may differ. The set of sparse modifications may be configured for calibration of the pre-trained AI model. Calibration may for example involve adapting the pre-trained AI model to the original domain on which the AI model was initially trained, with a focus on accommodating specific device characteristics. For example, the set of sparse modifications may be configured to calibrate the pre-trained AI model to a specific camera. This may ensure that the adapted model can still generalize effectively, even in the face of camera-specific variations. Still further, calibration may refer to calibrating a model in a smart assistant etc. to user's voice and speech pattern or a user's gestures. The pre-trained AI model may be a vision language model that comprises an image encoder and a text encoder. For example, the pre-trained AI model may be a Contrastive Language-Image Pre-training (CLIP) model, or the like. The electronic device may be configured to transform a class name into a text prompt describing the class, the text prompt being used as input to the text encoder. The circuitry may be configured to use the cross-entropy loss for updating the pre-trained AI model. The circuitry may be configured to provide multiple sets of sparse modifications that are individually tailored for specific use cases. The circuitry may be configured to determine the updated AI model by inserting the set of sparse modifications into the pre-trained AI model, thereby replacing respective original parameters of the pre-trained AI model. The circuitry may be configured to apply the updated AI model to the input data to generate predictions or outputs. In this way, during an inference phase, the machine learning model may be deployed into a production environment, making it possible to generate predictions for inputs provided by end-users. The updated AI model may be a neural network and the circuitry may be configured to pass the input data to the first layer of the neural network. The receiving of the input data may for example be part of an inference phase. The updated AI model may for example provide an output to be used in a given task. This output of the updated AI model may for example be used for controlling a setting of an electronic device (like a TV or smartphone), for recognizing speech, recognizing images, or recognizing gestures The set of sparse modifications may for example be received from an input, from a service provider, over a network, or via a data file. The circuitry may for example be configured to select the set of sparse modifications from multiple sets of sparse modifications. This selection may for example be based on a user input, or on a particular task to be achieved. The circuitry may for example be configured to detect context information and to select the set of sparse modifications based on the context information. For example, context information may indicate if a camera chip is currently used in the dark or in bright light, and depending on this context information a set of sparse modifications that optimizes the AI model for the dark or an alternative set of sparse modifications that optimizes the AI model for the bright light is chosen. As another example, the context information may describe a captured movement and apply an appropriate update. Still alternatively, the update may relate to a smart assistant that optimizes the AI model for different environments like office or during commute. Some embodiments may pertain to a computer-implemented method including updating weights of a pre-trained AI model based on a set of sparse modifications to obtain an updated AI model, and to receive input data which is used as input to the updated AI model. Some embodiments pertain to a computer-implemented method of training an AI model, the method comprising determining, based on a pre-trained AI model a set of sparse modifications. The set of sparse modifications may comprise learnable weights. Determining the set of sparse modifications (the “fine-tuning of the model”) may be based on training data. That is, determining the set of sparse modifications (the “fine-tuning of the model”) may be based on an “in-distribution” dataset and the evaluation can then be done on this dataset but also other “out- of-distribution” datasets. The training data may be configured for fine-tuning the pre-trained AI model to a new domain or data distribution. The training data may be configured for calibration of the pre-trained AI model. The method may include determining, based on the pre-trained AI model and based on the set of sparse modifications, an updated AI model. The method may include determining the set of sparse modifications with training data in a sparse update. The method may include updating, during the sparse update, only a subset of weights of the pre- trained AI model. Frozen weights of the pre-trained AI model may remain unchanged during sparse update. The method may include defining a maximum size of the set of sparse modifications. For example, the size of the subset of sparse modifications may in one embodiment be determined by a maximum size of the sparse updates that can be accommodated by the device (e.g. by memory size, disk size, etc.). The method may include identifying the weights that have the greatest influence on the model's performance on training data. The method may further include determining a selection parameter which directly or indirectly defines the amount of data in set of sparse modifications. The method may include determining a mask which is configured to identify weights of the pre- trained AI model as learnable weights and to identify other weights of the pre-trained AI model as frozen weights. The method may include including into the set of sparse modifications only learnable weights, whereas frozen weights are not included. The method may include resetting frozen weights to their original values. The method may include updating only a subset of weights whose gradient magnitude is large. For example, the method may include update exclusively the weights possessing the maximum absolute gradient. The method may include computing the gradient of a likelihood function of the pre-trained AI model with respect to the parameters of the model. The method may include averaging the gradient of the likelihood function over training data. In this way, a matrix may for example be determined that captures the importance of the model parameters. The method may include keeping, during sparse update, a first encoder frozen while only a second encoder is updated. The first encoder that is kept frozen may be a text encoder and the second encoder that is updated may be an image encoder. Keeping the text encoder frozen may result in increased performance, especially when dealing with open-class scenarios. The method may include returning a fine-tuned model with parameters that comprise the set of sparse modifications. The method may comprise returning the set of sparse modifications. The method may include storing the set of sparse modifications. Some embodiments pertain to an electronic device comprising circuitry configured to implement any of the methods including any of the features described above. The methods as described herein may correspond to the electronic device and circuitry described above, and the methods may accordingly exhibit any feature described above with respect to the electronic device and / or circuitry and / or any suitable feature described below with respect to any one of the figures. The method may be performed by the circuitry and / or by the electronic device described above. The processes described here may also be realized within a device that realizes a computing system. The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed. Out-of-distribution (OOD) generalization The embodiments described below address the problem of handling distribution shifts from training, known as out-of-distribution (OOD) generalization. Out-of-distribution (OOD) generalization aims at training a model that can perform well on new data that may have different distributions from the training data. For example, a model trained on images of cats and dogs may fail to recognize other animals that it has never seen before. OOD generalization requires the model to learn features and patterns that are invariant across different environments and contexts, and to avoid overfitting to the specific characteristics of the training data. While a vision-language model like CLIP has demonstrated remarkable zero-shot performance, further adaptation of the model to downstream tasks, unfortunately, leads to undesirable degradation for OOD data. “Zero-shot classification” refers to a setup in deep learning where, at test time, a learner observes samples from classes which were not observed during training, and needs to predict the class that they belong to. In other words, it is a method of classifying data into categories that were not present in the training data. There are many methods and techniques that have been proposed to address the OOD generalization problem, such as unsupervised representation learning, supervised model learning, and optimization. These methods try to find a balance between exploiting the information in the training data and exploring the possible variations in the test data. Some of the methods also leverage causal inference and domain adaptation to identify and remove the confounding factors that may cause spurious correlations or biases in the data. OOD generalization can be beneficially applied in a variety of AI use cases, such as image recognition, medical diagnosis, autonomous driving, natural language understanding, and many others. In the embodiments described below, processes for OOD generalization are described within the framework of a visual language model use case. However, the embodiments are not restricted to this type of use case. The principles of the embodiments can equally well be applied to other uses cases that may benefit from OOD generalization. Vision language models A vision language model is a type of artificial intelligence model that can process both images and natural language text. It usually consists of an image encoder, a text encoder, and a strategy to fuse information from the two encoders. Vision language models can perform various tasks that require understanding both visual and linguistic information, such as image captioning, visual question answering, image retrieval, and more. Vision language models are often pre- trained on large-scale datasets of image-text pairs, and then fine-tuned or transferred to specific downstream tasks. Some examples of vision language models are CLIP, VLM, and ViLT. Although the processes described in the embodiments below are general and can be applied to any neural network, the embodiments described here exemplarily focus on an improvement of the generalization of CLIP on fine-tuning downstream tasks. CLIP (Contrastive Language-Image Pre-training), which is described in more detail in Reference [2], efficiently learns visual concepts from natural language supervision. It uses a neural network that is trained on a large dataset of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. For example, given an image of a dog, CLIP can be asked to rank different captions such as “a dog”, “a cat”, “a flower”, or “a car”. CLIP will assign the highest score to “a dog”, since it matches the image best. CLIP can also generate captions for images by using a language model that is conditioned on the image features extracted by CLIP. Fig.1 schematically describes a supervised learning setting related to a vision language model comprising an image encoder and a text encoder. In this supervised learning setting data could come from different domains or distributions. Let be a training example sampled from some distribution , where is a raw image with the class label. Given a set of training examples, the embodiments described below in more detail learn a classifier , which maps an input image to its class label. The classifier is evaluated under two settings: in-distribution (LID) and out-of-distribution (LOOD). In ID settings, test examples are drawn from the same training distribution . In OOD settings, test examples are drawn from a distribution , which is different from . The latter is more challenging since test examples might come from domains that are different from training domains. Following Reference [1], the embodiments consider OOD settings into two scenarios, including domain shift and open-class. In the domain shift scenario, test examples have the same class labels as in training data, whereas in the open-class scenario, test data might contain novel classes, which are not present in training data. The embodiments described here consider CLIP as the backbone for a vision-language model. Let be an image encoder that maps an input image into an embedding Iiof dimension and let be a text encoder that maps a text caption t into an embedding T of dimension . Given an input pair of an image and its caption, the embeddings are computed by passing inputs to the encoders and , i.e., and . To perform zero-shot classification, the embodiments first transform class names y into a suitable text prompt t describing the class. For example, “a photo of a [CLASS]”, where the “[CLASS]” token is replaced by the actual class name. Finally, the inference is performed by comparing the image embedding with the embedding of each text prompt as where is the number of class labels in the test set and denotes the temperature parameter, which in CLIP controls the prediction confidence for classification. According to the well-known principles of CLIP, the output of the text encoder, i.e. the embeddings T1, …, TLare compared to the output of the image encoder Iito obtain similarities Ii.T1, …,.TLwhere “.” denotes the inner-product between two vectors. This is then fed to a Softmax function 10 which is used to convert the output of the model into a probability distribution over all possible labels. The probability distribution obtained by Softmax 10 is used in a loss function to perform the training of the model with a gradient descent algorithm. Although contrastive loss is used to train CLIP, in the embodiments described below, the cross- entropy loss 11 is considered to fine-tune CLIP for downstream tasks. In an embodiment, during fine-tuning, the text encoder is kept frozen, while only the image encoder is updated. Fine-tuning the text encoder might lead to overfitting on specific knowledge from downstream training classes since the inherent semantic relationships of concepts learned from a diverse range of data might be destroyed. Keeping the text encoder frozen may result in increased performance, especially when dealing with open-class scenarios. Sparse Adaptation for Fine-Tuning (SAFT) In the embodiments described below in more detail, a Sparse Adaptation for Fine-Tuning (SAFT) is introduced. SAFT is a method that may prevent fine-tuning from forgetting the general knowledge in the pre-trained model. In particular, SAFT only updates a small subset of important weights whose gradient magnitude is large. Extensive experiments show that with only 0.1% of the model weights or even less, SAFT can significantly improve the performance of, e.g., a vision language model such as CLIP. This approach has demonstrated its effectiveness in enhancing performance on the specific task for which fine-tuning is applied. Remarkably, it also maintains, and often even enhances, performance on datasets beyond the initial domain. The process described in the embodiments consistently outperforms baseline methods across several benchmarks. On the few-shot learning benchmark of ImageNet and its variants, SAFT gives a gain of 5.15% on average over the conventional fine-tuning method in OOD settings. Fig.2 describes a schematic overview of SAFT and / or calibration of an AI model. To employ SAFT and / or calibration of an AI model, the embodiments commence at 21 (Phase 1) by identifying the weights that have the greatest influence, indicated by the highest absolute gradient, on the model's performance concerning the fine-tuning / calibration dataset. Here, "fine- tuning" pertains to adapting the model to a new domain or data distribution, while "calibration" involves adapting it to the original domain on which the AI model was initially trained, with a focus on accommodating the specific sensor characteristics (where the specific sensor characteristics are taken into account by either passing the data through the sensor or by using a good simulation of the specific sensor). Subsequently, at 22 (Phase 2), the embodiments exclusively update the weights possessing the maximum absolute gradient. This approach offers the benefit of making minimal alterations to the original model, which can be efficiently stored on the device (e.g., the original AI model could be stored in a ROM where only during runtime the SAFT modifications are loaded or the AI model could be part of the operating system and is available to programs through a library call). Additionally, it allows us to tailor the AI model to the particular sensor's characteristics while preserving its ability to generalize effectively. According to an embodiment, in Phase 1 (21 of Fig 2), during sparse update, the gradient of the log-likelihood function is computed with respect to the parameters , i.e., - . These gradients are averaged over training data PID,where N is the number of training pairs The vector captures the importance of weights for the downstream tasks. In phase 2 (22 of Fig.2), following the direction of negative gradients, the process can fine-tune the learnable weights by maximizing the log-likelihood function according to the well-known principles of back propagation. However, due to the limitation of training data, one can easily overfit this loss function. Consequently, the fine-tuned model cannot generalize to other downstream tasks. To address this issue, the embodiments estimate which parameters are important to the task and therefore only update those parameters. In Phase 1, the process determines only weights that have high gradient magnitudes since those weights should be modified in order to increase . Pre-trained models can be seen as multi-task learners, where each part of the model is specialized on a specific task. By updating only this part, we expect the network to focus only on relevant features to solve the task. As a result, it can be robust to out-of-distributions. Fig.3 shows an exemplifying implementation of the SAFT process. The process obtains a model that has been pre-trained with a large training distribution P. The process further obtains a number of N pairs of input data for fine-tuning from the ID training data PID. Still further, the process obtains a predefined selection ratio which defines the number of learnable weights, i.e., the ratio of learnable weights to the overall number N of weights. In phase 1, all elements of the weight importance vector , are first set to zero. In a for loop over all training pairs from i = 1 to N, a gradient vector of the weights is determined according to: where is the cross-entropy loss function and where indicates the gradient operator in the weight domain. The thus obtained gradients are accumulated in the importance matrix : Here, the cross-entropy loss function is applied, since this is a classification problem. In alternative embodiments, other loss functions may be applied, for instance, a mean square errors loss function. Then based on the predefined selection ration p, the top p elements of the importance matrix with largest absolute value are determined. A mask is defined which identifies these top p elements of the importance matrix . According to an example implementation, mask may for example chosen to be the importance vector , but providing a 1 at each position that indicates a learnable weight (one of the top p elements of the importance matrix) and a zero at each position that identifies a frozen weight (all other weights that are not one of the top p elements of the importance matrix). In phase 2, following the direction of negative gradients , the process fine-tunes the weights, maximizing the log-likelihood function according to the well-known principles of back propagation. That is, in a for loop over all training pairs from i = 1 to N, the weights as obtained in iteration i are updated in a conventional way using the gradients . After this update, the frozen weights identified by mask are reset to their original values : Here, indicates a dot product in the weight domain and 1 indicates a vector with all elements set to one. In this way, only the learnable weights as identified by mask are updated, whereas the frozen weights remain unchanged. After having finished the loop over the training data, the algorithm returns a fine-tuned model with parameters that comprises the updated learnable weights as identified by mask and the frozen weights that remain unchanged. It should be noted that the process may return the fine-tuned model in the form of a SAFT update which defines set of SAFT weights that should be updated (52 in Fig.5 described below in more detail) and that is associated to the pre-trained model . The fine-tuned model can then be obtained based on the pre-trained model by replacing only the learnable weights with the weights defined by the SAFT update, as described in more detail with regard to Fig.5 below. That is, the training phase results in SAFT update data that is used for fine-tuning of the pre-trained model and that addresses the Out-of-distribution (OOD) generalization problem. Fig.4 schematically shows a graphic representation of a fine-tuned model obtained via the sparse update. In Fig.4, x identifies the input of the network layer, w identifies the weights of the network layer, and h identifies the output of the network layer. During sparse update, only the “learnable” weights of the image encoder as identified by the process of Fig.2 are updated. These learnable weights are identified in Fig.4 as black boxes. The remaining weights of the image encoder are “frozen”, i.e., they are not updated. These frozen weights are identified in Fig. 4 as gray boxes. It should be noted that although the embodiment provided above describes a vision language model, the principle of sparse update can be used on several application domains, especially for the adaptation of AI models to specific device models or specific target domains. Sparse update can be used to adapt any AI model – e.g., an ASR model for speech recognition (input: audio waveform, output: text) or an image recognition model for vision (input: image, output: detected object), or any other model as described in section “Out-of-distribution (OOD) generalization” above. Inference Phase The embodiments disclosed above relate to the training phase of the machine learning cycle. In the training phase, the model parameters are obtained based on a training data set. The training phase results in a pre-trained machine learning model. In a fine-tuning stage, SAFT update data is obtained that is used for fine-tuning of the pre-trained model. As described above, this updating addresses the Out-of-distribution (OOD) generalization problem. In the following, the inference phase is described. The inference phase is the second stage of the machine learning cycle, which involves applying a pre-trained machine learning model to new data to generate predictions or outputs. During the inference phase, the machine learning model is deployed into a production environment, making it possible to generate predictions for inputs provided by end-users. Fig.5 schematically shows a fine-tuned AI model obtained by the processes above in the inference phase. A pre-trained AI model 55, which is called here ‘foundation model’ is stored in a read only memory ROM of a device. The device obtains SAFT weights 52 that have been determined with ID training data in, e.g., a sparse update process as described in the embodiments above. SAFT weights 52 comprise only the learnable weights, but not the frozen weights. An updated AI model 56 is obtained by inserting the SAFT weights 52 into the pre- trained AI model 55, thereby replacing the original weights by respective updated weights. The frozen weights of the pre-trained AI model 55 remain unchanged in this update process. During inference, the updated AI model 56 operates on the input 53 to obtain respective output 54. That is, the machine learning model is used to make predictions based on new inputs. In the embodiment of Fig.5, a single SAFT update is obtained to fine-tune the pre-trained model. In alternative embodiments, a dynamic model fine-tuning can be realized by providing multiple sets of SAFT update data that have individually been tailored for specific use cases. In such alternative embodiments, several SAFT updates can be stored on one device which could be applied depending on the current use case, temperature, noise, or lighting condition. It should be noted that only the updated weights (e.g., 0.1%) need to be stored as SAFT weights 52, which implies a very small additional memory requirement for each alternative model. The sparse weight updates on which the adaptation process relies may for example be efficiently stored, minimizing resource requirements. In particular, the frozen weights that are not updated are not included in a sparse weights update. To implement SAFT in a memory-efficient way. Each weight tensor in the model may be associated with a key (e.g., a string name) indicating its position in the neural network. A dictionary data structure may be used to store the indices of learnable weights corresponding to that key. Fine-tuning is then performed on only weights that have their indices stored in the dictionary data structure. Providing the SAFT update for example allows to tailor an AI model to a particular device through "calibration" or to a specific usage scenario through "fine-tuning". This adaptation may result in an enhanced performance, contributing to an improved user experience. Here, "fine- tuning" pertains to adapting the model to a new domain or data distribution, while "calibration" involves adapting it to the original domain on which the AI model was initially trained, with a focus on accommodating the specific sensor characteristics (where the specific sensor characteristics are taken into account by either passing the data through the sensor or by using a good simulation of the specific sensor). As regards to "calibration", for example, cameras from different model series, as well as individual cameras from the same series, exhibit distinct sensor characteristics such as dead pixels and other variations (e.g., color variations). This variation poses a challenge as artificial intelligence models, which are typically trained on standardized datasets. However, standardized datasets may not be well-suited to the specific characteristics of a given camera. Consequently, the performance of these AI models can suffer. Merely updating the entire model is not an ideal solution, as it may lead to catastrophic forgetting issues, especially when dealing with factors like sensor temperature changes or aging or if the sensor is used in a new environment / application scenario. Such an update of the whole model would result in a degradation of performance over time. The methods described in the embodiments can, as an example of “calibration”, efficiently adapt AI models with minimal alterations to the original model. These methods may ensure that the adapted model can still generalize effectively, even in the face of camera-specific variations. As examples for specific usage scenario through "fine-tuning", for example, in image recognition a model trained on data from one dataset or task may need to generalize to data from other datasets or tasks, where the image quality, resolution, background, or object categories may differ. There exist many alternative examples of "fine-tuning" to a specific use case. For example, in medical diagnosis, a model trained on data from one hospital or region may need to generalize to data from other hospitals or regions, where the patient demographics, disease prevalence, imaging modalities, or labeling criteria may differ. Still further, in autonomous driving a model trained on data from one city or country may need to generalize to data from other cities or countries, where the road conditions, traffic rules, weather, or cultural norms may differ. Still further, in natural language understanding, a model trained on data from one domain or genre may need to generalize to data from other domains or genres, where the vocabulary, syntax, semantics, or pragmatics may differ. In some contexts, the device / service provider may want to update the model to a new calibrated or fine-tuned version to adapt to new situations. For example, chatbots based on LLM will need to be adapted to more up-to-date information. In the case the SAFT update process of the embodiments can be used. These SAFT updates could be very small since only a very small portion of the weights need to be updated. This avoids having to update all the weights in the model. Beyond the specific device or usage scenario, the SAFT approach ensures that performance is not compromised on other tasks or in the face of changes in sensor characteristics, such as those caused by temperature fluctuations or aging. Implementation Fig.6 illustrates a network including a server providing a SAFT update to an electronic device for updating an AI model based on sparse updating as described in Fig.5. Electronic device 100 (e.g., 100 of Fig.7) is connected to network 40 and via network 40 to server 41. Server 41 stores a SAFT update (e.g., SAFT weights 52, Fig.5) for fine-tuning a pre- trained AI model, i.e., a foundation model (e.g., 55, Fig.5). The SAFT update may be obtained based on a SAFT process as described in Figs.2 and 3. The electronic device 100 stores the foundation model (the pre-trained model), which is updated based on the SAFT update of the server 41. Alternatively, the server 41 also stores the foundation model. For updating the foundation model the server 41 provides the SAFT update to electronic device 100 via network 40. The server 41 can include a database of different SAFT updates for fine-tuning a pre-trained AI model based on different data, for example, based on different camera models, or based on different pre-trained AI models (e.g., models for speech recognition or models for image recognition). In this way, SAFT updating of pre-trained models can be implemented for multiple different purpose. The SAFT updates can also be shared between different electronic devices 100 via network 40. That is, an electronic device 100 which may implement the SAFT process as described in Figs.2 and 3 may transmit the SAFT update to server 41 via network 40 and the server 41 may then provide the SAFT update to another electronic device 100 for updating a pre- trained AI model stored in the other electronic device 100. Fig.7 illustrates an electronic device for implementing the SAFT process and sparse updating based on a SAFT update. Electronic device 100 may correspond to electronic device 100 of Fig.6. The electronic device 100 may be implemented as a terminal computer, a smartphone, a tablet or other mobile device, such as a laptop or a wearable device, such as smart glasses, a smart watch, headphones, earphones etc., as a vehicle or the like. The electronic device 100 includes a CPU 101 as processor. Additionally, or alternatively, other computation hardware, such as GPU, TPU, DSP etc. may be used. The electronic device 100 further includes camera(s) 206, microphone(s) 107 and loudspeaker(s) 108 that are connected to the processor 101. The processor 101 may for example implement any one or more of the features of the sparse updating process of Figs.2 and 3 and / or any one or more of the features of the inference process as described in Fig.5. The microphone 107 may be configured to receive any kind of audio signal and may for example be configured to generate data for fine-tuning (e.g., ) for the SAFT process as described in Fig.3 and / or to generate input (e.g., input 53, Fig.5) for the inference phase of the SAFT updated AI model (e.g., 56, Fig.5). The loudspeaker 108 may be configured to emit any kind of audio signal. The camera 106 may be one or more cameras, such as an RGB camera, and IR camera, a ToF camera, for example, an iToF or dTof, an event-based camera or the like. The camera 106 may be configured to generate data for fine-tuning (e.g., ) for the SAFT process as described in Fig.3 and / or to generate input (e.g., input 53, Fig.5) for the inference phase of the SAFT updated AI model (e.g.56, Fig.5). The connection between the processor 101 and the camera 106 may include a camera serial interface (CSI). The CSI is an interface between a camera 106 and a host processor 101. Thus, control signals and data from the processor 101 to the camera 106 as well as from the camera 106 to the processor 101 may be sent. The electronic device 100 further includes a user interface 109 that is connected to the processor 101. This user interface 109 acts as a man-machine interface and enables a dialogue between a user and the electronic device 100. For example, a user may make configurations to the system using this user interface 109. The electronic device 100 further includes a Bluetooth interface 104, and a WLAN interface 105. An ethernet interface may also be possible. These units 104, 105 act as I / O interfaces for data communication with external devices. For example, additional loudspeakers, microphones, and cameras, with WLAN or Bluetooth connection may be coupled to the processor 101 via these interfaces 104 and 105. The units 104, 105 may also be used for communicating with a server (e.g., 41, Fig.6) or other devices via a network (e.g.40, Fig.6a) as described in Fig.6. The units 104, 105 may also be used for offloading the computation from the electronic device 100 via the network (e.g., network 40 of Fig.6) to a server (e.g., 41, Fig.7) or to another device, such as a device close to the user. The electronic device 100 further includes a data storage 102 and a data memory 103 (here a RAM). The data memory 103 is arranged to temporarily store or cache data or computer instructions for processing by the processor 101, for example any of the processes as described in Figs.3 to 5. The data storage 102 is arranged as a long-term storage, e.g., for the pre-trained model (e.g., , Fig.3) for implementing the SAFT process, for the SAFT update data (e.g., SAFT weights 52, Fig.5), which may be obtained via the processor 101 or the AI processor 110 implementing the SAFT process, for the foundation AI model (e.g., 55, Fig.5) and / or for the updated AI model (e.g., 56, Fig.5) used during the inference phase. Furthermore, the electronic device 101 includes an artificial intelligence (AI) processor 110. The AI processor 110 may include a graphics processing unit (GPU) and / or a tensor processing unit 20 (TPU). The AI processor 110 may be configured to implement the SAFT process (see Figs.2 and 3), to generate the updated AI model based on the SAFT weights and the foundation model and / or to execute the updated AI model during the inference phase (see Fig.5). *** It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. Please note that the division of the control 100 into units is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, the control 100 could be implemented by a respective programmed processor, field programmable gate array (FPGA) and the like. A method for controlling an electronic device, such as mobile terminal 100 discussed above, is described in the following and under reference of Figs.6 and 7. The method can also be implemented as a computer program causing a computer and / or a processor, such as processor 100 discussed above, to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the method described to be performed. All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software. In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure. Note that the present technology can also be configured as described below. [1] An electronic device (100) comprising circuitry configured to update weights of a pre-trained AI model (51, ) based on a set of sparse modifications (52) to obtain an updated AI model (56, ), and to receive input data (53) which is used as input to the updated AI model (56, ). [2] The electronic device (100) of [1], wherein the update of the pre-trained AI model (51, ) with the set of sparse modifications (52) prevents the update from forgetting general knowledge in the pre-trained AI model (51, ). [3] The electronic device (100) of [1] or [2], wherein the set of sparse modifications (52) is configured to adapt the pre-trained AI model (51, ) with regard to out-of-distribution generalization. [4] The electronic device (100) of any one of [1] to [3], wherein the set of sparse modifications (52) is configured for fine-tuning the pre-trained AI model (51, ) to a new domain or data distribution. [5] The electronic device (100) of any one of [1] to [4], wherein the set of sparse modifications (52) is configured for calibration of the pre-trained AI model (51, ). [6] The electronic device (100) of any one of [1] to [5], wherein the pre-trained AI model (51, ) is a vision language model that comprises an image encoder ( ) and a text encoder ( ). [7] The electronic device (100) of [6], wherein the circuitry is configured to transform a class name (y) into a text prompt (t) describing the class, the text prompt (t) being used as input to the text encoder ( ). [8] The electronic device (100) of any one of [1] to [7], wherein the circuitry is configured to use the cross-entropy loss (11, ) for updating the pre-trained AI model (51, ). [9] The electronic device (100) of any one of [1] to [8], wherein the circuitry is configured to provide multiple sets of sparse modifications (52) that are individually tailored for specific use cases.

[0010] The electronic device (100) of any one of [1] to [9], wherein the circuitry is configured to determine the updated AI model (56, ) by inserting the set of sparse modifications (52) into the pre-trained AI model (51, ), thereby replacing respective original parameters of the pre-trained AI model (51, ).

[0011] The electronic device (100) of any one of [1] to

[0010] , wherein the circuitry is configured to apply the updated AI model (56, ) to the input data (53) to generate predictions or outputs (54).

[0012] The electronic device (100) of

[0011] , wherein the updated AI model (56, ) is a neural network and the circuitry is configured to pass the input data (53) to the first layer of the neural network.

[0013] The electronic device (100) any one of [1] to

[0012] , wherein the receiving of the input data (53) is part of an inference phase.

[0014] The electronic device (100) of any one of [1] to

[0013] , wherein the updated AI model (56, ) provides an output to be used in a given task.

[0015] The electronic device (100) of

[0014] , wherein the output of the updated AI model (56, ) is used for controlling a setting of an electronic device, for recognizing speech, recognizing images, or recognizing gestures.

[0016] The electronic device (100) of any one of [1] to

[0015] , wherein the set of sparse modifications (52) is received from an input, from a service provider, over a network, or via a data file.

[0017] The electronic device (100) of any one of [1] to

[0016] , wherein the circuitry is configured to select the set of sparse modifications (52) from multiple sets of sparse modifications (52).

[0018] The electronic device (100) of

[0017] , wherein the selection is based on a user input, or on a particular task to be achieved.

[0019] The electronic device (100) of any one of [1] to

[0018] , wherein the circuitry is configured to detect context information and to select the set of sparse modifications (52) based on the context information.

[0020] A computer-implemented method comprising updating weights of a pre-trained AI model (51, ) based on a set of sparse modifications (52) to obtain an updated AI model (56, ), and to receive input data (53) which is used as input to the updated AI model (56, ).

[0021] A computer-implemented method of training an AI model, the method comprising determining, based on a pre-trained AI model (51, ) a set of sparse modifications (52).

[0022] The method of

[0014] , wherein the set of sparse modifications (52) comprises learnable weights.

[0023] The method of

[0014] or

[0015] , wherein determining the set of sparse modifications (52) is based on training data (PID).

[0024] The method of

[0016] , wherein the training data (PID) is configured for fine-tuning the pre- trained AI model (51, ) to a new domain or data distribution.

[0025] The method of

[0016] , wherein the training data (PID) is configured for calibration of the pre-trained AI model (51, ).

[0026] The method of any one of

[0014] to

[0018] , the method comprising determining, based on the pre-trained AI model (51, ) and based on the set of sparse modifications (52), an updated AI model (56, ).

[0027] The method of any one of

[0014] to

[0019] , the method comprising determining the set of sparse modifications (52) with training data (PID) in a sparse update.

[0028] The method of

[0020] , further comprising updating, during the sparse update, only a subset of weights of the pre-trained AI model (51, ).

[0029] The method of any one of

[0014] to

[0028] , the method comprising defining a maximum size of the set of sparse modifications (52).

[0030] The method of any one of

[0014] to

[0029] , the method comprising identifying the weights that have the greatest influence on the model's performance on training data (PID).

[0031] The method of any one of

[0014] to

[0030] , further comprising determining a selection parameter (p) which directly or indirectly defines the amount of data in set of sparse modifications (52).

[0032] The method of any one of

[0014] to

[0031] , the method comprising determining a mask ( ) which is configured to identify weights of the pre-trained AI model (51, ) as learnable weights and to identify other weights of the pre-trained AI model (51, ) as frozen weights.

[0033] The method of any one of

[0014] to

[0032] , the method comprising including into the set of sparse modifications (52) only learnable weights, whereas frozen weights are not included.

[0034] The method of any one of

[0014] to

[0033] , the method comprising resetting frozen weights to their original values ( ).

[0035] The method of any one of

[0014] to

[0034] , the method comprising updating only a subset of weights whose gradient magnitude is large.

[0036] The method of any one of

[0014] to

[0035] , the method comprising computing the gradient of a likelihood function of the pre-trained AI model (51, ) with respect to the parameters ( ) of the model.

[0037] The method of any one of

[0014] to

[0036] , the method comprising averaging the gradient of the likelihood function over ID training data (PID).

[0038] The method of any one of

[0014] to

[0037] , the method comprising keeping, during sparse update, a first encoder ( ) frozen while only a second encoder ( ) is updated.

[0039] The method of

[0038] , wherein the first encoder ( ) that is kept frozen is a text encoder and the second encoder ( ) that is updated is an image encoder.

[0040] The method of any one of

[0014] to

[0039] , comprising returning a fine-tuned model ( ) with parameters ( ) that comprise the set of sparse modifications (52).

[0041] The method of any one of

[0014] to

[0040] , comprising returning the set of sparse modifications (52).

[0042] The method of any one of

[0014] to

[0041] , comprising storing the set of sparse modifications (52).

[0043] An electronic device comprising circuitry configured to implement the method of any one of

[0014] to

[0042] .

[0044] A computer program comprising program code causing a computer to perform the method according to anyone of

[0014] to

[0042] , when being carried out on a computer.

[0045] A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of

[0014] to

[0042] to be performed. REFERENCES [1] Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distributions. arXiv preprint arXiv:2302.00864, 2023. [2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.

Claims

CLAIMS 1. An electronic device comprising circuitry configured to update weights of a pre-trained AI model based on a set of sparse modifications to obtain an updated AI model, and to receive input data which is used as input to the updated AI model.

2. The electronic device of claim 1, wherein the update of the pre-trained AI model with the set of sparse modifications prevents the update from forgetting general knowledge in the pre- trained AI model.

3. The electronic device of claim 1, wherein the set of sparse modifications is configured to adapt the pre-trained AI model with regard to out-of-distribution generalization.

4. The electronic device of claim 1, wherein the set of sparse modifications is configured for fine-tuning the pre-trained AI model to a new domain or data distribution.

5. The electronic device of claim 1, wherein the set of sparse modifications is configured for calibration of the pre-trained AI model.

6. The electronic device of claim 5, wherein the pre-trained AI model is a vision language model that comprises an image encoder and a text encoder.

7. The electronic device of claim 6, wherein the circuitry is configured to transform a class name into a text prompt describing the class, the text prompt being used as input to the text encoder.

8. The electronic device of claim 1, wherein the circuitry is configured to use the cross- entropy loss for updating the pre-trained AI model.

9. The electronic device of claim 1, wherein the circuitry is configured to provide multiple sets of sparse modifications that are individually tailored for specific use cases.

10. The electronic device of claim 1, wherein the circuitry is configured to determine the updated AI model by inserting the set of sparse modifications into the pre-trained AI model, thereby replacing respective original parameters of the pre-trained AI model.

11. The electronic device of claim 1, wherein the circuitry is configured to apply the updated AI model to the input data to generate predictions or outputs 12. The electronic device of claim 11, wherein the updated AI model is a neural network and the circuitry is configured to pass the input data to the first layer of the neural network.

13. The electronic device of claim 1, wherein the receiving of the input data is part of an inference phase.

14. The electronic device of claim 1, wherein the updated AI model provides an output to be used in a given task.

15. The electronic device of claim 14, wherein the output of the updated AI model is used for controlling a setting of an electronic device, for recognizing speech, recognizing images, or recognizing gestures.

16. The electronic device of claim 1, wherein the set of sparse modifications is received from an input, from a service provider, over a network, or via a data file.

17. The electronic device of claims 1, wherein the circuitry is configured to select the set of sparse modifications from multiple sets of sparse modifications.

18. The electronic device of claim 17, wherein the selection is based on a user input, or on a particular task to be achieved.

19. The electronic device of claim 17, wherein the circuitry is configured to detect context information and to select the set of sparse modifications based on the context information.

20. A computer-implemented method comprising updating weights of a pre-trained AI model based on a set of sparse modifications to obtain an updated AI model, and to receive input data which is used as input to the updated AI model.

21. A computer-implemented method of training an AI model, the method comprising determining, based on a pre-trained AI model a set of sparse modifications.

22. The method of claim 21, wherein the set of sparse modifications comprises learnable weights.

23. The method of claim 21, wherein determining the set of sparse modifications is based training data.

24. The method of claim 23, wherein the training data is configured for fine-tuning the pre- trained AI model to a new domain or data distribution.

25. The method of claim 23, wherein the training data is configured for calibration of the pre- trained AI model.

26. The method of claim 23, the method comprising determining, based on the pre-trained AI model and based on the set of sparse modifications, an updated AI model.

27. The method of claim 23, the method comprising determining the set of sparse modifications with training data in a sparse update.

28. The method of claim 27, further comprising updating, during the sparse update, only a subset of weights of the pre-trained AI model.

29. The method of claim 23, the method comprising defining a maximum size of the set of sparse modifications.

30. The method of claim 23, the method comprising identifying the weights that have the greatest influence on the model's performance on training data..

31. The method of claim 23, further comprising determining a selection parameter which directly or indirectly defines the amount of data in set of sparse modifications.

32. The method of claim 23, the method comprising determining a mask which is configured to identify weights of the pre-trained AI model as learnable weights and to identify other weights of the pre-trained AI model as frozen weights.

33. The method of claim 23, the method comprising including into the set of sparse modifications only learnable weights, whereas frozen weights are not included.

34. The method of claim 23, the method comprising resetting frozen weights to their original values.

35. The method of claim 23, the method comprising updating only a subset of weights whose gradient magnitude is large.

36. The method of claim 23, the method comprising computing the gradient of a likelihood function of the pre-trained AI model with respect to the parameters of the model.

37. The method of claim 23, the method comprising averaging the gradient of the likelihood function over training data.

38. The method of claim 23, the method comprising keeping, during sparse update, a first encoder frozen while only a second encoder is updated.

39. The method of claim 38, wherein the first encoder that is kept frozen is a text encoder and the second encoder that is updated is an image encoder.

40. The method of claim 23, comprising returning a fine-tuned model with parameters that comprise the set of sparse modifications.

41. The method of claim 23, comprising returning the set of sparse modifications.

42. The method of claim 23, comprising storing the set of sparse modifications.

43. An electronic device comprising circuitry configured to implement the method of claim .

Images