A computer-implemented method for adapting a machine learning model of an advanced driver assistant system to an updated set of tasks

Abstract

Method of adapting a multi-task machine learning model of an advanced driver assistance system (ADAS) 16 to an updated set of tasks, comprising: providing a pre-trained machine learning model defined

Description

TECHNICAL FIELD The invention relates to a method for adapting a machine learning model of an advanced driver assistant system (ADAS). BACKGROUND In the field of machine learning (ML) model development for products typically requires extensive and iterative development processes based on large scale datasets. In addition to maintaining these datasets, e.g., by labelling, augmenting, and collecting data, especially for out-of-distribution edge cases, is not only timeconsuming human work, but also requires a substantial amount of computational resource, among them long term storage, RAM, and CPU time. S. Chen, et al., “Multi-Task Learning in Natural Language Processing: An Overview”, arXiv:2109.09138, discloses different multi-task learning concepts in the field of natural language processing. With the development of deep learning, the most common way to solve multiple tasks is through a hard-sharing parameter architecture. The multi-task learning model incorporates a shared backbone and a number of separate task-specific heads. The shared backbone typically extracts a universal representation that represents a compact version of features that can be used by different tasks. This approach typically necessitates accessing the training dataset of all tasks and requires a complete retraining of the model, whenever something is changed on the task specific heads, e.g., changing, removing, or adding a task. J. Fuest and M. Tacke, “Low Rank Adaptation for Multitask BERT”, available at https: / / web.stanford.edu / class / archive / cs / cs224n / cs224n.1234 / final-reports / final report-169568390.pdf, discloses a method for fine-tuning large language models for simultaneous better performance on different downstream tasks. With this approach the significant need for computational resources in fine-tuning pretrained large language models can be reduced. Additional reference is made to the following documents: [1] A. Dosovitskiy, et al. An image is worth 16x16 words: Transformers for image recognition at scale, 2021, https: / / doi.org / 10.48550 / arXiv.2010.11929. [2] E. J. Hu, et al. Lora: Low-rank adaptation of large language models, 2021, https: / / doi.org / 10.48550 / arXiv.2106.09685. [3] G. Ilharco, et al. Editing models with task arithmetic. arXiv:2212.04089, 2022. [4] X. Jin, et al. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849, 2022. [5] C. Li, et al. Measuring the intrinsic dimension of objective landscapes, 2018, https: / / doi.Org / 10.48550 / arXiv. 1804.08838. [6] W. Maddox, et al. A simple baseline for bayesian uncertainty in deep learning, 2019, https: / / doi.org / 10.48550 / arXiv.1902.02476. [7] M. Matena and C. Raffel. Merging models with fisher-weighted averaging, 2022, https: / / doi.org / 10.48550 / arXiv.2111.09832. [8] S.-A. Rebuffi, et al. Learning multiple visual domains with residual adapters, 2017, https: / / d0i.0rg / l 0.48550 / arXiv. 1705.08045. [9] P. Yadav, et al. Resolving interference when merging models. arXiv:2306.01708, SUMMARY OF THE INVENTION It is the object of the invention to provide measures that allow for a more computationally resource-efficient training and / or deployment of multi-task ML models. The invention provides a computer-implemented method for adapting a machine learning model of an ADAS to an updated set of tasks, the ADAS having a multi-task machine learning model to be adapted, the method comprising: a) providing a pre-trained machine learning model that is defined by a pre-trained weight matrix; b) for each task that is to be added to or modified in the updated set of tasks: generating a task specific adaptive weight matrix from a task specific training dataset by backpropagation, wherein the task specific adaptive weight matrix is formed by a matrix multiplication of a first LoRA matrix and a second LoRA matrix, wherein eachLoRA matrix has a lower rank than the pre-trained weight matrix; c) merging the task specific adaptive weight matrices to obtain a merged adaptive weight matrix; and d) adding the merged adaptive weight matrix to the pre-trained weight matrix to obtain a fully-trained weight matrix that defines an adapted machine learning model that is adapted to the updated set of tasks. In contrast to previous multi-task approaches, this idea adopts a different and generally more flexible method to tackle multitask learning. Typically, multi-task machine learning models are comparatively large, as a shared backbone and taskspecific heads are provided. With the disclosed approach both the design / development efforts and training computational costs of deep multitask learning models can be reduced. The idea is based on the insight that by merging single-task learning models in the parameter space it is possible to avoid an increment of the deployment costs. To reduce the training computations of the singletask learning models required, it is proposed to fine-tune pre-trained models using parameter-efficient fine-tuning (PEFT) via low-rank adaptation (LoRA), and then merge the low-rank adaptations. In addition to the general LoRA approach it is possible to merge the full models when there are already existing pre-developed single-task models. Preferably, the pre-trained machine learning model is pre-trained for performing computer vision tasks. In general computer vision tasks are very demanding. Preferably, in step b) the task is chosen from a group of computer vision tasks that consists of semantic segmentation, object detection, object classification, and image classification. In general computer vision tasks are computationally demanding, so an application of the ideas presented herein allows for a significant improvement regarding resource efficiency. Furthermore, this allows to adapt to different computer vision tasks more flexibly. Preferably, in step b) generating the task specific adaptive weight matrix involves a maximum likelihood estimation (MLE) followed by a stochastic gradient descent based on a stochastic weight averaging gaussian (SWAG) process. Preferably, in step b) the learning rate is kept constant. With a combination of MLE and SWAG a resource efficient way of learning is implemented. In contrast to conventional backpropagation the training time can be reduced. Preferably, in step b) each task specific adaptive weight matrix is stored as a checkpoint for further adaption. With this approach it is easier to remove or modify tasks. E.g. if a task is to be removed, a new adapted model can be determined from the task specific adaptive weight matrices that are still relevant. Furthermore, it is possible to further train specific tasks without starting the training from scratch. Preferably, in step c) the merged adaptive weight matrix is determined by the average of the task specific adaptive weight matrix. This is a resource efficient way of merging the task specific matrices and adapting the overall model. This also allows merging of full models. Preferably, in step c) the merged adaptive weight matrix is determined by stacking the first LoRA matrices of each task in a column vector and the second LoRA matrices of each task in a row vector and performing matrix multiplication of the column vector with the row vector. This approach is particularly suited for GPU hardware that is specialized in performing these kinds of multiplications. Furthermore, it is possible to merge full models. Preferably, the merged adaptive weight matrix is determined by performing an evolutionary based method. Preferably, the evolutionary based method involves generating a validation dataset from the task specific training datasets, wherein the validation dataset includes random samples from each task specific training dataset, and the validation dataset is smaller than or equal to the smallest task specific dataset. Preferably, the evolutionary based method involves generating a parent population of a predetermined size by determining a first randomly weighted sum of the first LoRA matrices of the tasks and a second randomly weighted sum of the second LoRA matrices. Preferably, the evolutionary based method involves generating an offspring population of the same size as the parent population. Preferably, the evolutionary based method involves generating candidate fully-trained weight matrices from the parent and offspring populations. Preferably, the evolutionary based method evaluating the candidate fully-trained weight matrices with the validation dataset and ranking the candidate fully-trained weight matrices by a performance criterion. Preferably, the evolutionary based method is repeated until a stopping criterion is met, and selecting the candidate fully-trained weight matrix with the best performance criterion as the fully-trained weight matrix. With an evolutionary based approach, the optimization of the task specific matrices can be performed with little effort. The invention provides an advanced driver assistant system (ADAS) for a vehicle, the ADAS comprising a machine learning model trained by a preferred method and a control unit that is configured to generate a control signal to control the vehicle based on the output of the machine learning model. The machine learning model can be flexibly adapted using the method described herein. Furthermore, as the model can be shrunk down in size, it is more suitable for being executed in an embedded system with limited computational resources at inference time. The invention provides a method for controlling a vehicle, the vehicle comprising an ADAS, the method comprising detecting sensor data with at least one sensor; feeding the sensor data to the ADAS to obtain a control signal; and controlling the vehicle according to the control signal. The invention provides a vehicle comprising means to carry out the controlling method. The ADAS may control the vehicle to perform an emergency brake procedure, or indicate something to the driver, e.g., obstacles, pedestrians, other vehicles, etc. The main idea is to make machine learning models usable in an embedded setting with limited computational resources, especially at inference time. They can be used in automated / autonomous agents or driving-assist systems for braking / parking, for example. A particular focus is the field of computer vision, e.g., for front-facing cameras and next generation camera-based computer vision applications. It is also possible to involve further systems, such as lidar / radar sensor systems and video sensor systems (incl. 3D Camera). The implementation can be used as a software module which is able to merge multiple single-task models together to achieve efficient computation and improved performance via multitask learning. It can also be used to reduce the computational costs on the edge devices. The software module can be used to reduce the development cycle of ADAS camera products, as well as optimise model deployments with limited computations. BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention are described in more detail with reference to the accompanying schematic drawings. Fig. 1 depicts an embodiment of a vehicle; Fig. 2 to Fig. 4 illustrate experimental results. DETAILED DESCRIPTION OF EMBODIMENT Referring to Fig. 1, a vehicle 10 comprises a vehicle cabin 12. The vehicle 10 is monitored and / or controlled by an operator 14. The vehicle 10 further comprises an advanced driver assistant system (ADAS) 16 for monitoring and controlling operation of the vehicle 10. The ADAS 16 comprises a control unit 18 that is configured to generate control signals for controlling the vehicle 10 based on different inputs, such as foot pedals, control knobs or buttons, touch input devices or other human machine interfaces in general. The control apparatus 18 is further configured to generate control signals based on sensor inputs. The ADAS 16 comprises at least one sensor 20 that is arranged to detect sensor data about a portion of an environment 22 around the vehicle 10. Each sensor 20 has a predetermined detection area 24 associated with it. The respective sensor 20 can monitor the respective detection area 24 with which it is associated. The sensors 20 may be chosen among imaging sensors 26, LIDAR sensors 28, RADAR sensors and / or ultrasonic sensors 30. The imaging sensor 26 generates image data. Depending on the type of imaging sensor 26 the image data vary. The imaging sensor 26 may be a simple monochrome or IR camera. In this case the image data have two dimensions, namely height and width. In some embodiments the imaging sensor 26 is an RGB camera. In this case the image data have three dimensions, namely height and width and three colour channels. In still further embodiments the imaging sensor 26 is a 3D camera. In this case the image data have four dimensions, as in addition to the colour channels the image data also include a depth map that is indicative of a distance estimate. The types of image sensor 20 can be combined to augment the image data, e.g., by installing two or more of a different type or by integrating their functionalities. One imaging sensor 26 can be arranged as a front-facing camera 32. The frontfacing camera 32 has its detection area 24 defined such that it in essence corresponds to the point of view that the operator 14 has, when looking through the windshield. The typical extent of the detection area 24 for the front-facing camera 32 is about 15 m to 25 m. Further imaging sensors 26 can be arranged as side-facing cameras 34 and / or a rear-facing camera 36. This configuration allows a surround view of the vehicle 10. In addition, the rear-facing camera 36 can be used in assisting parking. LIDAR sensors 28 and RADAR sensors generate distance data. The distance data are typically scalars, but depending on the sensor may also involve time information. In this case, the distance data can be a vector where each entry represents a point in time. LIDAR sensors 28 and / or RADAR sensors that are front-facing typically have a larger detection area 24 that extends beyond the detection areas 24 for image detection, e.g., up to 100 m sometimes beyond. Ultrasonic sensors 30 generate distance data that are substantially identical to the distance data by the LIDAR / RADAR sensors. Ultrasonic sensors 30 are generally used in the direct vicinity of the vehicle 10, e.g., maximum up to 2 m or 3 m. The ADAS 16, preferably the control unit 18, contains a fully trained multi-task machine learning (ML) model 38. The multi-task ML model 38 is configured for at least two different computer vision tasks, for example. A first task may be semantic segmentation of the image data. Further tasks may be based on object recognition, preferably on object detection, for example, regarding obstacles (such as obstruction, pedestrians, cyclists, other vehicles), traffic signs, road markings, etc. Preferably, each of these object detection tasks is developed as a separate task. The multi-task ML model 38 is taken from background document [1], which is incorporated herein by reference for the purposes of disclosure. Specific reference is made to document [1], figure 1 and sections 3 and 4 titled “Method” and “Experiments”, respectively, as well as appendix A. The multi-task ML model 38 is pre-trained and gets fully trained as subsequently described. In the following, a training method is described that adapts the multi-task ML model 38 to a new set of tasks, by removing, changing, or adding a task. The functionality and with that the different tasks performed by the multi-task ML model 38 is defined by the weights of the multi-task ML model 38 that are organized in a weight matrix W. The multi-task ML model 38 is obtained with a pre-trained weight matrix WPre-trained that needs to be fine-tuned to adapt the multi-task ML model 38 to an updated set of tasks. The pre-trained weight matrix WPre-trained is an element of ]Rdxd, i.e., it has d-by-d entries. During training the pre-trained weight matrix WPre-trained is kept separate and frozen from an adaptive weight matrix bW. The adaptive weight matrix bW gets updated during training. The adaptive weight matrix bW is typically also an element of Rdxd. The adaptive weight matrix bW is decomposed into two LoRA matrices Wa and Wb of lower rank r <d than the weight matrix bW. In other words, bW = Wb x Wa, where x designates the matrix product. In case of the ViT-base-16 of document [1], the full rank of the pre-trained weight matrix Wpre-trained is 768. The rank of the LoRA matrices Wa and Wb is chosen to be r = 16. Preferably, r is chosen in a range of the full rank divided by 23 to 25, i.e., 8 to 32. The LoRA matrix Wb is an element of IRdxr and the LoRA matrix Wa is an element of ]Rrxd, respectively. The LoRA matrices Wa and Wb are stored and updated during training. In this example, this means that instead of more than 500,000 entries for the full adaptive weight matrix bW, only about 25,000 entries for the pair of LoRA matrices Wa and Wb need to be stored. As the computation of the full adaptive weight matrix bW can be done on efficient hardware, such as graphics processing units (GPUs) or tensor processing units (TPUs), the small cost of GPU / TPU time is considerably outweighed by significantly decreasing the required memory by a factor of about 20. This can also avoid caching and other memory management operations, which further allows increase of computational efficiency and speed. Beyond reducing the number of parameters during training, faster convergence can be achieved, which allows for a significant decrease in training time. This timesaving aspect can be quite significant (approximately one and a half times faster compared to the traditional fine-tuning approach). Fine-tuning for an additional task k is initialized with a labelled training data set Xk, Yk, where Xk denotes the input data and Yk the labels. The pretrained weights are organized in the pre-trained weight matrix WPre-trained. The LoRA matrices Wa and Wb are added as a low rank perturbation to each linear layer in the multi-task ML model 38. After achieving a maximum likelihood estimation (MLE) for the adaptive weight matrix bW, an approximate posterior estimation for the parameter space is constructed by running a stochastic weight averaging gaussian (SWAG) method as described in document [6], The learning rate is kept constant at 0.1. While traversing through the loss landscape, the SWAG method determines the variance of the parameters T> / c held in the adaptive weight matrix bW. The approximate posterior distribution for a subset of the parameters is modelled as a normal distribution ^diagonal + ^low rank), Wherein ^k = is the sum z Z ~ lvi 2 ............................... 2 of M condition parameters, Zdiagonai = ^=0 diag^k) - (a**) ) and ___ ___ 7 %iow rank = - ¢^) ■ At this stage the pre-trained ML model is finished to be adapted to the new task k. The closed formed posterior distribution of the model for task k is stored as a checkpoint. With this approach the parameter space of the multi-task ML model 38 can be better aligned. The merging utilises the approximate posterior distribution to find the merged set of parameters that maximize the probability of all posterior distributions. The previous fine-tuning steps are repeated for each task k that is added to or changed in the multi-task ML model 38. Subsequently the task specific adaptive weight matrices bW or rather the LoRA matrices Wa and Wb are merged by means of one of three processes. In a first process, the LoRA matrices WAk and Wsk for each task k are matrix multiplied such that bWk = Wek x WAk and the adaptive weight matrix bW is obtained by averaging the task specific adaptive weight matrices bW\ The fully-trained weight matrix Wfuii is obtained by adding the adaptive weight matrix bW to the pre-trained weight matrix WPre-trained, i.e., Wfuii = Wpre-trained + bW, where the fully-trained weight matrix Wfuii represents the multi-task ML model 38. In a second process, first LoRA matrices WBk for each task k are stacked in a column vector and second LoRA matrices WAk are stacked in a row vector. The adaptive weight matrix 5W is obtained by matrix multiplication of the column vector with the row vector: x[< W2 ... WA] The result is added to the pre-trained weight matrix Wpre-trained to obtain the fully-trained multi-task ML model 38. In a third process, an evolutionary algorithm is used. While this approach is described with reference to two tasks, it should be noted that this approach also is applicable for any other number of tasks. Initially a small validation dataset is generated that contains a small portion of data used for training all tasks which are going to be merged. Random sampling from a gaussian standard distribution is used to obtain random weights. LoRA matrix Wa1 is added to LoRA matrix Wa2 after both have been multiplied with their own random weights to create a new merged LoRA matrix Wap. The same process is then applied to LoRA matrices Wb1 and Wb2 to obtain merged LoRA matrix Wbp. These steps are repeated for a predetermined amount of times n to obtain a parent population Pt of LoRA matrices. An offspring population Qt of the same size n is generated from the parent population Pt by using crossover and mutation operations on the parent population Pt. The crossover operation is performed between a pair of parents from the parent population Pt by randomly selecting one crossover point and swapping the parent model parameters to create a new model that is a member of the offspring population Qt. The mutation operation is performed by changing each offspring model parameter to the original corresponding LoRA model parameter from either Wa1, Wb1 for task k=1, or Wa2, Wb2 for task k=2 with a small probability, e.g., 1 %. The parent population Pt and the offspring population Qt are combined into a candidate population Rt of size 2n. Candidate models Wr' are generated in the usual manner by adding to the pre-trained weight matrix WPre-trained according to Wr' = WPre-trained + bW, where = Wb' x Wa'. The candidate models Wr' are evaluated using the validation dataset. The performance can be measured in any of mean square error, mean absolute error, accuracy, intersection-over-union. Accuracy is the preferred benchmark. The candidate models Wr' are ranked according to their performance. After ranking a new parent population Pt+i is generated by selecting the top n candidates and repeat the process this parent population Pt+i again. The process is repeated until a certain performance is achieved or a predetermined number of iterations have been performed. After the final iteration, the top ranked candidate model Wr' is selected for deployment and use at inference time. Experiments with an accuracy metric to measure the performance of the various methods are performed on the Visual Domain Decathlon of background document [8]. The Visual Domain Decathlon contains 10 different domains for an image classification task. All the classification accuracy is reported in %. The classification accuracy of 10 domains (tasks) has been reported in Fig. 2. The column “‘Overall’” demonstrates the average accuracy of all domains. The method ‘“MTL fine-tune’” indicates that the pretrained model is fined-tuned with all data from all domains mixed together, same as the multitask learning setup. The method ‘“Ind LoRA” is trained and evaluated individually in each domain using low-rank adaptation with the rank r = 16, which gives the upper bound performance. The ‘“LoRA-avg”’ and ‘“LoRA-stack”’ are two proposed methods in this invention. We can see that without any extra fine-tuning and training dataset, using only the low-rank parameters, ‘“LoRA-stack”” achieves decent performance with only level than 5% drop in performance compared with ‘“MTL fine-tune”. As evident from the experiments, the pre-trained model demonstrates exceptional generalization across the majority of tasks. As a result, its performance is already comparable to finetuning the model for each domain. However, we have observed two domains, namely AIRCRAFT and UCF101, where a significant performance gap exists between the pre-trained model and the finetuned one. This implies that the rank of lora is large, increasing the likelihood of encountering contradictions during merging. That in turn can diminish the model’s performance after merging in parameter space. Further results of experiments on these two difficult domains using our proposed methods are shown in Fig. 3. While there is a larger performance gap between the fine-tuned method and the merging method, our proposed method still outperforms Fisher [7] with more computations and stored information. Further experiment by merging two lora trained models on different domains. There are 10 domains in total so the number of combinations is 90. The LoRA-stack results are shown in Fig. 4. It is evident that the approach disclosed herein outperforms Fisher merging in most cases, even though Fisher merging has used additional data other than the parameters themselves. REFERENCE SIGNS vehicle vehicle cabin operator advanced driver assistant system (ADAS) control unit sensor environment detection area imaging sensor LIDAR sensor ultrasonic sensor front-facing camera side-facing camera rear-facing camera multi-task machine learning (ML) model

Claims

1. A computer-implemented method for adapting a machine learning model of an ADAS (16) to an updated set of tasks, the ADAS (16) having a multi-task machine learning model (38) to be adapted, the method comprising:a) providing a pre-trained machine learning model that is defined by a pre-trained weight matrix;b) for each task that is to be added to or modified in the updated set of tasks: generating a task specific adaptive weight matrix from a task specific training dataset by backpropagation, wherein the task specific adaptive weight matrix is formed by a matrix multiplication of a first LoRA matrix and a second LoRA matrix, wherein each LoRA matrix has a lower rank than the pre-trained weight matrix;c) merging the task specific adaptive weight matrices to obtain a merged adaptive weight matrix; andd) adding the merged adaptive weight matrix to the pre-trained weight matrix to obtain a fully-trained weight matrix that defines an adapted machine learning model that is adapted to the updated set of tasks.

2. The method of claim 1, wherein the pre-trained machine learning model is pre-trained for performing computer vision tasks, and in step b) the task is chosen from a group of computer vision tasks that consists of semantic segmentation, object detection, object classification, and image classification.

3. The method of any of the preceding claims, wherein in step b) generating the task specific adaptive weight matrix involves a maximum likelihood estimation followed by a stochastic gradient descent based on a stochastic weight averaging gaussian process.

4. The method of any of the preceding claims, wherein in step b) the learning rate is kept constant.

5. The method of any of the preceding claims, wherein in step b) each task specific adaptive weight matrix is stored as a checkpoint for further adaption.

6. The method of any of the preceding claims, wherein in step c) the merged adaptive weight matrix is determined by the average of the task specific adaptive weight matrix.

7. The method of any of the preceding claims 1 to 5, wherein in step c) the merged adaptive weight matrix is determined by stacking the first LoRA matrices of each task in a column vector and the second LoRA matrices of each task in a row vector and performing matrix multiplication of the column vector with the row vector.

8. The method of any of the preceding claims 1 to 5, wherein the merged adaptive weight matrix is determined by performing an evolutionary based method.

9. The method of claim 8, wherein the evolutionary based method involves generating a validation dataset from the task specific training datasets, wherein the validation dataset includes random samples from each task specific training dataset, and the validation dataset is smaller than or equal to the smallest task specific dataset.

10. The method of claim 9, wherein the evolutionary based method involves generating a parent population of a predetermined size by determining a first randomly weighted sum of the first LoRA matrices of the tasks and a second randomly weighted sum of the second LoRA matrices;wherein the evolutionary based method involves generating an offspring population of the same size as the parent population;generating candidate fully-trained weight matrices from the parent and offspring populations;evaluating the candidate fully-trained weight matrices with the validation dataset and ranking the candidate fully-trained weight matrices by a performance criterion.

11. The method of claim 10, wherein the evolutionary based method is repeated until a stopping criterion is met, and selecting the candidate fully-trained weight matrix with the best performance criterion as the fully-trained weight matrix.

12. An ADAS (16) for a vehicle (10), the ADAS (16) comprising a machine learning model trained by a method of any of the preceding claims and a control unit (18) that is configured to generate a control signal to control the vehicle (10) based on the output of the machine learning model.

13. A method for controlling a vehicle (10), the vehicle (10) comprising an ADAS (16) of claim 12, the method comprising detecting sensor data with at least one sensor (20); feeding the sensor data to the ADAS (16) to obtain a control signal; and controlling the vehicle (10) according to the control signal.

14. A vehicle (10) comprising means to carry out the method of claim 13.

Images