SYSTEM AND COMPUTER-IMPLEMENTED METHOD FOR DYNAMIC ADJUSTMENT OF MOTOR VEHICLE FUNCTIONS

Description

[0001] The present invention relates to a system and computer-implemented method for improving the efficiency and security in dynamic adaption of automotive functions.

[0002] In modern vehicular environments, sensor date, image data, audio data etc. from an environment of a vehicle and a vehicle cabin are accessed and processed in order to provide intelligent vehicle functions and to enhance data management of the vehicle. In order to efficiently and reliably process the above-mentioned data, huge amounts of a plurality of vehicles need to be processed. Such an environment of data generated by a plurality of vehicles that comprise context-aware, real-time systems in the sense of ubiquitous computing, can be summarized as real-time distributed system. The transmission of the generated data towards the Backend for generating intelligent vehicle functions is even with the use of new technologies in mobile networks (e.g. 5G) costly and requires enormous storage and processing power. Therefore in-vehicle processing is required and efficient transferring of data needed. Moreover, due to the increase of in-vehicle applications and due to the fact that data accessed comprises sensitive, personal data, there is a need to perform data computation, data analytics and machine-learning near the data source - in the sense of Edge Computing - rather than by a Backend - in the sense of Cloud Computing. Each vehicle can be considered to represent an edge device in the sense of Edge Computing. In order to preserve data privacy, it is known to apply Federated Learning.

[0003] From the article "A Transaction Classification Model of Federated Learning" [Ahmed Usman et.al.; Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK; ISBN: 978-3-030-58594-5] a Federated Learning is illustrated by the example of a server-client structure for shops (= local clients) in a city for modelling a supply and demand analysis. Each client is provided with both a generalized model and a local own-iterative model for a training phase, and the client then ultimately makes a true choice for one of the models.

[0004] Federated Learning has been introduced within the context of Edge Computing, aiming to solve edge learning under privacy sensitive, non-independent and identically distributed (non-i.i.d.), and unbalanced data generated from massively distributed devices using the technique of machine learning. For example, from the article "Real-time End-to-End Federated Learning" [Hongyi Zhang et.al.; 45th Annual Computers, Software and Applications Conference (COMPSAC)] one gets taught how to improve Federated Learning. Starting from the Machine Learning Approach, which conducts model training on a single central server the learning process shall be improved by means of an asynchronous decentralized learning process. In a first process phase the model is trained to converge by several iterations carried out between a local client and a server to find a final model state. After that, the client may select then the best model for each iteration to be provided with the actual, local hold out data.

[0005] Federated Learning within this document is also a machine learning setting where multiple entities (clients) collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client's raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective. The main drawback of Federated Learning in the context of an automotive application is, that the learning model according to Federated Learning is at some point considered to be optimized. However, vehicles as edge devices are highly dynamic and the data changes dynamically with the dynamic vehicle environment and vehicle interior situation. Therefore, the model can never be considered to be optimized. Moreover, for intelligent vehicle functions and around the autonomous driving context, the usage of personal and local behaviour adaption of the model is of high importance. Mainly because a generalized model from the Backend cannot provide the most accurate and best qualitative results a personal adapted model can do, towards the individual local environment the vehicle is placed at.

[0006] It is an object of the present invention to overcome the above-mentioned drawbacks by providing a solution that enables a dynamic, efficient and secure adaptation of automotive functions.

[0007] This problem is solved by the independent claims 1 and 3. Preferred embodiments are described in the dependent claims 2 and 4, respectively.

[0008] The Backend comprises a generalized federated model f FL deployed thereon. The System can be an Edge Computing System where Federated Learning is applied at the Backend. Within this application, an Edge Computing System is a distributed system where different entities are connected to communicate with each other forming a network. Throughout this document, Edge Computing can be understood as processing the data at an Internet of Things (loT)-device level. IoT is a ubiquitous computing paradigm in which everyday objects, e.g. vehicles, are enriched with embedded computational power towards monitoring the environment by collecting data without any user interaction in real-time. The collected data is processed and analyzed to generate knowledge. In the sense of Edge Computing, the Backend is represented as one level, but can be extended towards a plurality of levels or tiers, respectively, forming a complex hierarchy of the architecture. Processing and computation is moved further towards the devices level, wherein each device or edge is a vehicle. Accordingly, the latency, the computational power, and the storage of the Backend decreases. On the other hand, bandwidth, privacy, and responsiveness increases by pushing the intelligence and processing towards the vehicles instead of the Backend. Enabling computation, decision making, and intelligence at the vehicles provides the ability to perform machine learning in the vehicles close to the data sources collected by a plurality of well-known sensors, allowing to build low-latency and location-aware applications with keeping the privacy of the data collected and / or generated in each of the vehicles. Generally, the data collected and / or generated at the vehicles and the gained intelligence through machine learning can be used to support and improve the underlying infrastructure and management or to enable in-vehicle applications and services. Such services and applications supported by machine-learning techniques in the sense of Edge Computing can be found in the area of autonomous driving and / or autonomous driving use cases.

[0009] Federated Learning (FL) is a state of the art - approach for applying machine learning in edges, with reference to the above-mentioned system in the vehicles. Federated Learning has been introduced within the context of Edge Computing, aiming to solve the problem of machine-learning in edges under privacy sensitive data generated from massively distributed devices. According to the Federated Learning - Paradigm, multiple edges collaborate in solving a machine learning problem, under the coordination of the central Backend. Each vehicle's raw data is stored locally, in a storage device of one or more computational units of each respective vehicle and not exchanged or transferred to the Backend. Rather, focused updates intended for immediate aggregation are used to achieve the learning objective.

[0010] Each vehicle receives and deploys one global model, the generalized federated model f FL , that is globally trained by the central Backend. To do so, the Backend transmits the generalized federated model f FL via a Network, e.g. the mobile network, to each of the vehicles. Each vehicle is storing and deploying the globally trained generalized federated model f FL received from the Backend locally, e.g. in the storage device of the computational unit. With the locally generated / accessed data, each vehicle is locally optimizing the locally stored model. Rather than transmitting locally accessed / generated data to the Backend, each vehicle is transmitting update parameters, e.g. from the computational unit to the Backend.

[0011] In an exemplary implementation, Federated Learning (FL) - as applied by the Backend is a distributed machine learning optimization problem in a network of vehicles. The aim is to locally optimize an objection function J over distributed datasets. In the following, the vehicles will be also referred to as Edge Devices. In the following, the generation and updating of the generalized federated model f FL in the Backend and the Edge Devices is explained in more detail:

[0012] A network of k ∈ {1,...,k,...,K} of Edge Devices is designed with each holding a local subset of data D k so that D = {D 1 ,...,D k ,...,D K }.

[0013] This dataset D k is generated through Sensing and Actuating Nodes (SANs) sensing continuously contextual data in form of a d-dimensional vector x t with x ∈ ℝ d and t ∈ T = 1 , … , T with T ∈ T.

[0014] Each time t a new contextual vector Xt is received at the Edge Device. In centralized learning this data is forwarded to the Backend that minimizes the objective function over the entire dataset D. The optimization of the loss function (x, y; w) can be approximated by exploring the Empirical Risk Minimization (ERM) instead of the expected risk. Therefore, an approximation of over a training set D of size N using a set of models f ∈ with represented parameters w ∈ ℝ d the optimal solution for the ERM can be found. This is summarized in the following Equation: arg min w ∈ F J w ≈ J ^ N w = 1 N ∑ i = 1 N L x i y i w

[0015] In FL the optimization of the function is performed by locally optimizing an objective function in each Edge Device k over the local dataset D k with length n k so that ∑ k = 1 K n K = N .

[0016] The Backend is aggregating the local generated objective functions J k so that: J w = ∑ k = 1 K n k N J k w = ∑ k = 1 K n k N 1 n k ∑ i ∈ D k L x i , y i , w

[0017] The loss function (x i , y i , w) or local optimization function (w) can be also noted as j i (w). One exemplary algorithm to solve the problem of Equation (5.2) is using the Stochastic Gradient Descent (SGD). Each iteration t the Edge Device (ED) k aims to converge towards the minimum loss function , over its local data D k at time t given a new datapoint X t . This is by adapting the model parameters w k i with some factor η, called learning rate, and the gradient to the previous model parameters ∇j k (w k ). Each Edge Device k is performing the following Equation at time t, as shown by Equation 5.3: w k t + 1 ← w k t − η ∇ j k w k t

[0018] After multiple iterations, the updated model parameters of each Edge Device k are sent to the Backend.

[0019] The Federated Learning algorithm within the Backend aggregates the received model parameters inside the central coordinator towards the new generalized model. This averaging is summarised in Equation 5.4. w t + 1 = ∑ k = 1 K n k N w k t + 1

[0020] After merging the local gradients of each Edge Device k at the Backend, the final federated model is distributed to the Edge Devices and used locally for inference. Each ED k is selected at a random time epochs with s = {1, ... , s, ... , S} to update the distributed generalized federated model f FL with its local data stored in a Sliding Window (SLW) W k t and send the updated model parameters w k for the generalized federated model back to the Backend.

[0021] As outlined above, only implementing Federated Learning introduces multiple drawbacks within an application for an automotive environment. The first issue is the adaption of the generalized model - i.e. the final federated model distributed to the Edge Devices- to constantly changing environments in a vehicular context - e.g. highly dynamic environment, highly dynamic situation in the vehicle interior, etc. - in which the assumption of converging to a global minimization of the objective function does not hold. The second problem arises as each Edge Device k always overwrites the locally adapted model f̂ FL with a newly received generalized federated model f FL , such that locally learned context for each Edge Device is lost.

[0022] Moreover, with only applying Federated Learning, the global, final federated model is - at some point - considered to be optimized and is no longer updated by the Backend. However, when using machine learning in Edge Device environments and in particular in the automotive environment where each vehicle represents an Edge Device, the main challenge is the highly heterogeneous nature of the vehicles, resulting in non-i.i.d. data and evolving environment with possible concept drifts of the data collected by the vehicles.

[0023] In order to solve the above-mentioned problems, each vehicle further deploys a personalized model f k in a second computational unit locally in the respective vehicle.

[0024] In other words, each vehicle k has deployed two models, the above-mentioned generalized federated model f FL , received from the Backend, in a first computational unit and, in parallel, an evolving personalized model f k in a second computational unit.

[0025] For each prediction of a relevant automotive application - for example with reference to autonomous driving - a decision unit locally decides which of the personalized model f k or the generalized federated model f FL is the most accurate model to be used for the prediction ŷ: y ^ with either y ^ = f FL x or y ^ = f k x

[0026] Since each vehicle has limited resources, complex algorithms for the decision making cannot be used. Therefore, a lightweight method for balancing the personalized model f k and the generalized federated model f FL needs to be applied. In other words, since the vehicles resources are limited, a complex machine learning algorithm cannot be implemented for the decision making. Therefore, each vehicle of the system needs to apply continuous data prediction, forecasting and monitoring of the decision-making process.

[0027] Moreover, each vehicle needs to decide how and when to update both models, i.e. the generalized federated model f FL and the personalized model f k . Each decision unit needs to decide which of the above-mentioned models to choose from for prediction ŷ. As outlined above, at epoch s, the vehicle k of vehicles receives a globally trained model, i.e. the generalized federated model f FL from the Backend.

[0028] According to the invention, the decision unit, when locally deciding how using the generalized federated model f FL and the personalized model f k for the prediction ŷ, and when performing the prediction ŷ, performs the following steps: calculating a prediction error ε k for the local, personalized model f k deployed in the second computational unit; calculating a prediction error ε FL for the generalized federated model f FL deployed in the first computational unit; calculating a reward value θ for the calculated prediction error ε k and ε FL considering a historical performance of the local, personalized model f k and the generalized federated model f FL ; and calculating the prediction ŷ by balancing the prediction of the local, personalized model f k and the generalized federated model f FL .

[0029] As outlined above, each vehicle k has deployed two models, the generalized federated model f FL in the first computational unit and, in parallel, the evolving personalised model f k in the second computational unit. For each prediction ŷ of a relevant automotive application - for example with reference to autonomous driving - the decision unit could locally decide which of the personalized model f k or the generalized federated model f FL is the best model to be used for the prediction ŷ: y ^ with either y ^ = f FL x or y ^ = f k x

[0030] Since each vehicle has limited resources, complex algorithms for the decision making cannot be used. Therefore, a lightweight method for balancing the personalized model f k and the generalized federated model f FL are applied.

[0031] As outlined above, at epoch s, the vehicle k of vehicles receives the globally trained model, i.e. the generalized federated model f FL from the Backend.

[0032] As also outlined above, in order to enable a global update for the generalized federated model f FL , k vehicles perform over its locally stored data in a SLW Wk the update parameters using Equation (5.3). These updates are forwarded to the Backend and used into the new (a globally trained) model by combining all selected k vehicles update parameters, as shown in Equation (5.4). In a next step, each vehicle receives the new globally trained model, i.e. the updated generalized federated model f FL which can be used for prediction ŷ to perform the predictive analytics locally.

[0033] The personalized model f k is set to be at t = 0 the centrally received generalized federated model f FL . The choice of using a pre-build model as starting point lies in the theories of transfer learning, which highlights that using a baseline model for retraining is generating an earlier convergence and higher accuracy than starting from scratch. SANs continuously measure the surrounding of each vehicle and generates respective measurement data. Each vehicle receives this measurement data from the SANs at each time t with t ∈ T = 1 , … , t , … , T and T ∈ T. Each time t a d-dimensional data vector x t is collected and stored in the local window storage W k of size M. Only at the selected epoch s this data is used to update the newly received generalized federated model f FL . If a vehicle needs to perform the prediction ŷ, it is using the previous cached generalized federated model f FL .

[0034] In contrast, the personalized model f k is updated and retrained each time t at the vehicle using a local SGD process.

[0035] One main technical advantage and keeping both models inside the vehicles is mainly to overcome the problem of losing generalization and the adaption to unseen data by only deploying a personal model. Moreover, the previously unknown relationship between the vehicles in evolving systems can be changing from an i.i.d. to a non-i.i.d. data relationship. This leads towards better accuracy to the generalized model or personalized model depending on the connection. Using a balancing mechanism of the generalized and personalized model inside each vehicle provides the benefits of both data relations towards qualitative prediction results.

[0036] The methodology presented in this application contributes towards the solution of the above-mentioned problems by using a reward system for choosing the right model for the prediction. In particular, the decision unit of vehicle k is operable to weigh the personalized model f k and generalized federated model f FL to generate a combined prediction ŷ. The final prediction ŷ is calculated locally in each vehicle k of vehicles by using the local personalized model f k and the generalized federated model f FL with an adaptive balancing weight α. The prediction ŷ is performed / calculated locally in a decision unit of the respective vehicle.

[0037] The decision unit is operable to perform the steps of calculating a prediction error ε k for the local, personalized model f k deployed in the second computational unit; and calculating a prediction error ε FL for the generalized federated model f FL deployed in the first computational unit. To do so, the adaptive weight is calculated through multiple steps. First, whenever a vehicle receives from the SANs a new contextual vector x t at time t, the prediction error ε k for the local, personalized model f k with respect to the actual prediction y t and the prediction error ε FL for the generalized federated model f FL is calculated as shown in Equation (5.5) and (5.6), respectively. ϵL=yt−fkxtϵFL=yt−fFLxt

[0038] Given these two errors of Equation (5.5) and (5.6) inside each vehicle k of vehicles, it is possible to generate a reward value θ that is used as the source for balancing the personalized model f k and the generalized federated model f FL . In each vehicle k of vehicles at each time t based on the prediction error ε FL and the prediction error ε k a reward θ is set to 0 if the personalized model f k is performing better than the generalized federated model f FL . The reward θ is set to 1 if the generalized federated model f FL performs better, as is shown in the following Equation: θ t = 0 , ϵ FL > ϵ L 1 , ϵ FL ≤ ϵ L

[0039] As shown in Equation (5.7), a positive reward is given to the generalized federated model f FL if the absolute error ε FL is smaller than the absolute error ε k of the personalized model f k . As the reward system is deployed inside the respective vehicle 210 A ... 210 N should not only incorporate the current performance of the two models but also the historical performance, the reward-values θ at each time t are stored in a Sliding Window (SLW) O k with size U for the last t - U times. The SLW is defined in Equation (5.8) and contains just zeros and ones: O k t = θ t − U , … , θ t

[0040] The SLW O k consists of the most recent U rewards of θ ∈ {0, 1}. At each time t, the SLW of rewards is used as a reference to generate a ratio that represents the performance of both models over the time horizon t - U. This ratio is defining the adaptive weighting value α. An exemplary computation of a can be seen in Equation (5.9) and represents the historical performance of both models inside a respective vehicle. α = 1 U ∑ i = 1 U θ i

[0041] Each time t vehicle k is performing a prediction ŷ, the two model predictions of the personalized model f k and the generalized federated model f FL are weighted to the final prediction ŷ. The adaptive value α is used to combine the generalized federated model f FL and the local personalized model f k towards ŷ. The balancing of these two models is defined as shown in Equation (5.10): f ASM = αf FL + 1 − α f k

[0042] An exemplary implementation algorithm for the above-mentioned steps is as follows: Backend 230:

[0043] Respective vehicle 210 A ... 210 N (run at each vehicle k), wherein CL is the Backend 230:

[0044] The value range of α is between 1 and 0. A value of α → 1 means more influence towards the model predictions coming of the generalized federated model f FL , whereas a value of α → 0 places more importance on the locally generated predictions of the personalized model f k .

[0045] According to a further embodiment of the invention, the dynamic adaptation of the automotive application comprises one or more of the following: adapting any function of any driving assistance system application of the vehicle based on the calculated prediction value ŷ; adapting any function of any autonomous driving system application of the vehicle based on the calculated prediction value ŷ; and / or adapting any function of any infotainment system application of the vehicle based on the calculated prediction value ŷ; and / or adapting any appropriate function of any system application of the vehicle based on the calculated prediction value ŷ.

[0046] Considering as an example a function of a driving style of a user of the vehicle, a behavior or mood detection algorithm using audio, video and other sensor data inside the car can be used as input factors for generating the above-explained function. Based on the prediction value ŷ for the driving style / mood / behavior of the user of the vehicle, one possibility for an above-mentioned driving assistance system application can be the adaptation of speed and distance control of the vehicle after the function / prediction identified a tired or distracted driver as user of the vehicle. This increases safety and security while the driving. Further it is possible to adapt the infotainment system towards stimulus music or individual interior light design given a driver's mood and behavior as user of the vehicle to either e.g. calm a stressed driver down or wakeup a tired driver with their favorite dance music. Moreover but not limited, can be the adaption towards autonomous driving applications by using a predefined selection of driving modes (e.g. eco, electric, ...), routes (coffee shop, restaurant, scenic drive, ...) or routines (call a certain person, reserve a treatment, ...) given the learning and prediction of the personal preferences towards a certain environment.

[0047] According to another aspect of the invention, a computer-implemented method for improving the efficiency and security in dynamic adaptation of an automotive application in a vehicle is provided.

[0048] According to the invention, the locally deciding, by the decision unit, how using the generalized federated model f FL and the personalized model f k for the prediction ŷ, and the performing of the prediction ŷ comprises the steps of: calculating a prediction error ε k for the local, personalized model f k deployed in the second computational unit; calculating a prediction error ε FL for the generalized federated model f FL deployed in the first computational unit; calculating a reward value θ for the calculated prediction error ε k and ε FL considering a historical performance of the local, personalized model f k and the generalized federated model f FL ; and calculating the prediction ŷ by balancing the prediction of the local, personalized model f k and the generalized federated model f FL .

[0049] According to yet a further embodiment of the invention, the dynamic adaptation of the automotive application comprises one or more of the following: adapting any function of any driving assistance system application of the vehicle based on the calculated prediction value ŷ; adapting any function of any autonomous driving system application of the vehicle based on the calculated prediction value ŷ; and / or adapting any function of any infotainment system application of the vehicle based on the calculated prediction value ŷ; and / or adapting any appropriate function of any system application of the vehicle based on the calculated prediction value ŷ

[0050] The aspects defined above and further aspects of the present invention are apparent from the examples of embodiment to be described hereinafter and are explained with reference to the examples of embodiment. The invention will be described in more detail hereinafter with reference to examples of embodiment. However, it is apparent to the person skilled in the art that the invention is not limited to the examples of embodiment. Figure 1 shows an exemplary Edge Computing System where Federated Learning is applied; Figure 2 shows an exemplary system for improving the efficiency and security in dynamic adaptation of an automotive application in a vehicle; Figure 3 shows an exemplary method for improving the efficiency and security in dynamic adaptation of an automotive application in a vehicle.

[0051] Figure 1 shows an exemplary Edge Computing System 100 where Federated Learning is applied. Within this application, an Edge Computing System is a distributed system where different entities are connected to communicate with each other forming a network. Throughout this document, Edge Computing can be understood as processing the data at an Internet of Things (loT)-device level. IoT is a ubiquitous computing paradigm in which everyday objects, e.g. vehicles 110 A ... 110 N, 210 A ... 210 N, are enriched with embedded computational power towards monitoring the environment by collecting data without any user interaction in real-time. The collected data is processed and analyzed to generate knowledge. In the sense of Edge Computing, Backend 130 is represented as one level, but can be extended towards a plurality of levels or tiers, respectively, forming a complex hierarchy of the architecture. Processing and computation is moved further towards the devices level, wherein each device or edge is a vehicle 110 A ... 110 N, 210 A ... 210 N. Accordingly, the latency, the computational power, and the storage of the Backend 130 decreases. On the other hand, bandwidth, privacy, and responsiveness increases by pushing the intelligence and processing towards the vehicles 110 A ... 110 N, 210 A ... 210 N instead of the Backend 130. Enabling computation, decision making, and intelligence at the edges 110 A ... 110 N, 210 A ... 210 N provides the ability to perform machine learning in the edge devices or vehicles 110 A ... 110 N, 210 A ... 210 N close to the data sources, allowing to build low-latency and location-aware applications with keeping the privacy of the data collected and / or generated in each of the vehicles. Generally, the data collected and / or generated at edge devices / vehicles 110 A ... 110 N, 210 A ... 210 N and the gained intelligence through machine learning can be used to support and improve the underlying infrastructure and management or to enable in-vehicle applications and services. Such services and applications supported by machine-learning techniques in the sense of Edge Computing can be found in the area of autonomous driving and / or autonomous driving use cases.

[0052] Federated Learning (FL) is a state of the art - approach for applying machine learning in edges 110 A ... 110 N, 210 A ... 210 N. Federated Learning has been introduced within the context of Edge Computing, aiming to solve the problem of machine-learning in edges 110 A ... 110 N, 210 A ... 210 N under privacy sensitive data generated from massively distributed devices. According to the Federated Learning - Paradigm, multiple edges 110 A ... 110 N, 210 A ... 210 N collaborate in solving a machine learning problem, under the coordination of a central Backend 130. Each edge's raw data is stored locally, e.g. in a storage device of a computational unit 110 A ... 110 N, 210 A ... 210 N and not exchanged or transferred to the Backend 130. Rather, focused updates intended for immediate aggregation are used to achieve the learning objective. Each edge 110 A ... 110 N, 210 A ... 210 N receives and deploys one global model 132 that is globally trained by the central Backend 130. The Backend 130, 230 transmits a globally trained model 132 via a Network, e.g. the mobile network 140, 240 to the edges 110 A ... 110 N, 210 A ... 210 N. Each edge 110 A ... 110 N, 210 A ... 210 N is storing and deploying the globally trained model 132 received from the Backend 130, 230 locally, e.g. in a storage device of the computational unit 120 A ... 120 N, 220 A ... 220 N. With the locally generated / accessed data, each edge 110 A ... 110 N, 210 A ... 210 N is locally optimizing the locally stored model. Rather than transmitting locally accessed / generated data to the Backend 130, 230, each edge 110 A ... 110 N, 210 A ... 210 N is only transmitting update parameters, e.g. from the computational unit 120 A ... 120 N, 220 A ... 220 N to the Backend 130, 230.

[0053] In an exemplary implementation, Federated Learning (FL) - as applied by the Backend 130, 230 is a distributed machine learning optimization problem in a network of Edge Devices 110 A ... 110 N, 210 A ... 210 N. The aim is to locally optimize an objection function J over distributed datasets.

[0054] A network of k ∈ {1,...,k,...,K} of Edge Devices 110 A ... 110 N, 210 A ... 210 N is designed with each holding a local subset of data D k , so that D = {D 1 ,...,D k ,...,D K }.

[0055] This dataset D k is generated through Sensing and Actuating Nodes (SANs) sensing continuously contextual data in form of ad-dimensional vector x t with x ∈ ℝ d and t ∈ T = 1 , … , T with T ∈ T.

[0056] Each time t a new contextual vector Xt is received at the Edge Device 110 A ... 110 N, 210 A ... 210 N. In centralized learning this data is forwarded to the Backend 130 that minimizes the objective function over the entire dataset D. The optimization of the loss function (x, y; w) can be approximated by exploring the Empirical Risk Minimization (ERM) instead of the expected risk. Therefore, an approximation of over a training set D of size N using a set of models f ∈ with represented parameters w ∈ ℝ d the optimal solution for the ERM can be found. This is summarized in the following Equation: arg min w ∈ F J w ≈ J ^ n w = 1 N ∑ i = 1 N L x i y i w

[0057] In FL the optimisation of the function is performed by locally optimizing an objective function in each Edge Device k 110 A ... 110 N, 210 A ... 210 N over the local dataset D k with length n k so that ∑ k = 1 K n K = N ..

[0058] The Backend 130, 230 is aggregating the local generated objective functions J k so that: J w = ∑ k = 1 K n k N J k w = ∑ k = 1 K n k N 1 n k ∑ i ∈ D k L x i y i w

[0059] The loss function (x i , yi, w) or local optimization function (w) can be also noted as j i (w). One exemplary algorithm to solve the problem of Equation (5.2) is using the Stochastic Gradient Descent (SGD). Each iteration t the Edge Device (ED) k aims to converge towards the minimum loss function , over its local data D k at time t given a new datapoint Xt. This is by adapting the model parameters w k i with some factor , called learning rate, and the gradient to the previous model parameters ∇j k (w k ). Each Edge Device k is performing the following Equation at time t, as shown by Equation 5.3: w k t + 1 ← w k t − η ∇ j k w k t

[0060] After multiple iterations, the updated model parameters of each Edge Device k 110 A ... 110 N, 210 A ... 210 N are sent to the Backend 130, 230.

[0061] The Federated Learning algorithm within the Backend 130, 230 aggregates the received model parameters inside the central coordinator towards the new generalized model. This averaging is summarized in Equation 5.4. w t + 1 = ∑ k = 1 K n k N w k t + 1

[0062] After merging the local gradients of each Edge Device k 110 A ... 110 N, 210 A ... 210 N at the Backend 130, 230, the final federated model 132 is distributed to the Edge Devices 110 A ... 110 N, 210 A ... 210 N and used locally for inference. Each ED k is selected at a random time epochs with s = {1, ... , s, ... , S} to update the distributed federated model f FL with its local data stored in a Sliding Window (SLW) W k t and send the updated model parameters w k back to the Backend 130, 230.

[0063] As outlined above, an implementation of Federated Learning introduces multiple drawbacks within an application for an automotive environment. The first issue is the adaption of the generalized model - i.e. the final federated model 132 distributed to the Edge Devices f FL -to constantly changing environments in a vehicular context - e.g. highly dynamic environment, highly dynamic situation in the vehicle interior, etc. - in which the assumption of converging to a global minimization of the objective function does not hold. The second problem arises as each Edge Device 110 A ... 110 N always overwrites the locally adapted model f̂ FL with a newly received generalized model f FL , such that locally learned context for each Edge Device 110 A ... 110 N is lost.

[0064] Moreover, with the application of Federated Learning, the global, final federated model 132 is - at some point - considered to be optimized and is no longer updated by the Backend 130. However, when using machine learning in edge device environments and in particular in the automotive environment where each vehicle 110 A ... 110 N represents an edge-device, the main challenge is the highly heterogeneous nature of the vehicles, resulting in non-i.i.d. data and continuous evolving of the data collected by the vehicles 110 A ... 110 N. This problem, however, cannot be addressed by only using the above-explained Federated Learning.

[0065] Figure 2 shows an exemplary system 200 improving the efficiency in security in dynamic adaption of automotive functions for a plurality of vehicles 210 A ... 210 N in a highly dynamic external and internal environment. The system 200 is in accordance with the system 100 as explained with reference to Figure 1 above for the generalized federated model f FL , and comprises the following additional features.

[0066] Each vehicle k 210 A ... 210 N has deployed two models, for example a generalized federated model f FL in a first computational unit 220 A ... 220 N and, in parallel, an evolving personalized model f k in a second computational unit 230 A ... 230 N. For each prediction of a relevant automotive application - for example with reference to autonomous driving - a decision unit 240 A ... 240 N, locally decides which of the personalized model f k or the generalized federated model f FL is the right model to be used for the prediction ŷ: ŷ with either ŷ = f FL (x) or ŷ = f k (x)

[0067] Since each vehicle 210 A ... 210 N has limited resources, complex algorithms for the decision making cannot be used. Therefore, a lightweight method for balancing the personalized model f k and the generalized federated model f FL needs to be applied. In other words, in vehicles 210 A ... 210 N, a complex machine learning algorithm cannot be implemented for the decision making. Therefore, each vehicle of the system needs to apply continuous data prediction, forecasting and monitoring of the decision-making process.

[0068] Therefore, each vehicle 210 A ... 210 N needs to decide how and when to update both models, i.e. the generalized federated model f FL and the personalized model f k . Moreover, each decision unit 240 A ... 240 N needs to decide which of the above-mentioned models to choose from for prediction ŷ. As outlined with reference to Figure 1 above, at epoch s, the vehicle k of vehicles 210 A ... 210 N receives a globally trained model 232, i.e. the generalized federated model f FL from the Backend 230.

[0069] To do so, the vehicle k of vehicles 210 A ... 210 N performs over its locally stored data in a SLW Wk the update parameters using Equation (5.3). These updates are forwarded to the Backend 230 and used into the new a globally trained model 232 by combining all selected k vehicles 210 A ... 210 N update parameters, as shown in Equation (5.4). In a next step, each vehicle 210 A ... 210 N receive the new globally trained model 232 which can be used for prediction ŷ to perform the predictive analytics locally.

[0070] The personalized model f k is set to be at t = 0 the centrally received generalized federated model f FL The choice of using a pre-build model as starting point lies in the theories of transfer learning, which highlights that using a baseline model for retraining is generating an earlier convergence and higher accuracy than starting from scratch. SANs continuously measure the surrounding of each vehicle 210 A ... 210 N and generates respective measurement data. Each vehicle 210 A ... 210 N receives this measurement data from the SANs at each time t with t ∈ T = 1 , … , t , … , T and T ∈ T. Each time t a d-dimensional data vector x t is collected and stored in the local window storage W k of size M. Only at the selected epoch s this data is used to update the newly received generalized federated model f FL . If a vehicle 210 A ... 210 N needs to perform the prediction ŷ, it is using the previous cached model generalized federated model f FL .

[0071] In contrast, the personalized model f k is updated and retrained each time t at the vehicle 210 A ... 210 N using a local SGD process.

[0072] One main technical advantage and keeping both models inside the vehicles 210 A ... 210 N is mainly to overcome the problem of losing generalization and the adaption to unseen data by only deploying a personal model. Moreover, the previously unknown relationship between the vehicles in evolving systems can be changing from an i.i.d. to a non-i.i.d. data relationship. This leads towards better accuracy to the generalized model or personalized model depending on the connection. Using a balancing mechanism of the generalized and personalized model inside each vehicle provides the benefits of both data relations towards qualitative prediction results.

[0073] The methodology presented in this application contributes towards the solution of the above-mentioned problem by using a reward system to weigh the personalized model f k and generalized federated model f FL combined prediction ŷ. The final prediction ŷ is calculated locally in each vehicle k of vehicles 210 A ... 210 N by using the local personalized model f k and the generalized federated model f FL with a adaptive balancing weight α. The prediction ŷ is performed / calculated locally in a decision unit 240 A ... 240 N of the respective vehicle 210 A ... 210 N.

[0074] The adaptive weight is calculated through multiple steps. First, whenever a vehicle 210 A ... 210 N receives from the SANs a new contextual vector x t at time t, the prediction error ε k for the local, personalized model f k with respect to the actual prediction ŷ t and the prediction error ε FL for the generalized federated model f FL is calculated as shown in Equation (5.5) and (5.6), respectively. ϵ L = y t − f k x t ϵ FL = y t − f FL x t

[0075] Given these two errors of Equation (5.5) and (5.6) inside each vehicle k of vehicles 210 A ... 210 N, it is possible to generate a reward value θ that is used as the source for balancing the personalized model f k and the generalized federated model f FL . In each vehicle k of vehicles 210 A ... 210 N at each time t based on the prediction error ε FL and the prediction error ε k a reward θ is set to 0 if the personalized model f k is performing better than the generalized federated model f FL . The reward θ is set to 1 if the generalized federated model f FL performs better, as is shown in the following Equation: θ t = 0 , ϵ FL > ϵ L 1 , ϵ FL ≤ ϵ L

[0076] As shown in Equation (5.7), a positive reward is given to the generalized federated model f FL if the absolute error ε FL is smaller than the absolute error ε k of the personalized model f k . As the reward system is deployed inside the respective vehicle 210 A ... 210 N should not only incorporate the current performance of the two models but also the historical performance, the reward-values θ at each time t are stored in a Sliding Window (SLW) O k with size U for the last t - U times. The SLW is defined in Equation (5.8) and contains just zeros and ones: O k t = θ t − U , … , θ t

[0077] The SLW O k consists of the most recent U rewards of θ ∈ {0, 1}. At each time t, the SLW of rewards is used as a reference to generate a ratio that represents the performance of both models over the time horizon t - U. This ratio is defining the adaptive weighting value α. An exemplary computation of a can be seen in Equation (5.9) and represents the historical performance of both models inside a respective vehicle 210 A ... 210 N. α = 1 U ∑ i = 1 U θ i

[0078] Each time the respective vehicle 210 A ... 210 N is performing a prediction ŷ, the two model predictions of the personalized model f k and the generalized federated model f FL are weighted to the final prediction ŷ. The adaptive value α is used to combine the generalized federated model f FL and the local personalized model f k towards ŷ. The balancing of these two models is defined as shown in Equation (5.10): f ASM = αf FL + 1 − α f k

[0079] An exemplary implementation algorithm for the above-mentioned steps is as follows: Backend 230:

[0080] Respective vehicle 210 A ... 210 N (run at each vehicle k, wherein CL is the Backend 230):

[0081] The value range of α is between 1 and 0. A value of α → 1 means more influence towards the model predictions coming of the generalized federated model f FL , whereas a value of α → 0 places more importance on the locally generated predictions of the personalized model f k .

[0082] The dynamic adaptation of the automotive application may comprise one or more of the following: adapting any function of any driving assistance system application of the vehicle k of vehicles 210 A ... 210 based on the calculated prediction value ŷ; adapting any function of any autonomous driving system application of the vehicle k of vehicles 210 A ... 210 based on the calculated prediction value ŷ; and / or adapting any function of any infotainment system application of the vehicle k of vehicles 210 A ... 210 based on the calculated prediction value ŷ; and / or adapting any appropriate function of any system application of the vehicle k of vehicles 210 A ... 210 based on the calculated prediction value ŷ.

[0083] Considering as an example a function of driving style, behavior or mood detection algorithm of a user of vehicle k of vehicles 210 A ... 210, audio, video and other sensor data inside vehicle k of vehicles 210 A ... 210 can be taken as input factors for generating the above-explained function. Based on the prediction value ŷ for the driving style / mood / behavior of a user of vehicle k of vehicles 210 A ... 210, one possibility for a driving assistance system application can be the adaption of speed and distance control of vehicle k of vehicles 210 A ... 210 after the function / prediction identified a tired or distracted driver as user of vehicle k of vehicles 210 A ... 210. This increases safety and security while the driving. Further it is possible to adapt the infotainment system of vehicle k of vehicles 210 A ... 210 towards stimulus music or individual interior light design given a mood and behavior of a user of vehicle k of vehicles 210 A ... 210 to either e.g. calm the stressed driver / user down or wakeup the tired driver / user with their favorite dance music, respectively. Moreover but not limited, can be the adaption towards autonomous driving applications by using a predefined selection of driving modes (e.g. eco, electric,...), routes (coffee shop, restaurant, scenic drive,...) or routines (call a certain person, reserve a treatment,...) of vehicle k of vehicles 210 A ... 210 given the learning and prediction of personal preferences of a user of vehicle k of vehicles 210 A ... 210 towards a certain environment.

[0084] Figure 3 shows an exemplary computer-implemented Method 300 for improving the efficiency and security in dynamic adaptation of an automotive application in a vehicle 210 A ... 210 N. The method 300 can be performed and / or executed by the System 200 as outlined with reference to Figure 1 and Figure 2 above.

[0085] The locally deciding 330, by the decision unit 240 A ... 240 N, how using the generalized federated model f FL and the personalized model f k for the prediction ŷ, and the performing 340 of the prediction ŷ may comprise the steps of: calculating a prediction error ε k for the local, personalized model f k deployed in the second computational unit 230 A ... 230 N; calculating a prediction error ε FL for the generalized federated model f FL deployed in the first computational unit 220 A ... 220 N; calculating a reward value θ for the calculated prediction error ε k and ε FL considering a historical performance of the local, personalized model f k and the generalized federated model f FL ; and calculating the prediction ŷ by balancing the prediction of the local, personalized model f k and the generalized federated model f FL .

[0086] The dynamic adaptation of the automotive application 350 may comprise one or more of the following: adapting any function of any driving assistance system application of the vehicle 210 A ... 210 N based on the calculated prediction value ŷ; adapting any function of any autonomous driving system application of the vehicle 210 A ... 210 N based on the calculated prediction value ŷ; and / or adapting any function of any infotainment system application of the vehicle 210 A ... 210 N based on the calculated prediction value ŷ; and / or adapting any appropriate function of any system application of the vehicle based 210 A ... 210 N based on the calculated prediction value ŷ .

Claims

1. Vehicle system (200) for improving the efficiency and security in dynamic adaptation of an automotive application in a vehicle (210 A ... 210 N), the system (200) comprising: a Backend (230) comprising a generalized federated model fFL deployed thereon, wherein the vehicle (210 A ... 210 N) is operable to receive and deploy the generalized federated model fFL in a first computational unit (220 A ... 220 N); wherein the vehicle (210 A ... 210 N) is operable to deploy a personalized model fk in a second computational unit (230 A ... 230 N); and wherein the vehicle (210 A ... 210 N) comprises a decision unit (240 A ... 240 N) that is operable to locally decide, for a prediction ŷ of the automotive application, how using the generalized federated model fFL and the personalized model fk; calculating, by the decision unit (240 A ... 240 N), the prediction ŷ using the decided model; and adapting the function of the automotive application based on the prediction ŷ, wherein the decision unit (240 A ... 240 N), when locally deciding how using the generalized federated model fFL and the personalized model fk for the prediction ŷ, and when performing the prediction ŷ performs the following steps: - calculating a prediction error εk for the local, personalized model fk deployed in the second computational unit (230 A ... 230 N); - calculating a prediction error εFL for the generalized federated model fFL deployed in the first computational unit (220 A ... 220 N); - calculating a reward value θ for the calculated prediction error εk and εFL - considering a historical performance of the local, personalized model fk and the generalized federated model fFL; and - calculating the prediction ŷ by balancing the prediction of the local, personalized model fk and the generalized federated model fFL, wherein the two model predictions of the personalized model and the generalized federated model are weighted to the final prediction ŷ, wherein historical performance is incorporated by storing the reward-values θ at each time t in a sliding window Ok with size U for the last t - U times, where the sliding window Ok consists of the most recent U rewards of θ , and wherein at each time t, the sliding window Ok of rewards is used as a reference to generate a ratio that represents the performance of both models over the time horizon t - U, this ratio then defining an adaptive weighting value α representing the historical performance of both models inside a respective vehicle and using the adaptive value α to combine the generalized federated model and the local personalized model towards the prediction ŷ.

2. Vehicle system (200) according to the preceding claim and comprising a driving assistance system and / or an autonomous driving system and / or an infotainment system, wherein a dynamic adaptation of the function of the automotive application comprises one or more of the following: - adapting any function of the driving assistance system application of the vehicle (210 A ... 210 N) based on the calculated prediction value ŷ; - adapting any function of the autonomous driving system application of the vehicle (210 A ... 210 N) based on the calculated prediction value ŷ; and / or - adapting any function of the infotainment system application of the vehicle (210 A ... 210 N) based on the calculated prediction value ŷ; and / or - adapting any appropriate function of any system application of the vehicle based (210 A ... 210 N) based on the calculated prediction value ŷ.

3. Computer-implemented Method (300) for improving the efficiency and security in dynamic adaptation of an automotive application in a vehicle (210 A ... 210 N), the Method (300) comprising the following steps: - receiving and deploying (310), from a Backend (230), a generalized federated model fFL in a first computational unit (220 A ... 220 N) of the vehicle (210 A ... 210 N); - deploying (320) a personalized model fk in a second computational unit (230 A ... 230 N) of the vehicle (210 A ... 210 N); - locally deciding (330), by a decision unit (240 A ... 240 N) of the vehicle (210 A ... 210 N), for a prediction ŷ of the automotive application, how using the generalized federated model fFL and the personalized model fk; - calculating (340), by the decision unit (240 A ... 240 N) the prediction ŷ based on the decided model; and - adapting (350) a function of the automotive application, by the vehicle (210 A ... 210 N) based on the prediction ŷ, wherein the locally deciding (330), by the decision unit (240 A ... 240 N) how using the generalized federated model fFL and the personalized model fk for the prediction ŷ, and the performing (340) of the prediction ŷ comprises the steps of: - calculating a prediction error εk for the local, personalized model fk deployed in the second computational unit (230 A ... 230 N); - calculating a prediction error εFL for the generalized federated model fFL deployed in the first computational unit (220 A ... 220 N); - calculating a reward value θ for the calculated prediction error εk and εFL - considering a historical performance of the local, personalized model fk and the generalized federated model fFL; and - calculating the prediction ŷ by balancing the prediction of the local, personalized model fk and the generalized federated model fFL , wherein the two model predictions of the personalized model and the generalized federated model are weighted to the final prediction ŷ , wherein historical performance is incorporated by storing the reward-values θ at each time t in a sliding window Ok with size U for the last t - U times, where the sliding window Ok consists of the most recent U rewards of θ , and wherein at each time t, the sliding window Ok of rewards is used as a reference to generate a ratio that represents the performance of both models over the time horizon t - U, this ratio then defining an adaptive weighting value α representing the historical performance of both models inside a respective vehicle and using the adaptive value α to combine the generalized federated model and the local personalized model towards the prediction ŷ.

4. Method (300) according to claim 3, wherein the vehicle comprises a driving assistance system and / or an autonomous driving system and / or an infotainment system and wherein a dynamic adaptation of the function of the automotive application (350) comprises one or more of the following: - adapting any function of any driving assistance system application of the vehicle (210 A ... 210 N) based on the calculated prediction value ŷ; - adapting any function of any autonomous driving system application of the vehicle (210 A ... 210 N) based on the calculated prediction value ŷ; and / or - adapting any function of any infotainment system application of the vehicle (210 A ... 210 N) based on the calculated prediction value ŷ; and / or - adapting any appropriate function of any system application of the vehicle based (210 A ... 210 N) based on the calculated prediction value ŷ.