Methods and systems for training a neural network model via cross-silo federated learning

Abstract

A method is provided for increasing an efficiency of training a neural network using synchronized learning within a federated learning system. In one example, training the neural network model includes performing a weighted aggregation of a first set of accumulated gradients of a plurality of model parameters over a first plurality of batches of a first set of training data, and a second set of accumulated gradients of a second, identical neural network model trained on a second plurality of batches of a second set of training data; updating parameters of the neural network model based on the aggregated gradients; and in response to an accuracy of the neural network model decreasing, adjusting one or more of a batch size of the first plurality of batches, a first aggregation weighting coefficient of the first set of accumulated gradients, and a learning rate used during training.

Description

METHODS AND SYSTEMS FOR TRAINING A NEURAL NETWORK MODEL VIACROSS- SILO FEDERATED LEARNINGFIELD

[0001] The present disclosure relates to systems and methods for increasing a performance of neural network models using cross-silo federated learning.BACKGROUND

[0002] Standards governing data privacy and security protect data privacy and ensure data security of users. In some cases, these standards govern the creation of data, usage of data, storage of data, and transfer of data between foreign countries. Since vehicle Driver Monitoring Systems (DMS)ZOccupant Monitoring Systems (OMS) features are based on in-cabin operator and occupant monitoring, the data collected may be personal in nature. As such, these standards may hinder global enterprises that rely on training data transnationally collected from multiple nations to develop Machine Learning (ML) / Deep Learning (DL) models for DMS / OMS features employed in vehicles. To ensure data privacy and security standards are maintained, federated learning (FL) may allow training of ML / DL models across decentralized edge devices and / or servers that store local data samples without exchanging the data samples between them.

[0003] However, current methods for implementing FL for transnationally distributed systems are hindered by bandwidth constraints of transnational network communication, since training speed may be reduced by bandwidth constraints, and different data distributions of local data at different geographical locations may result in different convergent speeds during model training and different model generalizability. Consequently, the training efficiency of the models and model accuracy may be reduced due to the disadvantages of existing FL techniques.SUMMARY

[0004] The current disclosure at least partially addresses one or more of the above identified issues by a method for an advanced driver assistance system (ADAS) of a vehicle, the method comprising adjusting an operation of the vehicle based on images of an occupant of the vehicle captured via an in-cabin monitoring system of the vehicle, wherein the in-cabin monitoring system relies on a neural network model trained using synchronized learning within a cross-silo federatedlearning (FL) system, and training the neural network model includes performing a weighted aggregation of a first set of accumulated gradients of a plurality of parameters of the neural network model over a first plurality of batches of a first set of training data, and a second set of accumulated gradients of a second, identical neural network model trained on a second plurality of batches of a second set of training data; updating parameters of the neural network model based on the aggregated gradients ; and in response to an accuracy of the neural network model decreasing over a pre-determined number of iterations, adjusting one or more of a batch size of the first plurality of batches, a first aggregation weighting coefficient assigned to the first set of accumulated gradients, and a learning rate. The in-cabin monitoring system may be a DMS or an OMS of the vehicle.

[0005] Thus, the proposed cross-silo FL system advantageously makes global adjustments of training parameters of different copies of a same neural network model separately trained on different servers in a decentralized manner, based on gradients that are aggregated over the different copies. In this way, a plurality of different copies of a neural network may benefit from training data acquired in a transnationally distributed system, where data privacy and security standards prevent data stored in servers at different geographical locations (referred to herein as local servers) from being exchanged directly. Each server may be located at a specific geographical location and may generate and store data from that specific location. By accumulating the gradients locally and then exchanging the gradients between the different servers, rather than sending training data to a centralized, global neural network model, an amount of bandwidth used during training and an amount of time relied on for exchanging data may be reduced, increasing an efficiency of training, while ensuring that personal data is maintained within a geographical region.

[0006] Other systems, methods, features, and advantages of the present disclosure will be or become apparent to one with skill in the art upon examination of the following drawings and detailed description. It is intended that all such additional systems, methods, features, and advantages be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims.

[0007] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which some embodiments described herein pertain. Although methods and materials similar to or equivalent to thosedescribed herein can be used in the practice or testing of embodiments described herein, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative and are not intended to be necessarily limiting.

[0008] Implementation of the method and / or system of some embodiments described herein can involve performing or completing selected tasks automatically. Moreover, according to actual instrumentation and equipment of embodiments of the method and / or system described herein, several selected tasks could be implemented by hardware, by software or by firmware or by a combination thereof using an operating system.

[0009] For example, hardware for performing selected tasks according to some embodiments described herein could be implemented as a chip or a circuit. As software, selected tasks according to embodiments described herein could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In an exemplary embodiment described herein, one or more tasks according to exemplary embodiments of methods and / or systems as described herein are performed by a data processor, such as a computing platform for executing a plurality of instructions. Optionally, the data processor includes a volatile memory for storing instructions and / or data and / or a non-volatile storage, for example, a magnetic hard-disk and / or removable media, for storing instructions and / or data. Optionally, a network connection is provided as well. A display and / or a user input device such as a keyboard or mouse are optionally provided as well.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Some embodiments are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars are shown by way of example and for purposes of illustrative discussion of embodiments described herein. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments described herein may be practiced.

[0011] In the drawings:

[0012] FIG. 1 systematically shows a vehicle including an ADAS, according to one or more embodiments of the present disclosure;

[0013] FIG. 2 illustrates an example FL system for transnational^ distributed vehicle data stored in a plurality of local servers, according to one or more embodiments of the present disclosure;

[0014] FIG. 3A illustrates a first example system for updating local neural network models at local servers in a transnational^ distributed system, according to one or more embodiments of the present disclosure;

[0015] FIG. 3B illustrates a second example system for updating local neural network models at local servers in a transnational^ distributed system, according to one or more embodiments of the present disclosure;

[0016] FIG. 4 shows a block diagram of an exemplary embodiment of a node of an FL system configured to train a neural network model, in accordance with an embodiment of the present disclosure;

[0017] FIG. 5 shows a method for training a neural network model using synchronized learning, according to one or more embodiments of the present disclosure;

[0018] FIG. 6 shows a method for accumulating a plurality of sets of gradients for of a respective plurality of mini-batches of training data at a local server, according to one or more embodiments of the present disclosure; and

[0019] FIG. 7 shows a method for using a trained local neural network model to implement an ADAS intervention, according to one or more embodiments of the present disclosure.DETAILED DESCRIPTION

[0020] Methods are provided to increase an efficiency of training a neural network using synchronized learning within a federated learning (FL) system, where training data stored at a plurality of different servers may be used to train the neural network without copying, moving, exchanging, or sharing the training data. For example, the servers may be in different geographical locations, and the training data may include personal data that is protected under different laws applying to the different geographical locations.

[0021] In an embodiment, the personal data includes images or image data of faces of drivers of vehicles in the different locations, and the neural network is used to detect facial features and / or expressions of the drivers. For example, the facial features and / or expressions of the drivers may be captured by a driver monitoring system (DMS) or occupant monitoring system (OMS) of avehicle, and the neural network may be trained to identify the driver, and / or to predict a state of the driver (e.g., tired, alert, anxious, etc.). However, it may be appreciated that the systems and methods described herein may apply to other types of distributed data and the neural network models described herein may be trained to perform other actions based on the different types of data.

[0022] In the proposed cross-silo FL system, during training, identical copies of the neural network are trained in a synchronized manner at different local servers (e.g., corresponding to the different geographical locations). Each local neural network model of each local server may be trained by splitting a larger batch of training data into a plurality of smaller mini-batches. For example, a larger batch of training data may include 1280 training pairs and smaller mini-batch may include 64 training pairs. Each mini-batch may have a fixed batch size. Each mini-batch of training data is loaded into the respective local neural network model one at a time, and sets of gradients for a plurality of network parameters calculated via forward and backward propagation for each mini-batch of training data are collected and stored (e.g., accumulated).

[0023] Each local server may send a corresponding plurality of sets of accumulated gradients to the other local servers. Each local server may aggregate the accumulated gradients received from the other local servers and utilize the aggregated gradients to update the local neural network model stored in the associated local server by updating the parameters of the local neural network model using the aggregated gradients. Because the aggregated gradients are the same at each local server, the parameters of each local model are updated in a same way, such that at the end of each iteration of training, the parameters of each local model will be the same.

[0024] Responsive to model accuracy continuously decreasing on a local validation dataset after a pre-determined number of iterations, a batch size of training data may be decreased by a predetermined amount for all local servers, a global learning rate may be adjusted based on a new batch size, and an aggregation weighting coefficient of specific local servers may be increased. By globally adjusting parameters of each individual neural network model in response to the continuously decreasing model accuracy, training efficiency and model accuracy may be increased. By utilizing the plurality of mini-batches of training data to train the local neural network model at a local server, computational requirements of the local server are reduced without increasing FL implementation time, since the time limiting step occurs when aggregating the accumulated gradients.

[0025] After training of the local neural network model, the trained neural network model at a local server may be stored in memory of a plurality of vehicles located in the geographical location of the local server, where the training data to train the local neural network model was collected. For example, a DMS and / or an OMS may use the model to detect facial features of a vehicle operator and / or occupant, for example, to load a driver profde. For example, a DMS and / or OMS of a vehicle may be used by an Advanced Driver Assistance Systems (ADAS) of the vehicle, and the trained neural network model may be used by the ADAS to determine an emotional state of the driver based on detected facial expressions. The ADAS may adjust an operating parameter of the vehicle based on the emotional state of the operator.

[0026] The following description relates to a system and methods for training a local neural network model using synchronized at a local server communicatively coupled to a plurality of local servers within a FL system. A trained neural network model may be integrated within an ADAS of a vehicle, such as the vehicle shown in FIG. 1. An example of the FL system wherein the neural network model is trained on transnationally distributed vehicle data is shown in FIG. 2. Systems wherein updating of local neural network models at associated local servers occurs is illustrated in FIG. 3A and FIG. 3B. An exemplary node of the FL system configured to update a local neural network model is shown in FIG. 4. The local neural network model may be trained according to the methods described in FIG. 5 and FIG. 6. A trained local neural network model may be used to implement ADAS intervention according to the method described in FIG. 7.

[0027] Turning now to the figures, FIG. 1 schematically shows an exemplary vehicle 100. The vehicle 100 includes a dashboard 102, a driver seat 104, a first passenger seat 106, a second passenger seat 108, and a third passenger seat 110. In other examples, the vehicle 100 may include more or fewer passenger seats. The driver seat 104 and the first passenger seat 106 are located in a front of the vehicle, proximate to the dashboard 102, and therefore may be referred to as front seats. The second passenger seat 108 and the third passenger seat 110 are located at a rear of the vehicle and may be referred to as back (or rear) seats.

[0028] Additionally, the vehicle 100 includes a plurality of integrated speakers 114, which may be arranged around a periphery of the vehicle 100. In some embodiments, the integrated speakers 114 are electronically coupled to an electronic control system of the vehicle, such as to a computing system 120, via a wired connection. In other embodiments, the integrated speakers 114 may wirelessly communicate with the computing system 120. As an example, an audio fde may beselected by an occupant of the vehicle 100, such as a driver passenger, via a user interface 116, and the selected audio file may be projected via the integrated speakers 114. In some examples, audio alerts may be generated by the computing system 120 and also may be projected by the integrated speakers 114, such as will be elaborated herein.

[0029] The vehicle 100 includes a steering wheel 112 and a steering column 122, through which the driver may input steering commands for the vehicle 100. The vehicle 100 further includes one or more camera 118. The camera 118 may be one camera of a plurality of cameras. In the embodiment shown in FIG. 1 , the camera 118 is positioned to the side of the driver seat 104, which may aid in monitoring the driver in profile. However, in other examples, the camera 118 may be positioned in other locations in the vehicle, such as on the steering column 122, directly in front of the driver seat 104. Further, the camera 118 may be positioned to the side of the passenger seats 106, 108, and 110 or directly in front of the passenger seats 106, 108, and 110.

[0030] Additionally, in the embodiment shown in FIG. 1, the camera 119 is positioned on the exterior of the back end of the vehicle 100, which may aid in monitoring the position of the vehicle 100 in a lane and / or monitor the position of the vehicle 100 relative to other vehicles and / or the surrounding environment, as some examples. In other examples, the camera 119 may be positioned in other locations in the vehicle, such as the exterior of the front end of the vehicle 100 or the sides of the vehicle 100.

[0031] The camera 118 may include one or more optical (e.g., visible light) cameras, one or more infrared (IR) cameras, or a combination of optical and IR cameras having one or more view angles. In some examples, the camera 118 may have interior view angles as well as exterior view angles. In some examples, the camera 118 may include more than one lens and more than one image sensor. For example, the camera 118 may include a first lens that directs light to a first, visible light image sensor (e.g., a charge-coupled device or a metal-oxide-semiconductor) and a second lens that directs light to a second, thermal imaging sensor (e.g., a focal plane array), enabling the camera 118 to collect light of different wavelength ranges for producing both visible and thermal images. In some examples, the camera 118 may further include a depth camera and / or sensor, such as a time-of-flight camera or a LiDAR sensor.

[0032] In some examples, the camera 118 may be a digital camera configured to acquire a series of images (e.g., frames) at a programmable frequency (e.g., frame rate) and may be electronically and / or communicatively coupled to the computing system 120. Further, the camera 118 mayoutput acquired images to the computing system 120 in real-time so that they may be processed in real-time by the computing system 120 and / or a computer network. As used herein, the term “realtime” denotes a process that occurs instantaneously and without intentional delay. “Real-time” may refer to a response time of less than or equal to about 1 second, for example. In some examples, “real-time” may refer to simultaneous or substantially simultaneous processing, detection, or identification. Further, in some examples, the camera 118 may be calibrated with respect to a world coordinate system (e.g., world space x, y, z). In other examples, the camera 118 may acquire images to determine states, attributes, and poses of the driver or passenger (e.g., occupant), such as will be elaborated herein.

[0033] The vehicle 100 may further include a driver seat sensor 124 coupled to or within the driver seat 104 and a passenger seat sensor 126 coupled to or within the first passenger seat 106. The back seats may also include seat sensors, such as a passenger seat sensor 128 coupled to the second passenger seat 108 and a passenger seat sensor 130 coupled to the third passenger seat 110. The driver seat sensor 124 and the passenger seat sensor 126 may each include one or a plurality of sensors, such as a weight sensor, a pressure sensor, and one or more seat position sensors that output a measurement signal to the computing system 120. For example, the output of the weight sensor or pressure sensor may be used by the computing system 120 to determine whether or not the respective seat is occupied, and if occupied, a weight of a person occupying the seat. As another example, the output of the one or more seat position sensors may be used by the computing system 120 to determine one or more of a seat height, a longitudinal position with respect to the dashboard 102 and the back seats, and an angle (e.g., tilt) of a seat back of the corresponding seat.

[0034] In some examples, the vehicle 100 further includes a driver seat motor 134 coupled to or positioned within the driver seat 104 and a passenger seat motor 138 coupled to or positioned within the first passenger seat 106. Although not shown, in some embodiments, the back seats may also include seat motors. The driver seat motor 134 may be used to adjust the seat position, including the seat height, the longitudinal seat position, and the angle of the seat back of the driver seat 104 and may include an adjustment input 136. For example, the adjustment input 136 may include one or more toggles, buttons, and switches. The passenger seat motor 138 may be used to adjust the seat position, including the seat height, the longitudinal seat position, and the angle of the seat back of the first passenger seat 106 and may include an adjustment input 140. Theadjustment input 140 may include one or more toggles, buttons, and switches. Although not shown, in some embodiments, the back seats may be adjustable in a similar manner.

[0035] The computing system 120 may receive inputs via the user interface 116 as well as output information to the user interface 116. The user interface 116 may be included in a digital cockpit, for example, and may include a display and one or more input devices. The one or more input devices may include one or more touchscreens, knobs, dials, hard buttons, and soft buttons for receiving user input from a vehicle occupant.

[0036] The computing system 120 includes a processor 142 configured to execute machine readable instructions stored in a memory 144. The processor 142 may be single core or multi-core, and the programs executed by processor 142 may be configured for parallel or distributed processing. In some embodiments, the processor 142 is a microcontroller. The processor 142 may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 142 may be virtualized and executed by remotely-accessible networked computing devices configured in a cloud computing configuration. For example, the computing system 120 may be communicatively coupled with a wireless network 132 via a transceiver 146, and the computing system 120 may communicate with the networked computing devices via the wireless network 132.

[0037] The computing system 120 may include a DMS 147, which may monitor a driver of the vehicle 100. For example, the camera 118 may be located at the front of the vehicle 100 (e g., on dashboard 102) or on a side of the vehicle 100 next to the driver seat 104, and may be positioned to view a face of the driver. The DMS may detect facial features of the driver. In some embodiments, the DMS 147 may be used to retrieve a driver profde of the driver based on the facial features, which may be used to customize a position of the driver seat 104, steering wheel 112, and / or other components or software of the vehicle 100.

[0038] The computing system 120 may include an OMS 148, which may monitor one or more passengers of the vehicle 100. For example, the camera 118 may be located at a side of the vehicle 100 next to one or more of passenger seats 106, 108, and 110, and may be positioned to view a face of a passenger of the vehicle.

[0039] The computing system 120 may include an ADAS 149, which may provide assistance to the driver based at least partially on the DMS 147 For example, ADAS 149 may receive facialexpression data from the DMS 147, and the ADAS 149 may process the facial expression data to provide the assistance to the driver. For example, the ADAS 149 may process the facial expression data to determine whether the driver appears tired, or stressed. In response to detecting a tired or stressed condition of the driver, the ADAS 149 may alert the driver, or play music, or perform a different action to address the tired or stressed condition of the driver.

[0040] As described in greater detail herein, a trained neural network model may be integrated into the DMS 147, OMS 148, or the ADAS 149 that may facilitate detection of facial features or expressions. The trained model may utilize sensor and / or camera data from camera 118 to determine whether ADAS intervention is warranted, for example. The trained neural network model may be trained on driving data of a plurality of drivers. Further, the trained neural network model may be trained using synchronized learning and / or federated learning (FL). Training of the neural network model is described in greater detail below, in reference to FIGS. 5 and 6.

[0041] Additionally or alternatively, the computing system 120 may directly communicate with the networked computing devices via short-range communication protocols, such as Bluetooth®. In some embodiments, the computing system 120 may include other electronic components capable of carrying out processing functions, such as a digital signal processor, a field- programmable gate array (FPGA), or a graphic board. In some embodiments, the processor 142 may include multiple electronic components capable of carrying out processing functions. For example, the processor 142 may include two or more electronic components selected from a plurality of possible electronic components, including a central processor, a digital signal processor, a field-programmable gate array, and a graphics board. In still further embodiments, the processor 142 may be configured as a graphical processing unit (GPU), including parallel computing architecture and parallel processing capabilities.

[0042] Further, the memory 144 may include any non-transitory tangible computer readable medium in which programming instructions are stored. As used herein, the term “tangible computer readable medium” is expressly defined to include any type of computer readable storage. The example methods described herein may be implemented using coded instruction (e.g., computer readable instructions) stored on a non-transitory computer readable medium such as a flash memory, a read-only memory (ROM), a random-access memory (RAM), a cache, or any other storage media in which information is stored for any duration (e.g. for extended period timeperiods, permanently, brief instances, for temporarily buffering, and / or for caching of the information).

[0043] Computer memory of computer readable storage mediums as referenced herein may include volatile and non-volatile or removable and non-removable media for a storage of electronically formatted information, such as computer readable program instructions or modules of computer readable program instructions, data, etc. that may be stand-alone or as part of a computing device. Examples of computer memory may include any other medium which can be used to store the desired electronic format of information and which can be accessed by the processor or processors or at least a portion of a computing device. In various embodiments, the memory 144 may include an SD memory card, an internal and / or external hard disk, USB memory device, or a similar modular memory.

[0044] Further still, in some examples, the computing system 120 may include a plurality of sub-systems or modules tasks with performing specific functions related to performing image acquisition and analysis. As used herein, the terms “system,” “unit,” or “module” may include a hardware and / or software system that operates to perform one or more functions. For example, a module, unit, or system may include a computer processor, controller, or other logic-based device that performs operations based on instructions stored on a tangible and non-transitory computer readable storage medium, such as a computer memory. Alternatively, a module, unit, or system may include a hard-wired device that performs operations based on hard-wired logic of the device. Various modules or units shown in the attached figures may represent the hardware that operates based on software or hardwired instructions, the software that directs hardware to perform the operations, or a combination thereof.

[0045] FIG. 2 shows a transnationally distributed system 200 for leveraging connected vehicle data and resources from a plurality of local servers of a different geographical area, having different data protection laws that prevent sharing of personal data between the different geographical areas. Each local server may be a cloud network. Each local server may be considered a node of an FL system, as described in greater detail below.

[0046] Each local server (e.g., node) may include resources (e.g., memory, processor(s)) that may be allocated to update a local neural network model, and store and execute instructions that update the local neural network model at the local server based on vehicle data collected from aplurality of vehicles from different geographical locations, (e.g., shown in FIGS. 3 and 4), without sharing the vehicle data.

[0047] Each local server may store data that may be used for initializing and training one or more local neural network models. Transnationally distributed system 200 includes a first local server 212 for a first zone 202, a second local server 214 for a second zone 204, a third local server 216 for a third zone 206, and a fourth local server 218 for a fourth zone 208. However, it may be understood that other embodiments of the transnationally distributed system 200 may include less or additional zones and corresponding local servers without departing from the scope of the present disclosure.

[0048] The first local server 212, the second local server 214, the third local server 216, and the fourth local server 218 may each be one local server of the plurality of local servers. Each local server of the plurality of local servers may be communicatively coupled to a local database that stores vehicle data, such as image data or video data, collected within a specific geographical location. The first zone 202, the second zone 204, the third zone 206, and the fourth zone 208 may correspond to specific geographical locations. As one example, the first zone 202 may be associated with the United States, second zone 204 may be associated with India, third zone 206 may be associated with China, and fourth zone 208 may be associated with the United Kingdom.

[0049] In this way, the first local server 212 may receive vehicle data from a plurality of vehicles 222 located in the first zone 202, the second local server 214 may receive vehicle data from a plurality of vehicles 224 located in the second zone 204, the third local server 216 may receive vehicle data from a plurality of vehicles 226, located in the third zone 206, and the fourth local server 218 may receive data from a plurality of vehicles 228 located in the fourth zone 208. The vehicle data collected at a specific geographical location may be used as training data for a local neural network model trained at a local server in the specific geographical location. The plurality of local servers may be communicatively coupled, enabling each local server to communicate data, including but not limited to a plurality of sets of accumulated gradients, batch sizes of training data, and other data to other local servers to train a plurality of local neural network models using synchronized learning.

[0050] Although the plurality of local servers may be communicatively coupled, the local servers may be configured according to a set of instructions that prevent certain data from being exchanged between the plurality of servers. For example, in one embodiment, the vehicle data ofa specific geographical location may not be accessed by a different local server located in a different geographical location. In particular, the first local server 212 for the first zone 202 (e.g., located in United States), may not directly receive vehicle data from a plurality of vehicles 224 located in the second zone 204 (e.g., located in India), Instead, the plurality of local servers may indirectly receive vehicle data from different geographical locations in the form of a plurality of sets of accumulated gradients calculated at each local server.

[0051] FIG. 3 A shows a cross-silo FL system 300 based on a transnationally distributed system, which may be the transnationally distributed system described above with respect to FIG. 2. The transnationally distributed system includes a plurality of vehicles 222 located in the first zone 202, a plurality of vehicles 224 located in the second zone 204, and a plurality of vehicles 226 located in the third zone 206. Vehicle data for the plurality of vehicles 222, 224, and 226 may be stored in a first local database 302, a second local database 304, and a third local database 306, respectively. It may be understood that the transnationally distributed system may comprise additional or fewer zones with corresponding pluralities of vehicles and local databases without departing from the scope of the present disclosure. Cross-silo FL system 300 includes a plurality of local servers, such as a first local server 212, a second local server 214, and a third local server 216, which may be located within first zone 202, second zone 204, and third zone 206, respectively. The first local server 212, the second local server 214, and the third local server 216 may receive batches of training data from local databases 302, 304, and 306, respectively. For training a local neural network model, a first training dataset 303 may be generated from vehicle data from the plurality of vehicles 222 and stored in the first local database 302; a second training dataset 305 may be generated from vehicle data from the plurality of vehicles 224 and stored in second local database 304; and a third training dataset 307 may be generated from vehicle data from the plurality of vehicles 226 and stored in the third local database 306.

[0052] Each batch of training data received at each local server may train one local neural network of a plurality of local neural network models. For example, the first training dataset 303 may be used to train a first local neural network model 312; the second training dataset 305 may be used to train a second local neural network model 314; the third training dataset 307 may be used to train a third local neural network model 316. The first local neural network model 312, second local neural network model 314, and third local neural network model 316 may be trained concurrently using synchronized learning, as described in greater detail below.

[0053] During a computation stage 320 of training, during each iteration of the training of the first local neural network model 312, second local neural network model 314, and third local neural network model 316, a first set of accumulated gradients calculated during a backward propagation stage of the training may be stored at the first local server 212, a second set of accumulated gradients calculated during a backward propagation stage of the training may be stored at the second local server 214, and a third set of accumulated gradients calculated during a backward propagation stage of the training may be stored at the third local server 216. Accumulation of the gradients is described in greater detail below in reference to FIG. 5.

[0054] Further, during a communication stage 321 of training, during each iteration of the training of the first local neural network model 312, second local neural network model 314, and third local neural network model 316, the first set of accumulated gradients from the first local server 212 may be transmitted to the second local server 214 and the third local server 216; the second set of accumulated gradients from the second local server 214 may be transmitted to the first local server 212 and the third local server 216; and the third set of accumulated gradients from the third local server 216 may be transmitted to the first local server 212 and the second local server 214.

[0055] Training of the first local neural network model 312, the second local neural network model 314, and the third local neural network model 316 may include aggregating the first set of accumulated gradients, the second set of accumulated gradients, and the third set of accumulated gradients at each of the first local server 212, the second local server 214, and the third local server 216. At the end of each iteration of the training of the first local neural network model 312, the second local neural network model 314, and the third local neural network model 316, parameters of the first local neural network model 312, the second local neural network model 314, and the third local neural network model 316 may be updated based on the same set of aggregated gradients. Aggregation of the gradients is described in greater detail below in reference to FIG. 5.

[0056] FIG. 3B shows cross-silo FL system 300 during a validation stage, wherein the accuracy of the plurality of local neural model network models is verified. Verifying the accuracy of the first local neural network model 312, the second local neural network model 314, and the third local neural network model 316 includes testing a first performance of the first local neural network model 312 based on a local validation dataset 352 collected at the first local server 212; testing a first performance of the second local neural network model 314 based on a local validationdataset 354 collected at the second local server 214; and testing a first performance of the third local neural network model 316 based on a local validation dataset 356 collected at the third local server 216.

[0057] Local validation dataset 352 may include data from vehicles 222 of the first zone 202; local validation dataset 354 may include data from vehicles 224 of the second zone 204; and local validation dataset 356 may include data from vehicles 226 of the third zone 206, such that no validation data is shared between the different zones. Thus, the performance of each local neural network is assessed based on local data. In other words, each neural network model is trained on a local server associated with a geographical region, such that a first set of training data from a first geographical region does not include data from a second geographical region, and the second set of training data does not include data from the first geographical region. Additionally, training of each neural network model on a respective local server is performed without receiving or exchanging training data between geographical regions.

[0058] It may be understood that the examples provided are exemplary and do not limit the scope of the present disclosure. The examples provided may include a cross-silo FL system 300 with additional or less local servers in the plurality of local server. As one example, the cross-silo FL system 300 may include a fourth local server where a fourth neural network model is trained using synchronous learning.

[0059] Referring to FIG. 4, a local server 400 of a cross-silo FL system is shown, such as the cross-silo FL system 300 of FIGS. 3A and 3B. Local server 400 includes a node 402 of the crosssilo FL system for training a neural network on vehicle data of a transnationally distributed system, such as transnationally distributed system 200 of FIG. 2.

[0060] Node 402 includes one or more processors 404 configured to execute machine readable instructions stored in a non-transitory memory 406. The processor(s) 404 may be any suitable processor, processing unit, or microprocessor, for example. The processor(s) may be a multiprocessor system, and, thus, may include one or more additional processors that are identical or similar to each other and that are communicatively coupled via an interconnection bus. Processor 404 may be single core or multi-core, and the programs executed thereon may be configured for parallel or distributed processing. In some embodiments, the processor 404 may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and / or configured for coordinated processing. Tn some embodiments, one ormore aspects of the processor 404 may be virtualized and executed by remotely-accessible networked computing devices configured in a cloud computing configuration.

[0061] The non-transitory memory 406 may include one or more data storage structures, such as optical memory devices, magnetic memory devices, or solid-state memory devices, for storing programs and routines executed by the processor(s) of the local server to carry out various functionalities disclosed herein. The memory may include any desired type of volatile and / or nonvolatile memory such as, for example, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, read-only memory (ROM), etc.

[0062] Non-transitory memory 406 may include an accumulated gradients module 408, an aggregated gradients module 410, a neural network module 412, a training module 414, and a communication module 416. In some embodiments, the non-transitory memory 406 may include components disposed at two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the non-transitory memory 406 may include remotely-accessible networked storage devices configured in a cloud computing configuration.

[0063] Neural network module 412 may include one or more trained and / or untrained neural networks and may further include various data, or metadata pertaining to the one or more neural networks stored therein. The neural networks stored in neural network module 412 may be trained using synchronized learning as part of an FL learning system, as described in greater detail herein. Neural network module 412 may include a plurality local neural network model parameters, batch size, weighting coefficients, global learning rate, and the like. Training module 414 may comprise instructions for training one or more of the neural networks stored in neural network module 412. In particular, training module 414 may include instructions that, when executed by the processor 404, cause node 402 to conduct one or more of the steps of method 500 for training the neural network model to detect facial features of a drivers of vehicles, using data collected from vehicles of different geographical regions participating in the FL learning system, discussed in more detail below in reference to FIG. 5.

[0064] In various embodiments, the facial feature detection performed by the trained neural network model may be used in a DMS / OMS system of the vehicles. For example, the DMS / OMS system may rely on the trained neural network model to determine a driver profile of a driver of a vehicle, discussed in more detail below in reference to FIG. 7. As another example, an ADAS ofthe vehicles may use the DMS / OMS system including the trained neural network model to determine whether to alert the driver to a road condition. In some embodiments, training module 414 includes instructions for implementing one or more gradient descent algorithms, applying one or more loss functions, and / or training routines, for use in adjusting parameters of the one or more neural networks.

[0065] Accumulated gradients module 408 may include a plurality of sets of accumulated gradients calculated during training of the neural network model. Aggregated gradients module 410 may include aggregations of sets of accumulated gradients received from other neural network models trained on other servers in other regions of the transnationally distributed system.

[0066] The node 402 may include a communication module 416. The communication module 416 may facilitate transmission of electronic data (e.g., accumulated gradient data, batch size data, learning rates, etc.) used or generated during training of the neural network model to the other local servers having the other neural network models of other geographical locations of the transnationally distributed system.

[0067] Communication via the communication module 416 can be implemented using one or more protocols. The communication module can be a wired interface (e.g., a data bus, a Universal Serial Bus (USB) connection, etc.) and / or a wireless interface (e.g., radio frequency, infrared, near field communication (NFC), etc.). For example, the communication module may communicate via wired local area network (LAN), wireless LAN, wide area network (WAN), etc. using any past, present, or future communication protocol (e.g., BLUETOOTH™, USB 2.0, USB 3.0, etc.).

[0068] Node 402 may be operably / communicatively coupled to a user input device 432 and display device 434. User input device 432 may comprise one or more of a touchscreen, a keyboard, a mouse, a trackpad, a motion sensing camera, or other device configured to enable a user to interact with and manipulate data within node 402. Display device 434 may include one or more display devices utilizing virtually any type of technology. In some embodiments, display device 434 may comprise a computer monitor. Display device 434 may be combined with processor 404, non-transitory memory 406, and / or user input device 432 in a shared enclosure, or may be peripheral display devices and may comprise a monitor, touchscreen, projector, or other display device known in the art, which may enable a user to interact with various data stored in non- transitory memory 406, such as to adjust batch size and a weighting coefficient of a set of accumulated gradients.

[0069] It may be understood that node 402 shown in FIG. 4 is for illustration, not for limitation. Another appropriate node of an FL system may include more, fewer, or different components.

[0070] Turning now to FIG. 5, an example method 500 is shown for training a neural network model using synchronized learning, within an FL system, such that the neural network model may be trained on data that is stored in different regions of a transnationally distributed system (e.g., transnationally distributed system 200 and / or 300 of FIGS. 2 and 3A, respectively). In the embodiment described in reference to FIGS. 5, 6, and 7, the transnationally distributed system is a system of vehicles, where vehicle data is collected at each region of the system of vehicles, and not shared with other regions. The vehicle data includes image data acquired from one or more cameras (e.g., cameras 118 of FIG. 1) installed in each vehicle of the system of vehicles, where connected data availability is assumed.

[0071] Connected data availability may include communication with a connected vehicle population or a plurality of databases and / or servers using a V2V network, a V2I network, a cloud network, and the like. The image data is collected at each vehicle and transmitted to a local server in the region (e.g., local servers 212, 214, 216, and 218). The image data collected within each region may be processed at the local server of the region, where each local server acts as a node in the FL system, such as node 402 of FIG. 4. Instructions for carrying out at least part of method 500 may be stored in non-transitory memory 406 and executed by processor 404 of node 402 of FIG. 4.

[0072] In the embodiment described in FIGS. 5, 6, and 7, the image data is used to train the copies of the neural network model to detect facial features and / or expressions of drivers or occupants of the vehicles. The trained neural network model may be installed in a plurality of vehicles. For example, the trained neural network model may be incorporated into a DMS / OMS of a vehicle, the detected facial features may be used by an ADAS system of the vehicle, as described in greater detail below in reference to FIG. 7.

[0073] The image data may be protected by privacy laws that prevent the image data from being transferred outside the region. However, the image data from all the different regions may be used to train a plurality of copies of the neural network model stored in the different regions, without transferring the image data. It may be appreciated that vehicle data is used for illustrative reasons, and that method 500 may be applied to other types of training data collected at other transnationally distributed systems without departing from the scope of this disclosure.

[0074] During synchronized learning, a copy of the neural network model is installed at each node (e.g., at each local server) of the FL system. Each copy of the neural network model is trained on local data, but gradients calculated during each iteration of training at each copy of the neural network model may be shared across models. The gradients may be used to adjust training parameters of each of the copies of the neural network model.

[0075] At 502, the method includes performing forward and back propagation on a local copy of the neural network model using a plurality of mini-batches of training data, and accumulating (e.g., storing) gradients from each mini-batch. Each mini-batch may have a fixed batch size. Rather than training the local copy of the neural network model (also referred to herein as the local neural network model) with a larger batch of training data, the plurality of smaller mini-batches of training data may be utilized to reduce computation load of the local server when training the local neural network model, while increasing training efficiency and increase accuracy of the local neural network model. Training of the plurality of mini-batches of training data may include accumulating a plurality of sets of gradients for a plurality of parameters, according to the method described with respect to FIG. 6.

[0076] In one embodiment, training the local copy of the neural network model includes inputting training pairs of data into the neural network, where each training pair includes an input image of a face of a driver of a vehicle, and a ground truth classification of the driver. For example, the ground truth classification may be an identification of the driver, which may be used to load a driving profile of the driver during an inference stage. In other examples, the ground truth classification may be a degree of tiredness of the driver, or a known emotional state of the driver, or a different characteristic of the driver.

[0077] During forward propagation, input image data may be entered into an input layer of the neural network. In various embodiments, the neural network may be a convolutional neural network (CNN) with various hidden layers. For example, each pixel of the input image may be inputted into an input node of the neural network. The image data may be multiplied by parameters (e.g., weights) of the neural network at each input node to generate an output, which may then be inputted into a plurality of nodes of a first hidden layer of the various hidden layers. Outputs of each hidden layer may be inputted into a next hidden layer, until a final output of the neural network (e.g., a classification) is generated at an output layer of the neural network.

[0078] During backpropagation, the final output generated at the output layer is compared to the ground truth classification using one or more loss functions, and a calculated loss is backpropagated through the layers of the neural network to determine gradients at each node of each layer of the neural network. The gradients may be accumulated and stored to be used at a later stage to update the parameters, as described below.

[0079] At 504, the method includes sending accumulated gradients to other local neural network models at other nodes of the FL system. As described herein with regards to the systems and methods, a plurality of local servers may be communicatively coupled, and when executed, instructions stored in a training module, such as training module 414 of FIG. 4, may cause the processor to transmit a plurality of sets of accumulated gradients stored in non-transitory memory to the other local servers of the plurality of local servers.

[0080] At 506, the method includes receiving accumulated gradients from the other local neural network models at other nodes of the FL system. In other words, the plurality of sets of accumulated gradients used during training may be exchanged between the plurality of local servers.

[0081] At 508, the method includes aggregating the received accumulated gradients of each local neural network model. Instructions stored in the training module may be executed by the processor to cause the processor to aggregate the plurality of sets of gradients received from the plurality of local servers. Gradient aggregation based on the sets of accumulated gradients of the plurality of local neural network models may be performed according to the following equation:wherein g is the aggregated gradients, gcis the accumulated gradients, wmis a weighting coefficient of the accumulated gradient from the m111local server. The aggregated gradients may be stored in at least one address in non-transitory memory of the plurality of local servers, such as aggregated gradients module 410 of FIG. 4. The weighting coefficient may be an aggregation weighting coefficient that indicates a factor by which accumulated gradients of a given neural network model may be multiplied, to increase an efficiency of training.

[0082] For example, a transnationally distributed system may include two local servers, where training may include aggregating a first set of accumulated gradients of a first local neural network model with a second set of accumulated gradients of a second neural network model received from a second server, the second neural network model identical to the first neural network model (e.g.,copies of the same neural network), the second neural network trained on a second plurality of mini-batches of a second training dataset on the second server. Aggregating the first set of accumulated gradients with a second set of accumulated gradients may include summation of a product of a first weighting coefficient and the first set of accumulated gradients and a product of a second weighting coefficient and the second set of accumulated gradients, the second set of accumulated gradients received from the second server.

[0083] In one embodiment, the product of the first weighting coefficient and the first set of accumulated gradients may be computed locally. In another embodiment, the first weighting coefficient may be sent, for example, from the first server and received by the second server, and the second weighting coefficient may be sent from the second server and received by the first server. After all the accumulated gradients are sent by the plurality of servers, the product of the first weighting coefficient and the first set of accumulated gradients as well as the product of the second weighting coefficient and the second set of accumulated gradients may be computed globally. As one example, the first weighting coefficient may be 0.5, and the second weighting coefficient may be 0.5.

[0084] In some embodiments, the first set of training data of the first local neural network model may be smaller than the second set of training data of the second local neural network model. A smaller first weighting coefficient may be selected to apply to the accumulated gradients from the first set of training data during aggregation, and a larger second weighting coefficient may be selected to apply to the accumulated gradients from the second set of training data during aggregation. It may be understood that the weighting coefficient may be adjusted automatically (e.g., without human input) based on validation results. The smaller value for the first weighting coefficient may be selected to prevent catastrophic interference. In other words, the smaller value for the first weighting coefficient may be selected to prevent other local neural network models from forgetting the first set of training data, ensuring training of the other local neural network models is based on the first set of training data. In this way, the local network models do not forget how to do previously learned tasks when new tasks are encountered. Additionally, the larger value of the second weighting coefficient is selected to increase the convergence speed with the second set of training data of the second local neural network model.

[0085] At 510, the method includes updating parameters (e.g., weights) of the local neural network model based on the aggregated gradients. Instructions stored in the training module maybe executed by the processor to cause the processor to update the parameters based on aggregated gradients stored in non-transitory memory. The parameters may be updated according to the following equation:9 = 0 ~ Vbs - g (2) wherein gbsis the learning rate generated by a learning rate scheduler that is scaled based on batch size, g is the aggregated gradients, and 6 is a parameter of the local neural network model. There may be a plurality of parameters, each parameter with their own respective aggregated gradients based on their own respective sets of accumulated gradients. The instructions may further include storing the updated parameters in non-transitory memory for each local server of the plurality of local servers.

[0086] For example, in the transnational^ distributed system including two local servers, training may include updating parameters of the first local neural network model based on the aggregated gradients at the first local server. Updating parameters of the first local neural network may include updating each parameter of the parameters based on an aggregated gradient associated with the parameter and the learning rate, using equation 2 at the first local server. Similarly, parameters of the second local neural network model may be updated based on the aggregated gradients at the second local server, and updating parameters of the second local neural network model may include updating each parameter of the parameters based on an aggregated gradient associated with the parameter and the learning rate, using the same equation at the second local server. The set of aggregated gradients, batch sizes, and learning rates may be the same for both the first and second local neural networks, whereby a same set of updated parameters may be generated for the first local neural network and the second local neural network for a subsequent iteration of training.

[0087] At 512, the method includes checking an accuracy of the local neural network model via a local validation dataset after a pre-determined number of iterations of training. In other words, across the FL system, each local neural network model of the plurality of local servers may assess the local neural network model accuracy via a local validation set, as described above with respect to FIG. 3B, after a pre-determined number of iterations. The local validation sets may include validation pairs similar to the training pairs of a respective training dataset. For example, each training pair of a local validation set may include an image taken by a camera of a vehicle in a respective geographical region, and a corresponding ground truth classification.

[0088] At 514, the method includes determining whether model accuracy on a local validation dataset is decreasing over a number of iterations of the local neural network model using the local validation dataset. For example, the model accuracy may be checked on the local validation set every three epochs. If an error rate of the local neural network model continually increases over the number of iterations, the accuracy may be determined to be decreasing. If it is determined at 514 that the accuracy of the local neural network model is decreasing, the method proceeds to 516.

[0089] At 516, as a result of the model accuracy decreasing, the method includes determining a new batch size, a new weighting coefficient for a set of accumulated gradients of the local neural network model, and a new global learning rate based on the new batch size. In other words, to increase local neural network model accuracy and balance training speed, the batch size may be adjusted by a pre-determined amount in the local server. In some embodiments, the batch size may be automatically (e g., without human input) decreased by the pre-determined amount for the local neural network model. The learning rate may be adjusted based on the new batch size using the learning rate scheduler described above. Different learning rate schedulers may be used including LineaLR, Constant LR, Exponential, and CosineAnnealingLR. The running frequency of the learning rate schedulers may be every epoch, every n iterations, etc.

[0090] During initialization of training, the weighting coefficient for each local server may be set. The values of the weighting coefficients may be the same or different. As one example, the weighting coefficient may be one divided by the number of local server. Each local server may have a target accuracy for the local neural network model. When evaluating local neural model accuracy with the local validation dataset during training, after the local neural network model converges at a specific local server, the model accuracy is compared with the target model accuracy for that specific local server. If the model accuracy is not within a threshold accuracy, the weighting coefficient for the specific local server may be decreased to increase model accuracy and increase convergence rate during training.

[0091] Since the synchronized training is used to train the local neural network model, all local neural network models have the same parameters and local neural network models are also global models. The global models may continuously learn and achieve target accuracies on all the local validation datasets. The accuracies between different local validation datasets may be compared. The accuracies between models trained based on global data and models trained based on local data may be compared. Models trained based on global data may have equal or greater accuracy.

[0092] Adjustments to the batch size, weighting coefficient, and global learning rate may be performed according to a pre-determined set of instructions stored in the training module, in some embodiments. In other embodiments, manual adjustments of the batch size, weighting coefficient, and global learning rate may be performed via a user input device communicatively coupled to a user input device, as shown in FIG. 4.

[0093] At 517, the method includes synchronizing the batch sizes with other local neural network models across the FL system. The adjusted batch sizes for each local server are sent to the other local servers and each local server receives the adjusted batch sizes from the other local servers. The minimum adjusted batch size of all the adjusted batch size from the local servers is selected as the new batch size for each local server. For example, for a transnationally distributed system including at least two local servers, the pre-determined amount by which a batch size of a first training dataset at a first local server is adjusted may be different than the pre-determined amount by which a batch size of a second training data set at a second local server is adjusted. The adjusted batch size of both the first set of training data at the first local server and the second set of training data may be the minimum of the adjusted batch size of the first set of training data and the adjusted batch size of the second set of training data.

[0094] At 518, the method includes applying an adjusted batch size, new weighting coefficient, and updated learning rate for subsequent iterations of training, and the method proceeds back to 502 to continue training the local neural network model.

[0095] Returning to 514, if it is determined that the accuracy of the local neural network model is not decreasing, meaning that the local neural network is continuing to increase, the method proceeds to 520. At 520, the method includes determining whether a termination condition of the local neural network has been achieved. In some embodiments, the termination condition may be the model being trained for a maximum number of epochs or iterations. The maximum number of epochs or iterations may be set prior to training. In other embodiments, the termination condition may include a threshold accuracy being achieved. The threshold accuracy may depend on a type of training data used. For example, if the training data includes images of faces of drivers and associated driver identifications, the threshold accuracy may be a correct driver identification rate of 95%. If the model accuracy at a specific local server is continuously increasing (e.g., during a first validation and second validation), the weighting coefficient at that specific local server maybe decreased by a pre-determined amount until the local neural network model has converged at the specific local server.

[0096] If the termination condition has not been achieved, then the method proceeds back to 502, and the method includes continuing to train the local neural network model as described above. Alternatively, if it is determined at 520 that the termination condition has been achieved, the method proceeds to 522.

[0097] At 522, the method includes deploying the trained local neural network in the region corresponding to the node and local server. Deployment of the trained local neural network is described in greater detail below in reference to FIG. 7. Method 500 ends.

[0098] It may be understood that training of different local neural network models may have different convergence rates with regards to accumulating sets of gradients and aggregating sets of accumulated gradients. Since the topology of the transnationally distributed system is defined before the initiation of training, timing issues of non-uniform distributed training data among the plurality of local servers is addressed by configuring the system, according to a set of instructions, to wait for calculation of sets of accumulated gradients in each local server to be completed prior to aggregating the accumulated gradients for each iteration. As such, each local neural network model may be trained on the plurality of mini-batches at the respective local server at different rates without affecting aggregation of the accumulated gradients and updating the plurality the local neural network models prematurely.

[0099] Turning now to FIG. 6, an exemplary method 600 is shown for collecting accumulated gradients during training of a neural network model at a node of a cross-silo FL system, such as cross-silo FL system 300 of FIGS. 3A and 3B. In various embodiments, the local server may be a cloud-based server, such as local server 400 of FIG. 4. Instructions for carrying out method 600 may be stored on and executed by the cloud-based server. In one example, method 600 is executed as part of method 500 of FIG. 5 described above.

[0100] At 602, the method includes acquiring a larger batch of training data. Instructions stored in a training module, such as training module 414 of FIG. 4, and executed by the processor may cause the processor enable a local server to receive training data by accessing a local database wherein the larger batch of training data is stored at a local server. The local database may be for a specific geographical location, such as, for example, China. The larger batch of training data may include vehicle data, such as image or video data, for the specific geographical location.

[0101] In some examples, the image data or video data may include images or videos of a cabin of a vehicle, wherein a subject of the image or video is a vehicle operator and / or occupant(s). In other examples, the image data or video data may include images or videos of an exterior environment surrounding the vehicle

[0102] As one example, for a transnationally distributed system that includes at least two local servers, the larger batch of training data may include a first set of training data on a first local server and a second set of training data on a second local server. The first training dataset is stored in a first local database communicatively coupled to the first server and comprises training data collected at a first geographical location, and the second training dataset is stored in a second local database communicatively coupled to the second server and comprises training data collected at a second geographical location. The first training dataset comprises image data and / or video data from a first plurality of vehicles located in the first geographical location and the second training dataset comprises image data and / or video data from a second plurality of vehicles located in the second geographical location. Instructions may be executed in each of the first local server and the second local server to enable the first local server to access the first set of training data and the second local server to access the second set of training data.

[0103] It may be further noted that training of the first neural network model at the first local server and training of the second neural network model at the second local server is performed without exchanging training data between the first server and the second server. As such, training data collected and stored in the first local database at the first geographical location is not accessible by the second server and training data collected and stored in the second local database at the second geographical location is not accessible by the first server due to the first server and the second server being configured to not exchange training data between the first server and the second server.

[0104] At 604, the method 600 includes splitting the larger batch into smaller mini-batches. Instructions stored in the training module and executed by the processor may cause the processor to split the larger batch of training data into the plurality of mini-batches of training data, where the mini-batch size is fixed during training. As described above with FIG. 5, the local server stores information in memory of the local server (e g., in the training module 414) regarding the local neural network model, such as, batch size or an adjusted batch size. As such, the instructions splitthe larger batch of training data into a plurality of mini-batches of training data based on the fixed batch size.

[0105] At 606, the method includes loading one mini-batch of localized training data into the local neural network model. Instructions stored in the training module and executed by the processor may cause the processor to load one mini-batch of training data into the local neural network model at the local server.

[0106] At 608 the method includes collecting a plurality of sets of gradients for a respective plurality of parameters of a local neural network model. Instructions stored in the training module and executed by the processor may cause the processor to train the local neural network model by performing forward propagation and backward propagation using the mini-batch of training data, as described above in reference to FIG. 5. Backward propagation of the local neural network model at the local server calculates a plurality of sets of gradients for the respective plurality of parameters of the local neural network model. The gradients may be collected according to the following equation:wherein n is the mini-batch of training data at each local server, J (0) is a loss function, m is the index of local server, and gmis the corresponding gradient.

[0107] At 610, the method includes calculating accumulated gradients of the local neural network model. A plurality of sets of accumulated gradients may be calculated by accumulating a plurality of sets of gradients calculated by the local server. Gradient accumulation may be performed according to the following equation: g%c= g%c+ gm(4) wherein g^cis the accumulated gradients, gmis the corresponding gradients, and m refers to the mthlocal server. In this way, accumulating a set of gradients may include performing a summation of the plurality of sets of gradients that have been calculated. In other words, a set of gradients calculated during the backward propagation stage when training the local neural network model with the mini-batch of training data may be summed with other sets of gradients calculated during the backward propagation stage when training the local neural network using other mini-batches of the plurality mini-batches of training. The plurality of sets of gradients for the respective plurality of parameters may be stored in an address in non-transitory memory of a local server, such as accumulated gradients module 408 of FIG. 4.

[0108] At 612, the method 600 includes determining whether there are additional mini -batches of training data to train the local neural network model. Instructions stored in the training module and executed by the processor may cause the processor to utilize batch size of the mini-batches to determine the amount of total mini-batches in the plurality of mini-batches. In this way, the instructions may be able to determine the amount of mini-batches of localized training data that have been trained and the amount of mini-batches of localized training data that have not been trained.

[0109] Responsive to there being additional mini -batches of training data to train the local neural network model, the method 600 includes loading one mini -batch of localized training data into the local neural network model at 606. As such, the method 600 continues to collect a plurality of sets of gradients for the respective plurality of parameters and performing gradient accumulation to obtain a set of accumulated gradients for each local neural network model of the plurality of local servers until there are no remaining mini-batches of localized training data. Responsive to there being no additional mini-batches of training data to train the local neural network model, the method 600 then ends.

[0110] As illustrated in FIG. 7, an exemplary method 700 is shown for determining whether to adjust an operation of a vehicle based on an output of a trained neural network model of an ADAS of the vehicle. The trained neural network model may be a trained version of a local neural network model trained according to the methods described in FIGS. 5 and 6. The vehicle may be operating within a geographical location of a transnationally distributed system of vehicles, such as transnationally distributed system 200 of FIG. 2. Because the neural network model is trained using the cross-silo FL system described herein (e.g., cross-silo FL system 300), a same neural network sharing identical parameters may be used to perform inferences in a plurality of geographical regions of the transnationally distributed system of vehicles independently of neural network models in other geographical regions without relying on a network. Instructions for carrying out method 700 may be stored in non-transitory memory and executed by a processor of a vehicle computing system, such as computing system 120 of FIG. 1, and / or an ADAS of the vehicle, such as ADAS 149 of FIG. 1.[OHl] Method 700 begins at 702, where the method includes estimating and / or measuring vehicle operating conditions. Vehicle operating conditions may be estimated based on one or more outputs of various sensors of the vehicle (e g., such as oil temperature sensors, engine velocity orwheel velocity sensors, torque sensors, etc.). Vehicle operating conditions may include engine velocity and load, vehicle velocity, transmission oil temperature, exhaust gas flow rate, mass air flow rate, coolant temperature, coolant flow rate, engine oil pressures (e.g., oil gallery pressures), operating modes of one or more intake valves and / or exhaust valves, electric motor velocity, battery charge, engine torque output, vehicle wheel torque, etc. Estimating and / or measuring vehicle operating conditions may include determining whether the vehicle is being powered by an engine or an electric motor.

[0112] Estimating and measuring the vehicle operating conditions may also include determining a proximity of other vehicles to the vehicle, in a lane of traffic of the vehicle or in other lanes of traffic, via one or more cameras and / or sensors positioned on or in the vehicle (e.g., camera 118 of FIG. 1). The vehicle operating conditions may include road conditions (e.g., whether the road is paved or unpaved, wet, icy, etc ), weather conditions (e.g., whether it is raining or snowing), and / or other external conditions of the vehicle (e.g., whether it is daytime or night time, etc.).

[0113] At 704, the method includes receiving image data of a driver of the vehicle from a DMS of the vehicle. In various embodiments, the image data may be received from a camera installed in a cabin of the vehicle, such as camera 118 of FIG. 1. For example, the DMS may capture images of the driver as the driver views an upcoming segment of road, including traffic and / or other vehicles ahead of the vehicle. The DMS may capture images of the driver looking out of one or more side windows of the vehicle and / or glancing at the rear view mirror. The DMS may capture images of the driver interacting with other occupants of the vehicles.

[0114] At 706, the method includes processing the image data to determine whether to adjust an operation of the vehicle based on the image data. In the depicted embodiment, the ADAS may adjust the operation of the vehicle in response to a predicted level of alertness of the driver decreasing below a threshold level of alertness. For example, the level of alertness of the driver may be predicted based on facial expressions of the driver; postural indications of tiredness, such as slouching or head dipping; periods of time with no movement of the driver; detected movement or slipping of eyelids of the driver; and so on.

[0115] At 708, processing the image data to determine whether to adjust the operation of the vehicle includes using a trained neural network to predict the level of alertness. The image data may be inputted into the trained neural network, and the neural network may output a predicted alertness of the driver. For example, the predicted alertness may be a value between 0.0 and 1 . 0,where 1.0 indicates a highest level of alertness, and 0.0 indicates a lowest level of alertness. As described above in reference to FIGS. 5 and 6, the trained neural network may be trained on facial images including images of drivers with a range of levels of alertness, of drivers collected from various geographical regions, without sharing the facial images across the geographical regions.

[0116] In other embodiments, the ADAS may adjust the operation of the vehicle in response to a different predicted state of the driver, or an identification of the driver, or different characteristics of the driver’s behavior. It may be appreciated that the examples and embodiments described herein are for illustrative purposes, and the trained neural network model may be used by the ADAS or the DMS to adjust the operation of the vehicle in other ways without departing from the scope of this disclosure.

[0117] At 710, the method includes determining whether the predicted level of alertness outputted by the trained neural network model is below a threshold level of alertness. For example, the threshold level of alertness may be .5, where if the predicted level of alertness is .7, the driver is determined to be above the threshold level of alertness (e.g., alert enough to drive without ADAS intervention). Alternatively, if the predicted level of alertness is .4, the driver is determined to be below the threshold level of alertness (e.g., not alert enough to drive without ADAS intervention).

[0118] If at 710 it is determined that the predicted level of alertness output by the trained neural network model is not below the threshold level of alertness, the method proceeds to 712. At 712, the method includes continuing operation of the vehicle without an intervention of the ADAS system, and the method ends. Alternatively, if at 710 it is determined that the predicted level of alertness is below the threshold level of alertness, the method proceeds to 714.

[0119] At 714, the method includes adjusting the operation of the vehicle based on the predicted level of alertness. In other words, the ADAS may perform one or more actions to increase the level of alertness of the driver. For example, the ADAS may notify the driver via an audible sound alerting that the driver appears to be tired, or the ADAS may play an audio file with music or other content that may increase the level of alertness of the driver. In some embodiments, the ADAS may adjust an acceleration or a speed of the vehicle in response to the level of alertness being below the threshold level of alertness, or perform a different action or operation of the vehicle.

[0120] In this way, a trained neural network model may be integrated within an ADAS of a vehicle that may be used by the ADAS to determine whether and how to intervene during driving based on a condition of a driver detected or predicted from facial features or expressions of thedriver, where the facial features and / or expressions of the driver are captured via a DMS system of the vehicle. The neural network model may be trained using the cross-silo FL system for synchronized learning described herein. By using the proposed cross-silo FL system, an accuracy of the trained neural network model may be increased by leveraging image data from a larger and more diverse population of drivers than would be obtained from image data collected within a single geographical region, while respecting personal data privacy protections that may be different for different geographical regions. The method 700 then ends.

[0121] The neural network model may be trained using synchronized learning within a Federated Learning (FL) system. The FL system may comprise a plurality of local servers (e.g., nodes) wherein for each local server in the FL system, a local neural network model is trained at the respective local server with vehicle data received from the geographical location wherein the local server resides. A batch of vehicle data (e g., training data) may be received at the local server where the training data is split into a plurality of mini-batches of training data. In this way, bandwidth limitations may be reduced since a single mini-batch may be trained at a local server at a time. For each local server, the local neural network model may continue to load a single minibatch of training data and accumulate a plurality of gradients for a plurality of parameters during a backward propagation stage until there are no remaining mini-batches of training data. The accumulated gradients may be sent to the plurality of local servers to be aggregated. Each local server then uses the aggregated gradients received from the plurality of local servers to update the local neural network model.

[0122] In response to the accuracy of a local neural network model of a local server decreasing over the pre-determined number of iterations when validating the local neural network model using a local validation set, the batch size may be decreased and training of the local neural network model may be continued on the local server. Adjusting the batch size and continuing training of the local neural network model on the local server further may include adjusting at least one of the weighting coefficient and the global learning rate prior to continuing training of the first neural network. In particular, adjusting the batch size, the global learning rate based on the adjusted batch size, and the weighting coefficient further may include decreasing the batch size by a predetermined amount, adjusting the global learning rate based on the decreased batch size, and increasing the weighting coefficient by a pre-determined amount. By training the local neural network model in such a manner, vehicle data from specific geographical locations are not directlyexchanged, training efficiency and accuracy of the local neural network model is increased while protecting user privacy and maintaining data security.

[0123] As an example of how the cross-silo FL system may be used in practice, a transnationally distributed system may include two local servers. When accuracy of the plurality of local neural model network models is continuously increasing over a pre-determined number of iterations during validation with a plurality of local validation datasets, a first set of training data from the first local database may be collected on a first local server to train a first local neural network model on a first set of training data at the first local server using synchronized learning. A second set of training data from a second local database may be collected on at second local server to train a second local neural network model on a second set of training data at the second local server using synchronized learning.

[0124] During each iteration of the training of the first local neural network and second local neural network, a first set of accumulated gradients calculated during a backward propagation stage of the training may be stored at the first local server and a second set of accumulated gradients calculated during a backward propagation stage of the training may be stored at the second local server. Further, during each iteration of the training of the first local neural network and second local neural network, the first set of accumulated gradients from the first local server may be transmitted to the second local server and the second set of accumulated gradients from the second local server may be transmitted to the first local server.

[0125] Training of the first local neural network model and the second local neural network model may include aggregating the first set of accumulated gradients and the second set of accumulated gradients at each of the first server and the second server. Each iteration of the training of the first local neural network model and the second local neural network model may terminate when updating parameters of the first neural network model and the second neural network model based on the aggregated gradients.

[0126] In response to detecting at least one of an accuracy of the first local neural network model training and an accuracy of the second local neural network model training decreasing continuously over a pre-determined number of iterations, training includes updating a batch size of the first set of training data and the second set of training data, and updating a first weight applied to the first set of accumulated gradients and a second weight applied to the second set of accumulated gradients when aggregating the first set of gradients and the second set of gradients.

[0127] In particular, training includes decreasing a first batch size of the first set of training data by a first pre-determined amount and decreasing a second batch size of the second set of training data by a second pre-determined amount. In some embodiments, the first pre-determined amount may be different than the second pre-determined amount. As such, the adjusted batch size of both the first set of training data and the second set of training data may be the minimum of the adjusted batch size of the first set of training data and the adjusted batch size of the second set of training data. Training may continue in response to the plurality of local servers receiving the adjusted batch size.

[0128] The technical effect of adjusting a batch size, a learning rate, and an aggregation weighting coefficient of a neural network model during synchronized training in a cross-silo FL system based on the accuracy of a plurality of copies of the neural network model distributed across different geographical regions, is that an efficiency of training of the neural network model may be increased, leading to more accurate predictive models based on personal data, while reducing bandwidth limitations and protecting the personal data.

[0129] The disclosure also provides support for a method for an advanced driver assistance system (ADAS) of a vehicle, the method comprising: adjusting an operation of the vehicle based on images of an occupant of the vehicle captured via an in-cabin monitoring system of the vehicle, wherein the in-cabin monitoring system relies on a neural network model trained using synchronized learning within a cross-silo federated learning (FL) system, and training the neural network model includes: performing a weighted aggregation of a first set of accumulated gradients of a plurality of parameters of the neural network model over a first plurality of batches of a first set of training data, and a second set of accumulated gradients of a second, identical neural network model trained on a second plurality of batches of a second set of training data, updating parameters of the neural network model based on aggregated gradients, and in response to an accuracy of the neural network model decreasing over a pre-determined number of iterations, adjusting one or more of a batch size of the first plurality of batches, a first aggregation weighting coefficient assigned to the first set of accumulated gradients, and a learning rate.

[0130] In a first example of the method, the in-cabin monitoring system is one of a driver monitoring system (DMS) and an occupant monitoring system (OMS) of the vehicle. In a second example of the method, optionally including the first example: the first set of training data is collected from a first geographical region, the neural network model trained on a first local serverwithin the first geographical region, the second set of training data is collected from a second geographical region, a second neural network model trained on a second local server within the second geographical region. In a third example of the method, optionally including one or both of the first and second examples, the training of the neural network model on the first local server and training of the second neural network model on the second local server is performed without exchanging training data between the first local server and the second local server

[0131] In a fourth example of the method, optionally including one or more or each of the first through third examples, the first set of training data does not include data from the second geographical region, and the second set of training data does not include data from the first geographical region. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, performing the weighted aggregation further comprises performing a summation of a product of a first weighting coefficient and the first set of accumulated gradients and a product of a second weighting coefficient and the second set of accumulated gradients, the second set of accumulated gradients received from the second local server. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, the second weighting coefficient is received from the second local server.

[0132] In a seventh example of the method, optionally including one or more or each of the first through sixth examples, the method further comprises: sending at least one of an adjusted batch size, an adjusted aggregation weighting coefficient, and an adjusted learning rate to the second local server for training the second neural network model. In an eighth example of the method, optionally including one or more or each of the first through seventh examples, the accuracy of the neural network model is determined using a validation set of data collected from the first geographical region. In a ninth example of the method, optionally including one or more or each of the first through eighth examples, adjusting the one or more of the batch size of the first plurality of batches, the first aggregation weighting coefficient assigned to the first set of accumulated gradients, and the learning rate further comprises at least one of: decreasing the batch size by a first pre-determined amount, adjusting the learning rate based on a decreased batch size, and increasing an aggregation weighting coefficient by a second pre-determined amount. In a tenth example of the method, optionally including one or more or each of the first through ninth examples, adjusting the operation of the vehicle based on the images of the occupant of the vehicle captured via the in-cabin monitoring system further comprises one of: adjusting the operation ofthe vehicle based on an identification of a vehicle operator, and adjusting the operation of the vehicle based on a predicted state of the vehicle operator.

[0133] The disclosure also provides support for an advanced driver assistance system (ADAS) of a vehicle, the ADAS comprising: a user input device: a display device, a processor, and a non- transitory memory storing instructions that when executed, cause the processor to: adjust an operation of the vehicle based on an output of a neural network model trained on images of a driver of the vehicle captured via a driver monitoring system (DMS) of the vehicle using synchronized learning within a federated learning (FL) system, wherein training the neural network model includes: performing a weighted aggregation of a first set of accumulated gradients of a plurality of parameters of the neural network model over a first plurality of batches of a first set of training data, and a second set of accumulated gradients of a second, identical neural network model trained on a second plurality of batches of a second set of training data, updating parameters of the neural network model based on aggregated gradients, and in response to an accuracy of the neural network model decreasing over a pre-determined number of iterations, adjusting one or more of a batch size of the first plurality of batches, a first aggregation weighting coefficient assigned to the first set of accumulated gradients, and a learning rate.

[0134] In a first example of the system,: the first set of training data is collected from a first geographical region, the neural network model trained on a first local server within the first geographical region, the second set of training data is collected from a second geographical region, a second neural network model trained on a second local server within the second geographical region, the first set of training data does not include data from the second geographical region, and the second set of training data does not include data from the first geographical region, and training of the second neural network model on the second local server is performed without exchanging training data between the first local server and the second local server. In a second example of the system, optionally including the first example, the accuracy of the neural network model is determined using a validation set of data collected from the first geographical region.

[0135] In a third example of the system, optionally including one or both of the first and second examples, adjusting the one or more of the batch size of the first plurality of batches, the first aggregation weighting coefficient assigned to the first set of accumulated gradients, and the learning rate further comprises at least one of: decreasing the batch size, adjusting the learning rate based on a decreased batch size, and increasing an aggregation weighting coefficient. Tn a fourthexample of the system, optionally including one or more or each of the first through third examples, the batch size is decreased by a pre-determined amount, and the aggregation weighting coefficient is increased by an amount based on the accuracy of the neural network model. In a fifth example of the system, optionally including one or more or each of the first through fourth examples, adjusting the operation of the vehicle based on the images of an occupant of the vehicle captured via the DMS further comprises one of: adjusting the operation of the vehicle based on an identification of the driver, adjusting the operation of the vehicle based on a predicted state of the driver.

[0136] The disclosure also provides support for a method, comprising: collecting a first set of images of drivers of a first set of vehicles at a first geographical location on a first server, collecting a second set of images of drivers of a second set of vehicles a second geographical location different from the first geographical location, on a second server, training a first copy of a neural network model on the first set of images at the first server and training a second copy of the neural network model on the second set of images at the second server, using synchronized learning, during each iteration of the training of the first copy of the neural network model and the training of the second copy of the neural network model: storing a first set of accumulated gradients calculated during a backward propagation stage of the training at the first server, storing a second set of accumulated gradients calculated during the backward propagation stage of the training at the second server: transmitting the first set of accumulated gradients from the first server to the second server, transmitting the second set of accumulated gradients from the second server to the first server, aggregating the first set of accumulated gradients and the second set of accumulated gradients at both of the first server and the second server, updating parameters of the first copy of the neural network model and the second copy of the neural network model based on aggregated gradients, in response to either of an accuracy of the first copy of the neural network model on a first set of validation data collected from the first geographical location at the first server decreasing or an accuracy of the second copy of the neural network model on a second set of validation data collected from the second geographical location at the second server decreasing: decreasing a batch size of the first set of training data and the second set of training data, updating a first weighting coefficient applied to the first set of accumulated gradients and a second weighting coefficient applied to the second set of accumulated gradients when aggregating the first set of accumulated gradients and the second set of accumulated gradients, updating a learning rateduring training of the first copy of the neural network model on the first server and training of the second copy of the neural network model on the second server based on the batch size, and in response to the first copy of the neural network model achieving a threshold accuracy, using a trained first copy of the neural network model in a first vehicle in the first geographical location to adjust an operation of the first vehicle based on images of a first driver of the first vehicle captured via a first driver monitoring system (DMS), and in response to the second copy of the neural network model achieving the threshold accuracy, using a trained second copy of the neural network model in a second vehicle in the second geographical location to adjust operation of the second vehicle based on images of a second driver of the second vehicle captured via a second DMS.

[0137] In a first example of the method, storing the first set of accumulated gradients calculated during the backward propagation stage of the training at the first server and storing the second set of accumulated gradients calculated during the backward propagation stage of the training at the second server further comprises: splitting the first set of training data into a first plurality of minibatches of a fixed batch size, splitting the second set of training data into a second plurality of mini-batches of the fixed batch size, performing forward propagation and backward propagation to calculate a first plurality of gradients for a plurality of parameters for each mini-batch in the first plurality of mini-batches, performing forward propagation and backward propagation to calculate a second plurality of gradients for a plurality of parameters for each mini-batch in the second plurality of mini-batches, accumulating the first set of gradients by performing a summation of the first plurality of gradients on the first server, and accumulating the second set of gradients by performing a summation of the second plurality of gradients on the second server. In a second example of the method, optionally including the first example, updating the first weighting coefficient applied to the first set of accumulated gradients and the second weighting coefficient applied to the second set of accumulated gradients further comprises: in response to the accuracy of the first copy of the neural network model on the first set of validation data decreasing, increasing the first weighting coefficient, and in response to the accuracy of the second copy of the neural network model on the second set of validation data decreasing, increasing the second weighting coefficient.

[0138] “Systems,” “units,” or “modules” may include or represent hardware and associated instructions (e g , software stored on a tangible and non-transitory computer readable storage31medium) that perform one or more operations described herein. The hardware may include electronic circuits that include and / or are connected to one or more logic-based devices, such as microprocessors, processors, controllers, or the like. These devices may be off-the-shelf devices that are appropriately programmed or instructed to perform operations described herein.

[0139] The descriptions of the various embodiments described herein have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0140] It is expected that during the life of a patent maturing from this application many relevant systems, methods and computer programs will be developed and the scope of the terms ML model, DL model, neural network, and vehicle operational data are intended to include all such new technologies a priori.

[0141] The terms "comprises", "comprising", "includes", "including", “having” and their conjugates mean "including but not limited to". This term encompasses the terms "consisting of and "consisting essentially of.”

[0142] As used herein, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

[0143] The word “exemplary” is used herein to mean “serving as an example, an instance or an illustration”. Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0144] The word “exemplary” is used herein to mean “serving as an example, an instance or an illustration”. Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0145] It is appreciated that certain features of embodiments described herein, which are, for clarity, described in the context of separate embodiments, may also be provided in combination ina single embodiment. Conversely, various features of embodiments described herein, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable sub-combination or as suitable in any other embodiment described herein. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

[0146] Although the embodiments described herein have been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

[0147] It is the intent of the applicant(s) that all publications, patents and patent applications referred to in this specification are to be incorporated in their entirety by reference into the specification, as if each individual publication, patent or patent application was specifically and individually noted when referenced that it is to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the embodiments described herein. To the extent that section headings are used, they may not be construed as necessarily limiting. In addition, any priority document(s) of this application is / are hereby incorporated herein by reference in its / their entirety.

Claims

CLAIMS:

1. A method for an advanced driver assistance system (ADAS) of a vehicle, the method comprising: adjusting an operation of the vehicle based on images of an occupant of the vehicle captured via an in-cabin monitoring system of the vehicle, wherein the in-cabin monitoring system relies on a neural network model trained using synchronized learning within a cross-silo federated learning (FL) system, and training the neural network model includes: performing a weighted aggregation of a first set of accumulated gradients of a plurality of parameters of the neural network model over a first plurality of batches of a first set of training data, and a second set of accumulated gradients of a second, identical neural network model trained on a second plurality of batches of a second set of training data; updating parameters of the neural network model based on aggregated gradients; and in response to an accuracy of the neural network model decreasing over a pre-determined number of iterations, adjusting one or more of a batch size of the first plurality of batches, a first aggregation weighting coefficient assigned to the first set of accumulated gradients, and a learning rate.

2. The method of claim 1, wherein the in-cabin monitoring system is one of a driver monitoring system (DMS) and an occupant monitoring system (OMS) of the vehicle.

3. The method of either of claims 1 or 2, wherein: the first set of training data is collected from a first geographical region, the neural network model trained on a first local server within the first geographical region; the second set of training data is collected from a second geographical region, a second neural network model trained on a second local server within the second geographical region.

4. The method of claim 3, wherein the training of the neural network model on the first local server and training of the second neural network model on the second local server is performed without exchanging training data between the first local server and the second local server.

5. The method of either of claims 3 or 4, wherein the first set of training data does not include data from the second geographical region, and the second set of training data does not include data from the first geographical region.

6. The method of any one of claims 3-5, wherein performing the weighted aggregation further comprises performing a summation of a product of a first weighting coefficient and the first set of accumulated gradients and a product of a second weighting coefficient and the second set of accumulated gradients, the second set of accumulated gradients received from the second local server.

7. The method of claim 6, wherein the second weighting coefficient is received from the second local server.

8. The method of any one of claims 3-7, further comprising sending at least one of an adjusted batch size, an adjusted aggregation weighting coefficient, and an adjusted learning rate to the second local server for training the second neural network model.

9. The method of any one of claims 3-8, wherein the accuracy of the neural network model is determined using a validation set of data collected from the first geographical region.

10. The method of any one of claims 1-9, wherein adjusting the one or more of the batch size of the first plurality of batches, the first aggregation weighting coefficient assigned to the first set of accumulated gradients, and the learning rate further comprises at least one of: decreasing the batch size by a first pre-determined amount; adjusting the learning rate based on a decreased batch size; and increasing an aggregation weighting coefficient by a second pre-determined amount.

11. The method of any one of claims 2-9, wherein adjusting the operation of the vehicle based on the images of the occupant of the vehicle captured via the in-cabin monitoring system further comprises one of: adjusting the operation of the vehicle based on an identification of a vehicle operator; andadjusting the operation of the vehicle based on a predicted state of the vehicle operator.

12. An advanced driver assistance system (ADAS) of a vehicle, the ADAS comprising: a user input device: a display device; a processor; and a non-transitory memory storing instructions that when executed, cause the processor to: adjust an operation of the vehicle based on an output of a first neural network model trained on images of a driver of the vehicle captured via a driver monitoring system (DMS) of the vehicle using synchronized learning within a federated learning (FL) system, wherein training the first neural network model includes: performing a weighted aggregation of a first set of accumulated gradients of a plurality of parameters of the first neural network model over a first plurality of batches of a first set of training data, and a second set of accumulated gradients of a second, identical neural network model trained on a second plurality of batches of a second set of training data; updating parameters of the first neural network model based on aggregated gradients; and in response to an accuracy of the first neural network model decreasing over a predetermined number of iterations, adjusting one or more of a batch size of the first plurality of batches, a first aggregation weighting coefficient assigned to the first set of accumulated gradients, and a learning rate.

13. The system of claim 12, wherein: the first set of training data is collected from a first geographical region, the first neural network model trained on a first local server within the first geographical region; the second set of training data is collected from a second geographical region, a second neural network model trained on a second local server within the second geographical region; the first set of training data does not include data from the second geographical region, and the second set of training data does not include data from the first geographical region; and training of the second neural network model on the second local server is performed without exchanging training data between the first local server and the second local server.

14. The system of claim 13, wherein the accuracy of the first neural network model is determined using a first validation set of data collected from the first geographical region, and the accuracy of the second neural network model is determined using a second validation set of data collected from the second geographical location.

15. The system of any one of claims 12-14, wherein adjusting the one or more of the batch size of the first plurality of batches, the first aggregation weighting coefficient assigned to the first set of accumulated gradients, and the learning rate further comprises at least one of: decreasing the batch size; adjusting the learning rate based on a decreased batch size; and increasing an aggregation weighting coefficient.

16. The system of claim 15, wherein the batch size is decreased by a pre-determined amount, and the aggregation weighting coefficient is increased by an amount based on the accuracy of the first neural network model.

17. The system of any one of claims 12-16, wherein adjusting the operation of the vehicle based on the images of an occupant of the vehicle captured via the DMS further comprises one of: adjusting the operation of the vehicle based on an identification of the driver, adjusting the operation of the vehicle based on a predicted state of the driver.

18. A method, comprising: collecting a first set of images of drivers of a first set of vehicles at a first geographical location on a first server; collecting a second set of images of drivers of a second set of vehicles a second geographical location different from the first geographical location, on a second server; training a first copy of a neural network model on the first set of images at the first server and training a second copy of the neural network model on the second set of images at the second server, using synchronized learning; during each iteration of the training of the first copy of the neural network model and the training of the second copy of the neural network model:storing a first set of accumulated gradients calculated during a backward propagation stage of the training at the first server; storing a second set of accumulated gradients calculated during the backward propagation stage of the training at the second server: transmitting the first set of accumulated gradients from the first server to the second server; transmitting the second set of accumulated gradients from the second server to the first server; aggregating the first set of accumulated gradients and the second set of accumulated gradients at both of the first server and the second server; updating parameters of the first copy of the neural network model and the second copy of the neural network model based on aggregated gradients; in response to either of an accuracy of the first copy of the neural network model on a first set of validation data collected from the first geographical location at the first server decreasing or an accuracy of the second copy of the neural network model on a second set of validation data collected from the second geographical location at the second server decreasing: decreasing a batch size of the first set of training data and the second set of training data; updating a first weighting coefficient applied to the first set of accumulated gradients and a second weighting coefficient applied to the second set of accumulated gradients when aggregating the first set of accumulated gradients and the second set of accumulated gradients; updating a learning rate during training of the first copy of the neural network model on the first server and training of the second copy of the neural network model on the second server based on the batch size; and in response to the first copy of the neural network model achieving a threshold accuracy, using a trained first copy of the neural network model in a first vehicle in the first geographical location to adjust an operation of the first vehicle based on images of a first driver of the first vehicle captured via a first driver monitoring system (DMS); and in response to the second copy of the neural network model achieving the threshold accuracy, using a trained second copy of the neural network model in a second vehicle in thesecond geographical location to adjust operation of the second vehicle based on images of a second driver of the second vehicle captured via a second DMS.

19. The method of claim 18, wherein storing the first set of accumulated gradients calculated during the backward propagation stage of the training at the first server and storing the second set of accumulated gradients calculated during the backward propagation stage of the training at the second server further comprises: splitting the first set of training data into a first plurality of mini-batches of a fixed batch size; splitting the second set of training data into a second plurality of mini-batches of the fixed batch size; performing forward propagation and backward propagation to calculate a first plurality of gradients for a plurality of parameters for each mini-batch in the first plurality of mini-batches; performing forward propagation and backward propagation to calculate a second plurality of gradients for a plurality of parameters for each mini-batch in the second plurality of minibatches; accumulating the first set of gradients by performing a summation of the first plurality of gradients on the first server; and accumulating the second set of gradients by performing a summation of the second plurality of gradients on the second server.

20. The method of either claims 18 or 19, wherein updating the first weighting coefficient applied to the first set of accumulated gradients and the second weighting coefficient applied to the second set of accumulated gradients further comprises: in response to the accuracy of the first copy of the neural network model on the first set of validation data decreasing, increasing the first weighting coefficient; and in response to the accuracy of the second copy of the neural network model on the second set of validation data decreasing, increasing the second weighting coefficient.

Images