SYSTEMS AND METHODS FOR SELF-SUPERVISED FEDERATED LEARNING FOR AUTOMOTIVE APPLICATIONS - Patent application

Abstract

To provide Federated Learning training to a neural network while maintaining safety and user privacy.SOLUTION: A method includes receiving, from one or more server computers through a communication network, an edge model and collecting sensor data acquired by a sensor on a vehicle. The method also includes identifying a first data item from among the collected sensor data when the first data item is determined to satisfy a criterion. The method further includes applying a transformation to the identified first data item to generate a second data item to form a training dataset containing the first data item, the second data item, and a signal representing the transformation between the first data item and the second data item. The method further includes training with respect to the edge model on the training dataset and transmitting first data representing the trained edge model to the one or more server computers through the communication network.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates generally to systems and methods for providing training for neural networks in autonomous vehicle applications, and more particularly, to providing federated learning training for neural networks while maintaining safety and user privacy. [Background technology]

[0002] Neural networks may be integrated into applications deployed on many distributed edge devices (e.g., processors or computing devices implemented in hospitals or mobile phones). One method for training such neural networks is Federated Learning (FL), which trains machine learning (ML) models using large amounts of data while preserving user privacy.

[0003] To this end, the FL technique consists of a local training phase and a global aggregation phase. In the local training phase, each edge device trains its copy of the neural network with the data sensed and used by the application. By performing the training on the edge device, local data is not exposed or transmitted externally (e.g., to a remote coordinator or server), thereby ensuring data privacy for edge device users. Instead, only local updates to the neural network trained on the edge device are sent to the coordinator, which aggregates the updates to generate a new global model. The global model can then be provided to other edge devices for use in applications.

[0004] It is crucial that machine learning (ML) models integrated into safety-critical applications, such as computer vision (CL) or other machine learning (ML) applications (e.g., autonomous driving control) in autonomous vehicles, are trained on a large amount of data to ensure the accuracy of estimation and the safety of use in real-world environments. Although FL can be applied to these models, there is no reliable teacher signal (e.g., human annotation) for training in the vehicular context. As a result, the accuracy of estimation may be reduced when trained on local data in the vehicle. Summary of the Invention

[0005] SUMMARY OF THE DISCLOSURE In accordance with one or more exemplary embodiments, a system and method for providing driving information to non-driver users is provided.

[0006] According to an aspect of the disclosure, a method executed by one or more programmed processors includes receiving an edge model from one or more server computers over a communications network; collecting sensor data acquired by sensors on the vehicle; identifying a first data item from the collected sensor data when the first data item is determined to satisfy a criterion; applying a transformation to the identified first data item to generate a second data item to form a training data set including the first data item, the second data item, and a signal representative of the transformation between the first data item and the second data item; training an edge model on the training data set; and transmitting the first data representative of the trained edge model to one or more server computers over the communications network.

[0007] According to an aspect of the disclosure, a computing device includes a memory that stores instructions and a processor, where the processor is configured to execute instructions to receive an edge model from one or more server computers over a communications network; collect sensor data acquired by sensors on the vehicle; identify a first data item from the collected sensor data when it is determined that the first data item satisfies a criterion; apply a transformation to the identified first data item to generate a second data item; form a training dataset including the first data item, the second data item, and a signal representing the transformation between the first data item and the second data item; train an edge model on the training dataset; and transmit the first data representing the trained edge model to the one or more server computers over the communications network.

[0008] According to an aspect of the disclosure, a non-transitory computer-readable medium has instructions stored thereon, the instructions including one or more instructions that, when executed by one or more processors of the device, cause the one or more processors to perform the following: receiving an edge model from one or more server computers over a communications network; collecting sensor data acquired by sensors on the vehicle; identifying a first data item from the collected sensor data when it is determined that the first data item satisfies a criterion; applying a transformation to the identified first data item to generate a second data item to form a training data set including the first data item, the second data item, and a signal representative of the transformation between the first data item and the second data item; training an edge model on the training data set; and transmitting the first data representative of the trained edge model to the one or more server computers over the communications network.

[0009] Additional aspects will be set forth in part in the description that follows, or will be apparent from the description that follows, or may be learned from practice of illustrative embodiments of the present disclosure. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram of a system according to an embodiment. [Diagram 2] FIG. 2 is a diagram of components of the autonomous vehicle of FIG. 1 according to an embodiment. [Diagram 3] FIG. 3 is a diagram of data processing associated with training neural networks for multiple autonomous vehicles, according to an embodiment. [Figure 4] FIG. 4 is a diagram of the transformations involved in data processing associated with training a neural network for a single autonomous vehicle, according to an embodiment. [Diagram 5] FIG. 5 is a diagram of data processing associated with training a neural network for a single autonomous vehicle, according to an embodiment. [Figure 6] FIG. 6 is a flow chart of a method for training a neural network for an autonomous vehicle according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The above and other aspects, features and modes of embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0012] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings identify the same or similar elements, and in which:

[0013] 1 is a diagram of a system 100 according to an embodiment. The system 100 includes one or more vehicles 110a-n and one or more server computers 120a-n. The one or more server computers 120a-n are connected to each other and to each of the vehicles 110a-n, for example, via a communications network 130.

[0014] The disclosed embodiments include receiving from one or more server computers 120. Server computers 120 as used in this disclosure include general purpose computers, personal computers, workstations, mainframe computers, notebooks, global positioning devices, laptop computers, smart phones, personal digital assistants, network servers, and other electronic devices that may interact with a user to develop programming code.

[0015] In some embodiments, the server computer 120 includes a processor, a display device, a memory device, and other components including these components that facilitate electronic communication. Other components include user interface devices such as input / output devices. The server computer 120 includes computer hardware components such as a central processing unit (CPU) or processor, a bus, memory devices, storage units, data processors, input devices, output devices, network interface devices, and combinations of other types of components apparent to those skilled in the art. The server computer 120 can further include application programs including software modules, sets of instructions, routines, data structures, display interfaces, and other types of structures that perform the operations of the present disclosure.

[0016] The disclosed embodiments include receiving over a communications network 130. A communications network as used in this disclosure includes a set of computers (e.g., one or more server computers 120) that share resources on or provided by a network node. The set of computers communicate with each other using a common communications protocol over digital interconnects. These interconnects consist of physical wired, optical, wireless, radio frequency based electrical communications technologies arranged in a variety of network topologies. For example, these interconnects are made through databases, servers, RF (radio frequency) signals, cellular technologies, Ethernet, telephone, "TCP / IP" (Transmission Control Protocol / Internet Protocol), and other electronic communications formats. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN), a private network, an ad-hoc network, an intranet, the Internet, an optical fiber-based network, or equivalents thereof, and / or combinations thereof or other types of networks.

[0017] The number and arrangement of servers 120 and networks 130 shown in Figure 1 are provided as an example. In practice, there may be additional servers 120 and / or networks 130, fewer servers 120 and / or networks 130, different servers 120 and / or networks 130, or servers 120 and / or networks 130 arranged differently than those shown in Figure 1. Furthermore, two or more servers 120 shown in Figure 1 may be implemented within a single server 120, or a single server 120 shown in Figure 1 may be implemented as multiple distributed servers 130. Additionally or alternatively, a set of servers 120 (e.g., one or more servers 120) may perform one or more functions described as being performed by another set of servers 120.

[0018] In some embodiments, the communication network 130 is constructed as a neural network. A neural network is based on a collection of connected units or nodes called artificial neurons, loosely modeled on neurons in a biological brain. Each connection, like a synapse in a biological brain, can send a signal to another neuron. An artificial neuron receives and processes a signal, then sends the signal to other neurons connected to it. The signals in the connections are real numbers, and the output of each neuron is calculated by a nonlinear function of the sum of its inputs. The connections may be edges (like in the autonomous vehicle 110). Neurons and edges have weights that are adjusted as learning progresses. The weights increase or decrease the strength of the signal in the connection. Neurons have thresholds such that a signal is sent only if the aggregate signal exceeds that threshold. Neurons are aggregated into layers. Different layers perform different transformations on their inputs. Signals travel from the first layer (e.g., input layer) through potential hidden layers to the last layer (e.g., output layer), and may travel multiple times.

[0019] As described in further detail below, federated learning (FL) is used to train neural networks for safety-critical automotive applications by incorporating reliable self-supervision into local training performed on edge devices. By applying FL, a large amount of data can be used to train the neural network, which increases the accuracy of the inference. Furthermore, by applying FL, data privacy for the user (i.e., the operator of the vehicle 110) can be ensured. In addition, by incorporating reliable self-supervision, the accuracy of the inference or prediction by the neural network can be increased.

[0020] A more detailed view of the vehicles 110 can be seen in Figure 2. Each of the vehicles 110 includes one or more sensors 112, one or more transceivers 114, and a vehicle computer 116.

[0021] The one or more transceivers 114 as used in this disclosure include one or more components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the vehicle 110 to communicate with other vehicles 110 and / or one or more server computers 120, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The one or more transceivers 114 enable the vehicle 110 to receive information from and / or provide information to another vehicle 110 / server computer 120. For example, the one or more transceivers 114 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or other interface capable of transmitting or receiving electrical / electromagnetic information.

[0022] As seen in FIG. 2, a vehicle computer 116 as used in this disclosure includes a bus (not shown), a memory 117, a processor 118, input components (not shown), and output components (not shown).

[0023] The bus includes components that enable communication between the components of the vehicle computer 116 .

[0024] Processor 118 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 118 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. Processor 118 may include one or more processors that can be programmed to perform functions.

[0025] Memory 117 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions used by processor 118. Memory 117 also stores information and / or software related to the operation and use of vehicle computer 116. For example, memory 117 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium along with a corresponding drive.

[0026] The input components include components that allow the vehicle computer 116 to receive information, for example, via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone), and sensors for sensing information (e.g., a Global Positioning System (GPS) component, an accelerometer, a gyroscope, and / or an actuator).

[0027] Output components include components that provide output information from the vehicle computer 116 (eg, a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0028] Vehicle computer 116 performs one or more processes described herein. Vehicle computer 116 performs operations based on processor 118 executing software instructions stored by a non-transitory computer readable medium, such as memory 117. A computer readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0029] The software instructions may be read into memory 117 from another computer-readable medium or another device via one or more transceivers 114. When executed, the software instructions stored in memory 117 cause processor 118 to perform one or more operations described herein.

[0030] Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more operations described herein, and thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0031] The disclosed embodiments include receiving an edge model 210 (e.g., edge model 210a). The edge model 210 as used in this disclosure includes a machine learning model. The machine learning model is configured to be integrated into applications running on an autonomous vehicle (such as vehicle 110a). The applications running on autonomous vehicle 110 are safety-critical applications such as computer vision, autonomous vehicle control, and other machine learning applications related to the operation of autonomous vehicle 110. Autonomous vehicle control can include autonomous control of acceleration, braking, steering, gear shifting, and other systems that may affect the operation of vehicle 110 through its environment.

[0032] In some embodiments, the edge model 210 is associated with the detection of objects 140 encountered by the vehicle 110. An object may be another vehicle, a pedestrian, wildlife, a road hazard, or any other aspect of the environment that may interact with the vehicle 110. For example, in Figure 1, the object 140 is depicted as a bicycle on the road.

[0033] In some embodiments, the edge model 210 is associated with a sensory interpretation. For example, one type of sensory interpretation includes image segmentation. Image segmentation divides a digital image into multiple image segments (e.g., image regions or image objects (a set of pixels)). Image segmentation simplifies and / or changes the display of an image to one that is more meaningful and / or easier to analyze. Image segmentation is used to locate objects and boundaries (e.g., lines and curves) in an image. Image segmentation involves assigning labels to various pixels in an image such that pixels with the same label share certain characteristics.

[0034] 1 and 2, vehicle 110a receives edge model 210a via one or more transceivers 114 and stores the received edge model 210a, for example, in memory 117 of vehicle computer 116 of vehicle 110a. Similarly, other vehicles 110b-n each receive models (e.g., edge model 210b, edge model 210c, edge model 210n) similar to edge model 210a at their respective transceivers 114 and store their respective received edge models 210b-n, for example, in memory 117 of vehicle computer 116 of the other vehicles 110b-n.

[0035] The disclosed embodiments include collecting sensor data 220a acquired by one or more sensors 112 at the vehicle 110a. As used in this disclosure, a sensor 112 includes a camera, camcorder, microphone, lidar, or other device configured to collect the sensor data 220a. As used in this disclosure, sensor data 220a includes photographs, video recordings, audio recordings, lidar data, or other measurements of the vehicle 110a's surrounding environment. Similarly, other vehicles 110b-n each collect sensor data 220b-n via their respective sensors 112.

[0036] 1 and 2, vehicle 110a receives sensor data 220a via one or more sensors 112 and stores the received sensor data 220a, for example, in memory 117 of vehicle computer 116. Similarly, other vehicles 110b-n each store their collected sensor data 220b-n via their respective memories 117 of their vehicle computers 116.

[0037] The disclosed embodiments include a vehicle 110, such as vehicle 110a. As used in this disclosure, vehicle 110 includes a car, van, truck, bus, motorcycle, moped, drone, robot, or other mobile device capable of full or partial autonomous movement.

[0038] As seen in FIG. 1 , the system 100 includes a plurality of vehicles 110a-n. Each of the vehicles 110a-n is substantially similar to some of the other vehicles 110a-n or different from some of the other vehicles 110a-n. In some embodiments, all of the vehicles 110a-n are autonomous vehicles of the same model having similar sensing and dynamic capabilities / configurations. In other embodiments, all of the vehicles 110a-n are autonomous vehicles of different models having a variety of different sensing and dynamic capabilities / configurations. In other embodiments, some of the vehicles 110a-n are similarly configured and other vehicles are differently configured.

[0039] The disclosed embodiment includes identifying a first data item 222a from among the collected sensor data 220a. The first data item 222a as used in this disclosure includes a subset of the sensor data 220a received by the vehicle 110a, which is useful for training the edge model 210a to improve the accuracy of the estimation and the safety of use in a real-world environment. In a similar manner, as seen, for example, in FIG. 3, other vehicles 110b-n include identifying other first data items (e.g., first data item 222b, first data item 222c, first data item 222n) similar to identifying the first data item 222a.

[0040] The disclosed embodiment includes identifying when the first data item 222a is determined to meet a criterion. Criteria as used in this disclosure include (i) vehicle information (e.g., speed, steering and braking) when the data is detected (e.g., speed above a predetermined speed, or braking when the speed is above a predetermined speed, steering above a predetermined degree or amount, steering above a predetermined amount when the speed is above a predetermined speed, or other conditions related to vehicle operation that are useful for training the edge model 210); (ii) the position of the vehicle when the data is detected (e.g., as determined by an inertial measurement unit (IMU), global positioning system (GPS), or other sensor used to determine the relative or absolute position / orientation of the vehicle 110a), the time when the data is detected, driver monitor information when the data is transmitted; (iii) image recognition results (e.g., scene classification, variance in the number of objects detected, road structure, or other meaningful characteristics of the environment surrounding the vehicle 110a); (iv) uniqueness / clustering of image features; (v) uncertainty metrics; and (vi) and / or other recognizable characteristics that indicate usefulness for training the edge model 210. In a similar manner, the other vehicles 110b-n identify when their respective first data items 222b-n are determined to satisfy the criteria.

[0041] The disclosed embodiments include applying a transform 224aa to the identified first data item 222a. Transform 224aa as used in this disclosure includes rotation, flipping, contrast adjustment, or other manipulation of the first data item 222a to present relevant information in various ways. For example, as seen in FIG. 4, the first data item 222a is an image of a bicycle, and transform 224aa is a 45° rotation of the bicycle image. In some embodiments, additional transforms 224ab-an are applied to the first data item 222a. For example, as seen in FIG. 4, transform 224ab is a 90° rotation of the bicycle image, transform 224ac is a 135° rotation of the bicycle image, and transform 224an is a 180° rotation of the bicycle image. In a similar manner, the other vehicles 110b-n apply their own transforms 224ba-nn to their respective first data items 222b-n.

[0042] In some embodiments (such as the examples above), the transformation 224 (e.g., rotation) may be random, as long as the transformation parameters (e.g., amount of rotation) are stored and known by a trainer. By using training data where the transformation is known, the neural network is trained to recognize the rotation to apply to the images it acquires as input / trained without a supervision signal. A supervision signal as used in this disclosure includes training examples having input and desired output values.

[0043] The disclosed embodiments include applying a transform 224aa to generate a second data item 226aa. The second data item 226aa as used in this disclosure includes rotated, inverted, contrast adjusted, or other manipulated data transformed from the first data item 222a. For example, as seen in FIG. 4, the second data item 226aa is an image of a bicycle rotated by 45°. In some embodiments, additional data items 226ab-an are generated by the applied transform 224ab-an. For example, as seen in FIG. 4, the third data item 226ab is an image of a bicycle rotated by 90°, the fourth data item 226ac is an image of a bicycle rotated by 135°, and the nth data item 226an is an image of a bicycle rotated by 180°. In a similar manner, the other vehicles 110b-n generate their own second / additional data items 226ba-nn by applying their respective transforms 224ba-nn.

[0044] 4 are applied to the first data items 222a to generate the additional data items 226ab-an, but this need not be the case. For example, in some embodiments, the additional transforms 224ab-an are applied to some of the generated second / additional data items 226aa-an instead of to the first data items 222a to generate the additional data items 226ab-an.

[0045] The edge model 210a is run using the identified first data item as an input to the edge model 210a. In some embodiments, after (i) the edge model 210a is received from the server computer 120 and stored in the memory 117 of the vehicle computer 116 of the vehicle 110a, and (ii) the first data item 222a is stored in the memory 117 and identified by the processor 118 of the vehicle 110a, the processor 118 inputs the first data item 222a into the edge model 210a to detect the object 140 as an estimate. Running the edge model 210a results in the detection of the object 140 and generates an estimate having one or more particular confidence levels. These confidence levels indicate the degree to which the edge model 210a can accurately recognize the object 140 in the real-world environment. For example, the processor 118 determines that a bicycle has been detected with 90% confidence as an estimate after running the first data item 222a through the edge model 210a. In a similar manner, as seen, for example, in Figure 3, the other vehicles 110b-n operate their respective received edge models 210b-n using their respective first data items 222b-n as inputs to generate estimates, the confidence intervals of these other estimates being, respectively, similar to, less than, or greater than the estimate of vehicle 110a.

[0046] The disclosed embodiment includes forming a training data set 228a including a first data item 222aa, a second data item 226aa, and a signal 225aa representing a transformation 224aa between the first data item 222a and the second data item 226aa (e.g., as seen in FIG. 5). Forming the training data set 228a as used in this disclosure includes aggregating relevant information in a manner useful for training a machine learning model. The training data set 228a includes additional data items 226ab-nn (e.g., as shown in FIG. 4). Similarly, as seen in FIG. 3, for example, the other vehicles 110b-n generate their own training data sets (e.g., training data set 228b, training data set 228c, training data set 228n) including respective first data items 222b-n, second / additional data items 226ba-nn, and signals 225ba-nn representing transformations 224ba-nn between the first data items 222b-n and the second / additional data items 226ba-nn corresponding to the respective vehicles 110b-n.

[0047] The disclosed embodiments include training the edge model 210a on the training dataset 228a. Training as used in this disclosure includes a local training phase associated with FL. This training is characterized as self-supervised training because the components of the training dataset 228a do not include human-annotated labels. A label as used in this disclosure is a meaningful or useful characteristic of an object (e.g., object 140) that provides a context from which a machine learning model can learn. For example, a label corresponding to a bicycle includes two wheels, pedals, or handlebars. In some embodiments, the self-supervised training is performed based on the fact that a predetermined condition is met (e.g., the vehicle is parked, a WiFi connection is established, the vehicle is connected to an external power source, a power source is available, or other conditions that allow training to be performed in a safe and efficient manner).

[0048] As seen in FIG. 4, training the edge model 210a using the training data set 228a results in the generation of a trained edge model 230a. The trained edge model 230a can generate estimates with a higher confidence level than the untrained edge model 210a. For example, the processor 118, after running the first data item 222a through the trained edge model 230a, determines that a bicycle was detected as a trained first estimate with 95% confidence (up from 90% using the original edge model 210a). Similarly, the other vehicles 110b-n train their respective edge models 210b-n on their respective generated training data sets 228b-n to generate trained edge models 230b-n.

[0049] In some embodiments, training the edge model 210a includes training a copy of the received edge model 210a. By training a copy of the edge model 210a, the original edge model 210a is preserved after training. Thus, the performance of the original edge model 210a can be compared to the performance of the trained edge model 230a so that the models 210a, 230a capable of generating estimates with a high confidence level can be used in the future. Similarly, other vehicles 110b-n train themselves on copies of their respective edge models 210b-n.

[0050] The disclosed embodiment includes transmitting the first data 240a representing the trained edge model 230a to one or more server computers 120 via the communication network 130. In contrast to transmitting the training data set 228a to one or more servers 120 for training, transmitting the first data 240a representing the trained edge model 230a (obtained by training locally) can protect the privacy of the user's data. Similarly, as seen in FIG. 3, the other vehicles 110b-n generate their own first data (e.g., first data 240b, first data 240c, first data 240n) that are then transmitted to one or more servers 120.

[0051] The disclosed embodiment includes obtaining gradients 232a between the pre-trained edge model 210a and the trained edge model 230a as the first data 240a. The gradients 232a as used in this disclosure include update parameters (e.g., weights) that represent the difference between the edge model 210a and the trained edge model 230a. By transmitting only the gradients 232a rather than the entire updated / trained model 230a, the transmission overhead can be reduced, which can improve the performance of the communication network 130. Similarly, the other vehicles 110b-n obtain gradients 232b-n between their respective edge models 210b-n and the trained edge models 230b-n as their respective first data 240b-n, which are then transmitted to one or more servers 120.

[0052] The disclosed embodiment includes receiving second data 250a from one or more server computers 120 via the communication network 130, the second data 250a representing a model trained using model information aggregated from other edge models 230b-n. As used in this disclosure, the second data 250a includes the results of a global aggregation phase associated with the FL. For example, the one or more server computers 120 aggregate the first data 240a-n (e.g., trained models 230a-n or gradients 232a-n) received from each of the multiple edge vehicles 110a-n against the edge model 210a and update the edge model 210a accordingly. The second data 250a represents the updated edge model itself or the gradient between the updated edge model and the original edge model 210a. Similarly, each of the other vehicles 110b-n receives the second data 250b-n. The second data 250b-n represent updates to the respective models 210b-n based on an aggregation of the data 240a-n for the respective models 210b-n. The second data 250b-n may be substantially identical to or different from the second data 250a. In some embodiments, the second data 250a is transmitted from one or more server computers 120 to each of the edge vehicles 110a-n.

[0053] The disclosed embodiment includes updating the edge model 210a based on the second data 250a. After updating the edge model 210a (and possibly the trained edge model 230a, if both copies are stored in the memory 117) with the second data 250a, the updated edge model can generate a higher confidence level estimate for both the original edge model 210a and the trained edge model 230a. For example, the processor 118 determines that a bicycle was detected with 98% confidence (up from 95% using the trained edge model 230a and 90% using the original edge model 210a) as the updated first estimate after running the first data item 222a through the updated edge model. Similarly, the other vehicles 110b-n each update their respective edge models 210b-n with the second data 250b-n. Alternatively, the other vehicles 110b-n may update their respective edge models 210b-n with the second data 250a.

[0054] FIG. 6 is a flow chart of a method for providing an FL for training a neural network for an autonomous vehicle according to an embodiment. Referring to FIG. 6, in operation 302, the system receives an edge model from one or more server computers via a communication network. In operation 304, the system collects sensor data acquired by sensors on the vehicle. In operation 306, the system identifies a first data item from the collected sensor data when it is determined that the first data item meets a criterion. In operation 308, the system applies a transformation to the identified first data item to generate a second data item, forming a training data set including the first data item, the second data item, and a signal representing the transformation between the first data item and the second data item. In operation 310, the system trains the edge model on the training data set. In operation 312, the system transmits the first data representing the trained edge model to one or more server computers via the communication network.

[0055] It will be understood that one or more operations of the methods described above may be omitted or combined with other operations, and one or more additional operations may be added.

[0056] Utilizing the above-mentioned method, several advantages over conventional autonomous vehicle training techniques are realized. By training locally, as opposed to transmitting training data to a coordinator, privacy of the user's data is ensured. By applying reliable self-supervision, training can be performed in a vehicular context where a supervising signal is not easily or practically achievable, improving the accuracy of estimation. By transmitting only gradients rather than updated / trained models, the transmission overhead is reduced, thereby improving the performance of the communication network. By aggregating updates to the ML model from multiple edge devices, the ML model can be efficiently trained with a large amount of data, thereby improving performance (accuracy of estimation).

[0057] The foregoing disclosure provides examples and explanations, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. For example, in some embodiments, at least some of the sensor data 220 may be annotated (e.g., by a user via a computing device). In this case, the sensor data 220 may be transmitted or provided to a smartphone, mobile device, or other computing device of a user, where the user may annotate objects in the image. In other embodiments, after self-supervised training (e.g., predicting rotation), supervised training using human-annotated labels may be applied. Such initial self-supervised training of the neural network increases the performance of subsequent supervised learning. For prediction of rotation angles, the neural network requires an understanding of the orientation of objects in the image. Thus, the neural network can be trained to focus on specific image features. For example, in an image having a car facing forward in the image, it is important to understand that the license plate is usually located under the windshield in order to tell if the car is upside down in the image. Thus, if the windshield is located above the license plate, the car in the image is likely to be upside down. The neural network does not initially know the image features it extracts, called "license plate" or "windshield". Thus, typical self-supervised training forces the neural network to learn to extract useful image features rather than simple shapes such as lines or edges. In a second stage, when supervised training is applied after self-supervised learning, the neural network is taught how to use the learned image features to predict certain objects. In general, how well a neural network can extract useful image features correlates with higher performance, and better extraction generally correlates with the number of images the neural network is exposed to during training.Therefore, using more images through self-supervised learning helps increase performance over training on only a smaller number of images with human label annotations.

[0058] As used herein, the term "component" is intended to broadly comprise hardware, firmware, or a combination of hardware and software.

[0059] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement the systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0060] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below depends directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0061] No element, act, or instruction used herein should be construed as critical or required unless expressly stated. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Additionally, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with "one or more." When only one item is intended, the term "a" or similar language is used. Also, as used herein, the term "having" or similar language is intended to be an open-ended term. Additionally, the phrase "based on" is intended to mean "based at least in part on" unless expressly specified to the contrary.

[0062] An expression such as "at least one," when preceding a list of elements, modifies the entire list of elements and not each individual element of the list. For example, the expression "at least one of a, b, and c" should be understood to include a only, b only, c only, both a and b, both a and c, both b and c, all of a, b, and c, or any variation of the foregoing examples.

[0063] Although terms such as "first", "second", etc. are used to describe various elements, such elements should not be limited to the above terms. The above terms may be used only to distinguish one element from another element.

Claims

1. 1. A method executed by one or more programmed processors, comprising: receiving an edge model over a communications network from one or more server computers; collecting sensor data acquired by sensors on a vehicle; identifying a first data item from the collected sensor data when it is determined that the first data item satisfies a criterion; applying a transformation to the identified first data items to generate second data items to form a training data set including the first data items, the second data items, and a signal representative of the transformation between the first data items and the second data items; training the edge model on the training data set; transmitting first data representing the trained edge model to the one or more server computers via the communications network; A method comprising:

2. The method described in claim 1, wherein whether or not the criterion is met is determined based on at least one of vehicle information at the time the data is detected, the vehicle's position, time or driver monitor information at the time the data is detected, image recognition results, and an uncertainty index.

3. receiving second data from the one or more server computers over the communications network, the second data representing a model trained using model information aggregated from other edge models; updating the edge model based on the second data; and The method of claim 1 further comprising:

4. The method of claim 1 or 2, wherein training the edge model comprises training a copy of the received edge model.

5. The method of claim 1 or 2, further comprising obtaining as the first data a gradient between the edge model before the training and the edge model after the training.

6. The method of claim 3 , further comprising obtaining as the first data a gradient between the received edge model and a copy of the edge model updated by the training.

7. The method of claim 1 or 2, wherein applying the transformation comprises rotating the first data item.

8. The method of claim 1 or 2, wherein training on the training dataset comprises training without human annotation.

9. A memory for storing instructions; Processor and A computing device comprising: The processor, receiving an edge model from one or more server computers over a communications network; Collecting sensor data acquired by sensors on the vehicle; identifying a first data item from the collected sensor data when it is determined that the first data item satisfies a criterion; applying a transformation to the identified first data items to generate second data items, forming a training data set including the first data items, the second data items, and a signal representative of the transformation between the first data items and the second data items; training the edge model on the training data set; 23. A computing device configured to execute the instructions to transmit first data representing the trained edge model over the communications network to the one or more server computers.

10. The processor further comprises: receiving second data from the one or more server computers over the communications network, the second data representing a model trained using model information aggregated from other edge models; The computing device of claim 9 , configured to execute the instructions to update the edge model based on the second data.

11. A computing device according to claim 9 or 10, wherein the instructions for training the edge model comprise instructions for training a copy of the received edge model.

12. 11. The computing device of claim 9 or 10, wherein the processor is further configured to execute the instructions to obtain as the first data a gradient between the edge model before the training and the edge model after the training.

13. 12. The computing device of claim 11, wherein the processor is further configured to execute the instructions to obtain as the first data a gradient between the received edge model and a copy of the edge model updated by the training.

14. A computing device according to claim 9 or 10, wherein the instructions for applying the transformation comprise instructions for rotating the first data item.

15. 11. A computing device according to claim 9 or 10, wherein the instructions for training on the training dataset comprise instructions for training without human annotation.

16. A non-transitory computer-readable medium storing instructions, comprising: The instruction: receiving an edge model over a communications network from one or more server computers; collecting sensor data acquired by sensors on a vehicle; identifying a first data item from the collected sensor data when it is determined that the first data item satisfies a criterion; applying a transformation to the identified first data items to generate second data items to form a training data set including the first data items, the second data items, and a signal representative of the transformation between the first data items and the second data items; training the edge model on the training data set; transmitting first data representing the trained edge model to the one or more server computers via the communications network; A non-transitory computer-readable medium comprising one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to execute.

17. The instructions may further include: receiving second data from the one or more server computers over the communications network, the second data representing a model trained using model information aggregated from other edge models; updating the edge model based on the second data; and 20. The non-transitory computer-readable medium of claim 16, comprising one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to execute:

18. 18. The non-transitory computer-readable medium of claim 16 or 17, wherein causing the one or more processors to train the edge model comprises causing the one or more processors to train a copy of the received edge model.

19. 18. The non-transitory computer-readable medium of claim 16 or 17, wherein the instructions further include one or more instructions that, when executed by one or more processors of the device, cause the one or more processors to obtain as the first data a gradient between the edge model before the training and the edge model after the training.

20. 20. The non-transitory computer-readable medium of claim 18, wherein the instructions further include one or more instructions that, when executed by one or more processors of the device, cause the one or more processors to obtain as the first data a gradient between the received edge model and a copy of the edge model updated by the training.

21. 18. The non-transitory computer-readable medium of claim 16 or 17, wherein causing the one or more processors to apply the transformation comprises causing the one or more processors to rotate the first data item.

Images