Shared processing of deep neural networks

Abstract

A system includes a processor for performing one or more autonomous driving or assisted driving tasks based on a neural network. The neural network includes a base part for performing feature extraction for multiple tasks simultaneously on a single input data set. The neural network includes multiple subtask parts that perform the multiple tasks based on the feature extraction output from the base part. Each of the multiple subtask parts includes a node or layer of a neural network trained on a different training data set, and the base part includes a node or layer of a neural network trained using each of the different training data sets constrained by elastic weight solidification to restrict the base part from forgetting previously learned tasks.

Description

Technical Field

[0001] The present invention relates generally to systems and methods for shared processing in deep neural networks, and more particularly to shared processing of deep neural networks using elastic weight solidification in assisted and autonomous driving applications. Background Art

[0002] Vehicles provide an important part of transportation for commercial, government, and private entities. Autonomous vehicles and driver assistance systems are currently being developed and deployed to provide safety, reduce the amount of user input required, or even completely eliminate user involvement. For example, some driver assistance systems (such as collision avoidance systems) can monitor the driving, position, and speed of vehicles and other objects while humans are driving. When the system detects that a collision or impact is imminent, the collision avoidance system can intervene and apply brakes, steer the vehicle, or perform other avoidance or safety maneuvers. As another example, an autonomous vehicle can drive and navigate the vehicle with little or no user input.

[0003] Autonomous or assisted driving systems typically utilize data from sensors to determine information about the vehicle's surroundings. In some cases, this data must be processed to detect objects, roads, or to make driving or assisted driving decisions. Due to the variety of scenes, objects, or complex situations that can be presented to an autonomous or assisted driving vehicle, the computational load can be very high. In some cases, the computational load imposes limitations on the functions or features that can be provided by the autonomous or assisted driving system. Summary of the invention

[0004] According to the present invention, a method for training a neural network for multiple deep learning tasks is provided, the method comprising:

[0005] training the common portion and the first-task portion of the neural network for the first task; and

[0006] The common part and the second task part are trained for performing the second task, where the common part is constrained using elastic weight curing (EWC) to preserve memory of feature extraction used to perform the first task.

[0007] According to one embodiment of the present invention, training for the first task includes training using a first training dataset labeled for the first task, and wherein training for the second task includes training using a second training dataset labeled for the second task.

[0008] According to one embodiment of the present invention, training for a first task includes training a first task part and an unconnected second task part that receive an output of a common part, and wherein training for a second task includes training a second task part and an unconnected first task part that receive an output of the common part, and the method further includes connecting the first task part and the second task part to the common part during testing or deployment, wherein the common part simultaneously performs feature extraction for the first task and the second task and provides outputs to the first task part and the second task part, wherein the outputs are the results of feature extraction for the first task and the second task.

[0009] According to one embodiment of the invention, the method further comprises determining which weights are important for a particular task and applying the EWC constraints to those tasks during training for one or more subsequent tasks.

[0010] According to an embodiment of the present invention, the first task or the second task includes one or more of object detection, image segmentation, and lane finding.

[0011] According to the present invention, there is provided a non-transitory computer-readable storage medium storing instructions, which when executed by one or more processors cause the one or more processors to:

[0012] Processing the sensor data set using a neural network, the neural network comprising a base part and a plurality of task parts, wherein the base part receives the sensor data set at an input layer and outputs to each of the plurality of task parts, wherein each of the plurality of task parts outputs a result of the task based on an output of the base part; and

[0013] Execute an autonomous or assisted driving task based on the outputs of the multiple task parts;

[0014] The base portion comprises layers of a neural network trained sequentially for each of a plurality of tasks, wherein the training of the base portion for at least one task is constrained to preserve performance of previous tasks by using elastic weight solidification (EWC).

[0015] According to one embodiment of the present invention, the multiple task parts output results of different tasks among the multiple tasks, and at least one of the multiple tasks includes one or more of object detection, image segmentation, or lane finding.

[0016] According to one embodiment of the present invention, the sensor data set includes a sensor data frame captured by one or more sensors, wherein the sensor data frame includes at least one of the following parts:

[0017] Red-Green-Blue (RBG) images;

[0018] Infrared images;

[0019] Light Detection and Ranging (LIDAR) data frames; and

[0020] Radar data frame.

[0021] According to one embodiment of the present invention, the instructions further cause the one or more processors to sequentially train the base parts for different tasks when connected to the corresponding task parts.

[0022] According to one embodiment of the present invention, at least two of the multiple task parts are trained on different data sets, wherein the base part includes columns or nodes trained on the different data sets by the following steps:

[0023] training a first task based on a first dataset of the different datasets when connected to a first of at least two of the plurality of task portions; and

[0024] training a second task based on a second dataset in a different dataset when connected to a second of at least two of the plurality of task portions;

[0025] Wherein the basis is constrained by EWC during training of at least one task.

[0026] According to one embodiment of the present invention, performing an automated or assisted driving task includes performing a driving maneuver.

[0027] According to one embodiment of the invention, performing an automated or assisted driving task includes providing a notification to the driver.

[0028] According to one embodiment of the present invention, the base part comprises a convolutional neural network (CNN).

[0029] According to the present invention, a system is provided, comprising a processor for performing one or more autonomous driving or assisted driving tasks based on a neural network, the neural network comprising:

[0030] A base for performing feature extraction for multiple tasks simultaneously on a single input dataset;

[0031] A plurality of subtask parts for performing a plurality of tasks based on feature extraction output from the base part;

[0032] Each of the multiple subtask parts includes nodes or layers of a neural network trained on a different training data set, and the base part includes nodes or layers of a neural network trained using each of the different training data sets constrained by elastic weight solidification to restrict the base part from forgetting previously learned tasks.

[0033] According to one embodiment of the present invention, the system further comprises a sensor for obtaining sensor data, wherein the sensor comprises one or more of a camera, a LIDAR system, and a radar system.

[0034] According to one embodiment of the invention, the system further comprises a control system for controlling one or more actuators, a display or an audio system of the vehicle.

[0035] According to one embodiment of the present invention, the base part and the plurality of sub-task parts provide outputs based on a single input data set.

[0036] According to one embodiment of the present invention, the base part comprises a convolutional neural network (CNN).

[0037] According to one embodiment of the invention, the base part comprises a recurrent neural network (RNN).

[0038] According to one embodiment of the present invention, the multiple sub-task parts output results of different tasks among the multiple tasks, and the multiple tasks include two or more of object detection, image segmentation, and lane finding. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the accompanying drawings, wherein, unless otherwise specified, like reference numerals throughout the various drawings represent like components. The advantages of the present invention will become better understood with respect to the following description and drawings, in which:

[0040] Figure 1 is a schematic block diagram illustrating an embodiment of a vehicle control system including an automated driving / assistance system;

[0041] Figure 2 is a schematic block diagram illustrating the structure of a neural network during training of a first task according to one embodiment;

[0042] Figure 3 is a schematic block diagram illustrating the structure of a neural network during training of a second task according to one embodiment;

[0043] Figure 4 is a schematic block diagram illustrating the structure of a neural network during training of a third task according to one embodiment;

[0044] Figure 5 is a schematic block diagram illustrating the structure of a neural network trained to perform multiple tasks according to one embodiment;

[0045] Figure 6 is a schematic block diagram showing components of an automated driving / assistance system according to one embodiment;

[0046] Figure 7 is a schematic flow chart illustrating a method for processing sensor data using a neural network according to one embodiment;

[0047] Figure 8 is a schematic flow chart showing a method for training a neural network according to one embodiment; and

[0048] Fig. 9 is a schematic block diagram illustrating a computing system according to one embodiment. DETAILED DESCRIPTION

[0049] In many machine learning applications, multiple tasks must be completed simultaneously using the same input or multiple inputs. For example, an autonomous vehicle may need to perform object detection, semantic scene segmentation, and lane recognition all using the same input image (or the same set of input images). Each of these tasks can benefit from using a convolutional neural network (CNN) for feature extraction. To significantly reduce computation time, the same CNN base network can be used for each task. However, it is often impossible to find data suitable for training each task simultaneously. In addition, if the base CNN is adjusted for each task sequentially, the weights will be overwritten each time a new task is added.

[0050] Applicants have recognized that using elastic weight consolidation (EWC) on a base CNN and then connecting to multiple other networks to complete various tasks can be used to reduce computational load while limiting "forgetting" of previously learned tasks. EWC has recently been used to enable a single deep neural network to sequentially learn how to play multiple Atari games. See James Kirkpatrick et al., "Overcoming catastrophic forgetting in neural networks", Proceedings of the National Academy of Sciences (2017), which describes the use of EWC and its entire contents are incorporated herein by reference. When switching to a new task, the weights of the network will be protected according to their importance to the previous task. In this way, the network retains a memory of how to perform the current task when learning how to complete a new task. In the present invention, the term "forgetting" is used to describe how the weights or values ​​learned in a neural network are lost or changed to the extent that the neural network can no longer perform tasks at the desired level. The term "memory" is used to describe how the weights or values ​​learned in a neural network can be retained even after further training, so that the neural network can continue to perform tasks at the desired level.

[0051] In the present invention, applicants propose and present methods, systems and devices for using EWC only on a subset of networks. Many deep learning tasks can be broken down into multiple stages, one of the first stages of which requires feature extraction using a CNN. If the same CNN is used for feature extraction across tasks, a lot of computational time can be saved. When teaching a CNN to extract appropriate features for a new task, it is prudent not to lose the ability to extract features for the old task. Embodiments are not limited to using a CNN constrained by an EWC. Instead, other network types such as a recurrent neural network (RNN) may be used in some embodiments.

[0052] Applicants propose using EWC on a base neural network (e.g., CNN) for a feature extraction network. This network can then be sequentially connected to multiple new networks to train different tasks, such as object detection, image segmentation, and lane finding using red, green, and blue (RGB) images. Once a CNN is tuned for each of these subtasks, the CNN can be connected to all networks during deployment to perform the required subtasks simultaneously. This enables each task to be trained sequentially on different datasets while simultaneously performing each subtask on new images during deployment.

[0053] According to one embodiment, a method for training a neural network for multiple deep learning tasks may include training a base portion and a first task portion of the neural network for a first task. The method may include subsequently training the base portion and the second task portion for performing a second task. The base portion is constrained using an EWC during training the second task to retain a memory of feature extraction used to perform the first task. Any number of additional tasks may also be trained while using the EWC to constrain the base portion. In one embodiment, training the first task includes training a first task portion that receives an output of the base portion and an unconnected second task portion, and training the second task includes training a second task portion that receives an output of the base portion and an unconnected first task portion.

[0054] According to one embodiment, a system or device may process a sensor data set using a neural network, the neural network comprising a base part and a plurality of task parts. The base part receives the sensor data set at an input layer and outputs to each of the plurality of task parts, wherein each of the plurality of task parts outputs a result of the task based on the output of the base part. The system may perform an automatic or assisted driving task based on the output of the plurality of task parts. The base part comprises layers of a neural network that are sequentially trained together for each of the plurality of tasks. The execution of the previous task is maintained by constraining the training of the base part of at least one task using an EWC.

[0055] The system may include a processor for performing one or more autonomous driving or assisted driving tasks based on a neural network. The neural network includes: a base part for performing feature extraction for multiple tasks on a single input data set simultaneously and multiple sub-task parts for performing multiple tasks based on the feature extraction output from the base part. Each of the multiple sub-task parts includes a node or layer of a neural network trained on a different training data set. The base part includes a node or layer of a neural network that constrains the base part to forget previously learned tasks trained using each of the different training data sets constrained by the EWC.

[0056] By using EWC on only a subset of the network, processes or tasks that are common between different neural networks can be combined into a single network. This can reduce computing resources during deployment because common tasks (e.g., feature extraction) are only performed once for each image, and this feature extraction information can then be used by the dedicated part of the network to perform the desired task. In addition, reduced training load or faster training can also be achieved because after training the first task, the base part of the network may only need to be changed slightly between the second task. In addition, using a subset of the network constrained by EWC can allow for reduced restrictions on training data. For example, if a neural network is to be trained for multiple tasks, a single training dataset with all appropriate labels may be required. Because the base neural network can be trained using EWC to avoid forgetting previous tasks, different training data labeled only for a single task or less than all desired tasks can be used during sequential training steps.

[0057] Now referring to the accompanying drawings, Figure 1 An exemplary vehicle control system 100 that can be used for autonomous or assisted driving is shown. The autonomous driving / assistance system 102 can be used to automate or control the operation of a vehicle or provide assistance to a human driver. For example, the autonomous driving / assistance system 102 can control one or more of the vehicle's braking, steering, acceleration, lights, alarms, driver notifications, radio, or any other assistance systems. In another example, the autonomous driving / assistance system 102 may not provide any control over driving (e.g., steering, acceleration, or braking), but may provide notifications and alarms to help human drivers drive safely. The autonomous driving / assistance system 102 may use a neural network or other model or algorithm to detect or locate an object based on perception data collected by one or more sensors.

[0058] The vehicle control system 100 also includes one or more sensor systems / devices for detecting the presence of objects near or within the sensor range of a parent vehicle (e.g., a vehicle including the vehicle control system 100). For example, the vehicle control system 100 may include one or more radar systems 106, one or more laser radar (LIDAR) systems 108, one or more camera systems 110, a global positioning system (GPS) 112, and / or one or more ultrasonic systems 114. The vehicle control system 100 may include a data storage 116 for storing relevant or useful data for navigation and safety (such as map data, driving history, or other data). The vehicle control system 100 may also include a transceiver 118 for wireless communication with a mobile or wireless network, other vehicles, infrastructure, or any other communication system.

[0059] The vehicle control system 100 may include vehicle control actuators 120 that control various aspects of the driving of the vehicle (such as electric motors, switches or other actuators) to control braking, acceleration, steering, etc. The vehicle control system 100 may also include one or more displays 122, speakers 124 or other devices so that notifications to human drivers or passengers can be provided. The display 122 may include a head-up display, a dashboard display or indicator, a display screen, or any other visual indicator that can be seen by the driver or passenger of the vehicle. The head-up display can be used to provide notifications or indicate the location of detected objects or to cover instructions or to assist the driver in driving maneuvers. The speaker 124 may include one or more speakers of the vehicle's sound system, or may include a speaker dedicated to driver notifications.

[0060] It should be understood that Figure 1 The embodiments shown are given as examples only. Other embodiments may include fewer or more components without departing from the scope of the present invention. In addition, the components shown may be combined or included in other components without limitation.

[0061] In one embodiment, the autonomous driving / assistance system 102 is configured to control the driving or navigation of the parent vehicle. For example, the autonomous driving / assistance system 102 can control the vehicle control actuator 120 to drive a path on a road, parking lot, lane, or other location. For example, the autonomous driving / assistance system 102 can determine the path based on information or perception data provided by any of the components 106-118. The sensor systems / devices 106-110 and 114 can be used to obtain real-time sensor data so that the autonomous driving / assistance system 102 can assist the driver or drive the vehicle in real time.

[0062] In one embodiment, the autonomous driving / assistance system 102 may use one or more neural networks to process sensor data, make decisions, or control the operation of the vehicle control system 100. The neural network may include a network previously trained for multiple tasks. For example, a neural network may be used to detect one or more different types of objects, road boundaries, drivable surfaces, etc. In fact, any detection, decision-making, or other task or decision performed by the autonomous driving / assistance system may be implemented at least in part using a neural network. In one embodiment, tasks that use similar types of data or have at least similar requirements in a stage may share common portions of one or more layers that have been trained or trained for different tasks when subject to EWC constraints.

[0063] Figure 2-4 is a schematic block diagram illustrating the training of layers or subsets of a neural network for multiple tasks with one or more common layers 202 constrained by EWC. Figure 2 The neural network 200 is shown during training of the common layer 202 and the first task layer 204 for a first task. The common layer 202 may receive training data, such as images corresponding to the first task and labels of the images. The labels may correspond to the desired output of the first task. Using a training algorithm such as a loss function and back propagation, the values ​​of the nodes or edges in the common layer 202 and the first task layer 204 may be modified. This process may be repeated for each image or data set within the training data. Execution of the neural network 200 for a large amount of test data may be used to confirm whether the neural network 200 performs acceptably. In addition, the EWC constraints of the common layer 202 in the form of a Fisher information matrix may be calculated to know which values ​​in the common layer 202 are important for the first task.

[0064] Figure 3 3 shows a neural network 300 during training of a common layer 202 and a second task layer 304 for a second task. The common layer 202 may receive training data, such as images and image labels corresponding to the second task. Note that the common layer 202 may include Figure 2The values ​​and weights determined during the training of the first task of the neural network 200 of the present invention. The labels may correspond to the desired output of the second task. Using a training algorithm such as a loss function and back propagation, the values ​​of nodes or edges in the common layer 202 and the second task layer 304 may be modified. Any modification of the common layer 202 may be constrained using EWC based on the constraints calculated for the first task. The process may be repeated for each image or data set within the training data. The training data may include different or the same training sets used during the training of the first task. For example, although the labels may be different, the images used in the training may be the same or different. Because different images may be used, it is not necessary to use the same images labeled for different tasks between training steps. This can significantly reduce the complexity of training because existing training data sets and their existing labels may be used. This is in contrast to the situation where exactly the same training data set may need to include labels for each desired task to be performed by the neural network.

[0065] After training, execution of neural network 300 against a large amount of test data can be used to confirm whether neural network 300 performs acceptably. For example, execution of the second task as well as the first task can be tested again using modified weights or values ​​in common layer 202. In addition, the EWC constraints of common layer 202 in the form of a Fisher information matrix can be calculated so that it is known which values ​​in common layer 202 are important for the first and second tasks.

[0066] Figure 4 The neural network 400 is shown during training of the common layer 202 and the third task layer 404 for the third task. The common layer 202 may receive training data, such as images and labels of the images corresponding to the third task. Note that the common layer 202 may include Figure 2-3 The values ​​and weights determined for the first and second tasks in the neural networks 200 and 300 of the present invention. The labels may correspond to the desired output of the third task. Using a training algorithm such as a loss function and back propagation, the values ​​of nodes or edges in the common layer 202 and the third task layer 404 may be modified. It should be noted that since the first task layer 204 and the second task layer 304 are not included during the training of the neural network 400 for the third task, the values ​​or weights in the task layer 204 and the second task layer 304 are not changed during the training of the third task. Any modification of the common layer 202 may be constrained using EWC based on the constraints calculated for the first and second tasks. The process may be repeated for each image or data set within the training data. The training data may include different or the same training sets used during the training of the first or second tasks. For example, although the labels may be different (specific to the third task), the images used in the training may be the same or different. Because different images may be used, it is not necessary to use the same images labeled for different tasks between training steps.

[0067] After training, execution of neural network 400 against a large amount of test data can be used to confirm whether neural network 400 performs acceptably. For example, execution of the second task as well as the first task can be tested again using modified weights or values ​​in common layer 202. In addition, the EWC constraints of common layer 202 in the form of a Fisher information matrix can be calculated so that it is known which values ​​in common layer 202 are important for the first, second, and third tasks.

[0068] Any additional tasks can be trained in a similar manner using the common layers as the basis or feature extraction part of the neural network.

[0069] Figure 5 is a schematic block diagram showing a neural network 500 that can be used to test execution during different tasks or deployments. The neural network 500 includes a common layer 202 that is trained during training for each of the first, second, and third tasks. The neural network 500 also includes a first task layer 204 that is trained during training for the first task, a second task layer 304 that is trained during training for the second task, and a third task layer 404 that is trained during training for the third task. The output of the common layer 202 is set as the input of each of the first task layer 204, the second task layer 304, and the third task layer 404. Therefore, the calculations in the common layer 202 only need to be performed once for training for all tasks. Because EWC is used to constrain the common layer 202 during training, the execution of each of the training tasks may still be very good.

[0070] For example, a single camera image (or other single sensor data set) can be provided to the common layer 202 for use as the base portion of the neural network 500. The common layer 202 performs feature extraction on the image and provides one or more output values ​​reflecting the feature extraction. Because the common layer 202 uses EWC to train each task, a single feature extraction extracts the features required for all tasks. The feature extraction values ​​are output to the subtask portion including the first task layer 204, the second task layer 304, and the third task layer 404. Each of the first task layer 204, the second task layer 304, and the third task layer 404 processes the feature extraction values ​​from the common layer 202 to determine the output of its respective task.

[0071] It will be appreciated by those skilled in the art that a single neural network 500 is composed of multiple nodes and edges connecting the nodes. The weight or value of an edge or node is used to calculate the output of the edge connected to the subsequent node. A single neural network 500 may therefore be composed of multiple neural networks to perform one or more tasks. Figure 5The neural network 500 includes some common layers 202 as the base or common part of the neural network 500. The common layers 202 can be understood as forming sub-networks of the neural network 500. The calculations and processing completed in the common layers 202 are then used by the first task layer 204, the second task layer 206, and the third task layer 404. Therefore, the neural network 500 includes a branch topology, in which the results of the common layers 202 are then used independently by each of the multiple sub-networks in the branches of the neural network 500. Because the common layers 202 are trained sequentially on multiple tasks when using EWC to avoid forgetting previously trained tasks, the common layers 202 can perform tasks that serve each neural network branch well. In addition, the common layers 202 result in a reduction in computation because the tasks of the common layers 202 are performed once for all tasks represented by the branches, rather than once for each task. An example of a task performed by the common layers 202 is feature extraction. However, any task that may have a shared initial processing task can share a common layer.

[0072] Figure 6 is a schematic block diagram showing some components of the autonomous driving / assistance system 102 according to one embodiment. The autonomous driving / assistance system 102 includes one or more processors 602, a sensor data component 604, a neural network component 606, and a control component 608. Components 602-608 are given by way of illustration only and may not be included in all embodiments. In fact, some embodiments may include only one or any combination of two or more of the components 602-608. For example, some of the components 602-608 may be located outside or separate from the autonomous driving / assistance system 102. In addition, components 602-608 may include hardware, processors, computer-readable instructions, or a combination of both to perform functions and provide the structure discussed herein.

[0073] Processor 602 may include any type of processor or processing circuit. In one embodiment, processor 602 may include a conventional desktop, laptop, or server central processing unit (CPU). In one embodiment, processor 602 may include multiple parallel processors, such as those found in a graphics processing unit (GPU) or an accelerated processing unit (APU). Parallel processing may be helpful in performing the calculations required for a neural network.

[0074] The sensor data component 604 obtains sensor data from one or more sensors. For example, the sensor data component 604 can receive sensor data from one or more of the radar system 106, the LIDAR system 108, the camera system 110, the GPS 112, the ultrasonic system, or other sensors or sensing systems. The sensor data can be used for various tasks in the driving environment, such as object detection, image segmentation, lane detection, driving surface detection, driving condition detection, or any other task. The sensor data component 604 can provide the sensor data to the neural network component 606. For example, the sensor data component 604 can provide a single red green blue (RGB) image, infrared image, LIDAR data frame, or radar data frame at a time for processing by the neural network.

[0075] The neural network component 606 can store a neural network and / or perform calculations based on the neural network and / or values ​​or weights within the neural network to process sensor data. In one embodiment, the neural network component 606 processes a sensor data set (such as a frame or other data set) using a neural network including a base portion and multiple task portions. The base portion may include a common layer that performs a task (such as feature extraction) for each of the multiple task portions. The base portion or common layer can receive the sensor data set at an input layer and output to each of the multiple task portions. Each of the multiple task portions outputs a result of the task based on the output of the base portion.

[0076] The base part or the common layer of the base part may include weights or values ​​for training of multiple different tasks. During training, EWC may be used to constrain weights or values ​​to limit forgetting of training of previous task learning. The base part may include a CNN or RNN that performs a task that will benefit each specific task or subtask to be performed by the task part. In one embodiment, at least two of the multiple task parts are trained for different data sets. When multiple task parts are trained for different data sets, the base part (or common layer) may also be trained sequentially on different data sets by training the first task based on the first data set of different data sets when connected to the first of at least two of the multiple task parts and training the second task based on the second data set of different data sets when connected to the second of at least two of the multiple task parts. The base part is constrained by EWC during training at least one task.

[0077] The control component 608 can cause the autonomous driving / assistance system 102 to control the vehicle control system based on the output of the neural network component 606. For example, the output of each of the first task layer 204, the second task layer 304, and the third task layer 404 can be used to select a notification or driving maneuver to be performed by the autonomous driving / assistance system 102. For example, the control component 608 can select a notification to be provided to the driver or a driving maneuver to be performed by the vehicle based on the location of the detected object. The driving maneuver can include controlling steering, braking, or acceleration to avoid collision with an obstacle or driving to a destination. The notification can include warning of the presence of pedestrians or other objects in or near the road. The notification can be provided using the vehicle's head-up display, audio system, or dashboard display.

[0078] Figure 7 is a diagram showing a method for using a neural network such as Figure 5 Schematic flow chart of a method 700 for processing sensor data using a neural network 500 (e.g., a processor) of a neural network 500. For example, a processor may process an image frame or other sensor data set based on the structure and weights of the neural network. The method 700 begins and the neural network 500 receives 702 sensor data at an input layer. The method 700 includes processing 704 the sensor data using one or more common layers of the neural network. For example, the processor may process the sensor data based on the weights and values ​​of the base portion of the neural network 505 or the common layer 202. For example, the common layer 202 may include layers that train various tasks sequentially using EWC. The common layer 202 may include layers that process sensor data and produce outputs used by one or more branches of the neural network. For example, a branch may include a sub-portion of a neural network that performs a sub-task, such as object recognition (a specific object type), lane detection, or image segmentation. The method includes processing 706 the output of the common layer using multiple sub-portions (sub-networks) that perform multiple different sub-tasks. For example, each of the first task layer 204, the second task layer 304, and the third task layer 404 may constitute a different sub-network or branch of the neural network. Each sub-portion or sub-network may provide an output for the task indicating a classification or other result of the task. The automated driving / assistance system 102 may then perform 708 the maneuver or provide a notification to the driver based on the results of the different tasks or sub-tasks.

[0079] Figure 8800 is a schematic flow chart showing a method 800 for training a neural network. In one embodiment, the method 800 may be performed by a computing system before being deployed in a real world or online environment. The method 800 includes training 802 a base part and a first task part to perform a first task. The base part includes one or more layers or nodes that will be shared during the processing of multiple tasks. The base part may include a CNN, an RNN, or other deep neural network. The method includes training 804 a base part and an additional task part for additional tasks. During training 804, an EWC constraint is provided on the base part to limit or avoid forgetting the previous task. The method 800 includes determining 806 whether additional tasks need to be trained. If so (yes at 806), the method 800 trains 804 the base part and the additional task part for the additional task while limiting forgetting in the base part. Training 804 the base part and the additional task part may be performed multiple times as needed to train all tasks. After all tasks have been trained (no at 806), the base part will train multiple tasks using EWC, and each task part has been trained once for the corresponding task. The method 800 includes combining 808 the base portion and the plurality of task portions into a single neural network for testing or deployment. The base portion and the task portions are combined such that each task portion receives an output of the base portion and performs a task based at least in part on the output of the base portion.

[0080] Reference now Fig. 9 , a block diagram of an exemplary computing device 900 is shown. The computing device 900 can be used to perform various processes, such as the processes discussed herein. In one embodiment, the computing device 900 can be used as an autonomous driving / assistance system 102, a vehicle control system 100, a neural network training system, etc. The computing device 900 can perform various monitoring functions as discussed herein, and can execute one or more applications, such as the applications or functions described herein. The computing device 900 can be any of a variety of computing devices, such as a desktop computer, a dashboard computer, a vehicle control system, a notebook computer, a server computer, a handheld computer, a tablet computer, etc.

[0081] The computing device 900 includes one or more processors 902, one or more memory devices 904, one or more interfaces 906, one or more mass storage devices 908, one or more input / output (I / O) devices 910, and a display device 930, all of which are connected to a bus 912. The processor 902 includes one or more processors or controllers that execute instructions stored in the memory device 904 and / or the mass storage device 908. The processor 902 may also include various types of computer-readable media, such as cache memory.

[0082] The memory device 904 includes various computer-readable media such as volatile memory (e.g., random access memory (RAM) 914) and / or nonvolatile memory (e.g., read-only memory (ROM) 916). The memory device 904 may also include a rewritable ROM such as a flash memory.

[0083] The mass storage device 908 includes various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., flash memory), etc. Fig. 9 As shown, the particular mass storage device is a hard disk drive 924. Various drives may also be included in the mass storage device 908 to enable reading from and / or writing to various computer-readable media. The mass storage device 908 includes removable media 926 and / or non-removable media.

[0084] I / O devices 910 include various devices that allow data and / or other information to be entered into or retrieved from computing device 900. Exemplary I / O devices 910 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, and the like.

[0085] Display device 930 includes any type of device capable of displaying information to one or more users of computing device 900. Examples of display device 930 include a monitor, a display terminal, a video projection device, and the like.

[0086] The interface 906 includes various interfaces that allow the computing device 900 to interact with other systems, devices, or computing environments. Exemplary interfaces 906 may include any number of different network interfaces 920, such as interfaces to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet. Other interfaces include a user interface 918 and a peripheral device interface 922. The interface 906 may also include one or more user interface elements 918. The interface 906 may also include one or more peripheral interfaces, such as interfaces for a printer, a pointing device (a mouse, a tracking pad, or any suitable user interface now known or later discovered by a person of ordinary skill in the art), a keyboard, etc.

[0087] The bus 912 allows the processor 902, memory device 904, interface 906, mass storage device 908, and I / O device 910 to communicate with each other and with other devices or components coupled to the bus 912. The bus 912 represents one or more of several types of bus structures, such as a system bus, a peripheral controller interface (PCI) bus, an IEEE bus, a universal serial bus (USB) bus, and the like.

[0088] For purposes of illustration, programs and other executable program components are shown herein as discrete blocks, but it is understood that such programs and components may reside in different storage components of the computing device 900 at different times and be executed by the processor 902. Alternatively, the systems and programs described herein may be implemented in hardware or a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) may be programmed to perform one or more of the systems and programs described herein.

[0089] Example

[0090] The following examples relate to other embodiments.

[0091] Example 1 is a method comprising processing a sensor data set using a neural network, the neural network comprising a base portion and a plurality of task portions, wherein the base portion receives the sensor data set at an input layer and outputs to each of the plurality of task portions, wherein each of the plurality of task portions outputs a result of the task based on an output of the base portion. The method comprises executing an automatic or assisted driving task based on the outputs of the plurality of task portions. The base portion comprises layers of a neural network sequentially trained for each of the plurality of tasks, wherein the training of the base portion for at least one task is constrained by using an EWC to maintain execution of a prior task.

[0092] In Example 2, the multiple task parts in Example 1 each output a result of a different task of the multiple tasks, at least one of the multiple tasks including one or more of object detection, image segmentation, or lane finding.

[0093] In Example 3, the sensor data set of any one of Examples 1-2 comprises a sensor data frame captured by one or more sensors, wherein the sensor data frame comprises at least one of an RBG image, an infrared image, a LIDAR data frame, or a radar data frame.

[0094] In Example 4, the method of any one of Examples 1-3 includes sequentially training base parts for different tasks while connecting to corresponding task parts.

[0095] In Example 5, at least two of the plurality of task parts in any one of Examples 1-4 are trained on different data sets. The base part includes columns or nodes trained on different data sets by training a first task based on a first data set in the different data sets when connected to a first of the at least two of the plurality of task parts and training a second task based on a second data set in the different data sets when connected to a second of the at least two of the plurality of task parts. The base part is constrained by the EWC during training of at least one task.

[0096] In Example 6, the method of any one of Examples 1-5 includes: performing the automated or assisted driving task includes performing a driving maneuver.

[0097] In Example 7, performing the automated or assisted driving task in any of Examples 1-6 includes providing a notification to the driver.

[0098] In Example 8, the base portion of any one of Examples 1-7 includes a CNN.

[0099] Example 9 is a system including a processor for performing one or more autonomous driving or assisted driving tasks based on a neural network. The neural network includes a base part for performing feature extraction for multiple tasks on a single input data set simultaneously. The system includes multiple subtask parts that perform multiple tasks based on feature extraction output from the base part. Each of the multiple subtask parts includes a node or layer of a neural network trained on a different training data set, and the base part includes a node or layer of a neural network trained using each of the different training data sets constrained by elastic weight solidification to restrict the base part from forgetting previously learned tasks.

[0100] In Example 10, the system of Example 9 further includes a sensor for obtaining sensor data, wherein the sensor includes one or more of a camera, a LIDAR system, and a radar system.

[0101] In Example 11, the system of any of Examples 9-10 further includes a control system for controlling one or more actuators, a display, or an audio system of the vehicle.

[0102] In Example 12, the base part and the plurality of subtask parts in any one of Examples 9-11 provide outputs based on a single input data set.

[0103] In Example 13, the base portion in any one of Examples 9-12 includes a CNN.

[0104] In Example 14, the base portion of any one of Examples 9-12 includes a RNN.

[0105] In Example 15, the multiple subtask portions in any one of Examples 9-14 output results of different tasks in the multiple tasks, and the multiple tasks include two or more of object detection, image segmentation, and lane finding.

[0106] Example 16 is a method for training a neural network for multiple deep learning tasks. The method includes training a common portion and a first task portion of the neural network for a first task. The method includes training the common portion and the second task portion for performing a second task, wherein the common portion is constrained using an EWC to retain a memory of feature extraction used to perform the first task. Training the first task includes training a first task portion that receives an output of the common portion and an unconnected second task portion, and training the second task includes training a second task portion that receives an output of the common portion and an unconnected first task portion.

[0107] In Example 17, the training the first task of Example 16 includes training using a first training dataset labeled for the first task, and wherein training the second task includes training using a second training dataset labeled for the second task.

[0108] In Example 18, the method in any one of Examples 16-17 also includes connecting the first task portion and the second task portion to a common portion during testing or deployment, wherein the common portion simultaneously performs feature extraction of the first task and the second task and provides output to the first task portion and the second task portion, wherein the output is a result of the feature extraction of the first task and the second task.

[0109] In Example 19, the method of any one of Examples 16-28 further includes determining which weights are important for a particular task, and applying the EWC constraint to those tasks during training of one or more subsequent tasks.

[0110] In Example 20, the first task or the second task in any one of Examples 16-19 includes one or more of object detection, image segmentation, and lane finding.

[0111] Example 21 is a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method or implement a system or device as in any of Examples 1-20.

[0112] Example 22 is a system or apparatus including means for implementing a method or realizing a system or device as in any of Examples 1-20.

[0113] With reference to the accompanying drawings, the components of this document are described in the above invention, and are shown herein by describing specific embodiments in which the disclosure can be practiced. It is understood that other embodiments can be used and the structure can be changed without departing from the scope of the present invention. In the specification, the content related to "one embodiment", "embodiment", "exemplary embodiment" and the like indicates that the described embodiment may include special features, structures or characteristics, but each embodiment may not include the special features, structures or characteristics. In addition, such phrases may not necessarily refer to the same embodiment. Moreover, when describing special features, structures or characteristics in conjunction with an embodiment, it can be claimed that whether or not the description is clear within the common knowledge of those skilled in the art affects such features, structures or characteristics in conjunction with other embodiments.

[0114] The embodiments of the systems, devices and methods disclosed herein may include or use a special-purpose computer or a general-purpose computer, as discussed herein, which includes computer hardware, such as one or more processors and system memory. Embodiments within the scope of the present invention may also include traditional media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available medium, which can be accessed by a general-purpose computer system or a special-purpose computer system. The computer-readable medium storing computer-executable instructions is a computer storage medium (device). The computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example but not limited thereto, embodiments of the present invention include at least two significantly different types of computer-readable media: a computer storage medium (device) and a transmission medium.

[0115] Computer storage media (devices) include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk-ROM, solid-state drive (SSD) (e.g., RAM-based), flash memory, phase change memory (PCM), other types of memory, other optical disk storage, magnetic disk storage or other magnetic disk storage devices, or any other medium that can be used to store the required program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

[0116] The embodiments of the system, device and method disclosed herein can communicate through a computer network. "Network" is defined as one or more data connections that can transmit electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred or provided to a computer through a network or other communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer appropriately regards the connection as a transmission medium. The transmission medium may include a network and / or data connection, which may include a program code device that can be used to carry the needs in the form of a computer executable instruction or data structure and which can be accessed by a general-purpose computer or a special-purpose computer. The above combination should also be considered to be included in the scope of computer-readable media.

[0117] For example, computer executable instructions include instructions and data that, when executed in a processor, cause a general purpose computer, a special purpose computer, or a special purpose processing device to perform a function or group of functions. For example, computer executable instructions can be binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter is described in a specific language of structural features and / or methodological behaviors, it is understood that the subject matter defined by the dependent claims is not necessarily limited to the features described or the behaviors described herein. Of course, the described features and behaviors are described as exemplary forms of implementing the claims.

[0118] It will be appreciated by those skilled in the art that the present invention can be practiced in a network computing environment with many types of computer system configurations, including onboard computers, personal computers, desktop computers, notebook computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable home electronics, network personal computers (networkPC), microcomputers, mainframe computers, mobile phones, personal digital assistants (PDAs), tablet computers, pagers, routers, switches, various storage devices, etc. The present invention can also be practiced in a distributed system environment, where local computer systems and remote computer systems linked by a network (by a hardwired data link, a wireless data link, or a combination of a hardwired data link and a wireless data link) both perform tasks. In a distributed system environment, program modules can be located in local storage devices and remote storage devices.

[0119] Moreover, where appropriate, the functions described herein may be performed by one or more of the following components: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) may be programmed to implement one or more systems and programs described herein. Certain terms used throughout the specification and claims relate to specific system components. The terms "module" and "component" are used in the names of certain components to reflect their implementation independence in software, hardware, circuits, sensors, etc. As will be appreciated by those skilled in the art, components may be referred to by different names. It is not intended herein to distinguish between components that are distinguished in name but not in function.

[0120] It should be noted that the sensor embodiments discussed above include computer hardware, software, firmware, or any combination thereof to perform at least a portion of their functions. For example, the sensor may include computer code configured to be executed in one or more processors, and may include hardware logic circuits / hardware electronic circuits controlled by the computer code. These exemplary devices are provided herein for illustrative purposes and are not intended to be limiting. Embodiments of the present invention may be implemented in other types of devices known to those skilled in the art.

[0121] At least some embodiments of the present invention relate to a computer program product including such logic (e.g., in the form of software) stored on any computer-usable medium. Such software, when executed in one or more data processing devices, enables the device to operate as described herein.

[0122] Although various embodiments of the present invention have been described above, it should be understood that they are merely exemplary and non-restrictive. It is obvious to those skilled in the relevant art that various changes can be made in form and detail without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention should not be limited by any of the above exemplary embodiments, but should only be limited according to the attached claims and their equivalents. For the purpose of illustration and description, the foregoing description is presented. It is not exhaustive, nor is it intended to limit the disclosure to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. In addition, it should be noted that any or all of the above alternative embodiments can be used in any desired combination to form another hybrid embodiment of the present invention.

[0123] Furthermore, although specific embodiments of the invention have been described and shown, the invention is not limited to the specific forms or arrangements of parts so described and shown. The scope of the invention is defined by the appended claims, any future claims filed herein and in a different application, and their equivalents.

Claims

1. A method for training a neural network for multiple deep learning tasks, the method comprising: training the common portion and the first-task portion of the neural network for the first task; as well as Training the common part and the second task part for performing a second task; The common part is trained during training of each of the first task and the second task, and the output of the common part is set as the input of the first task part and the second task part; The common portion is constrained using elastic weight solidification (EWC) to preserve memory for feature extraction used to perform the first task.

2. The method of claim 1 , wherein training the first task comprises training using a first training data set labeled for the first task, and wherein training the second task comprises training using a second training data set labeled for the second task, wherein the first task comprises a first autonomous driving task, and the second task comprises a second autonomous driving task, and wherein EWC is not used for the first task portion and the second task portion.

3. A method according to claim 1, wherein training for the first task includes training the first task part that receives the output of the common part and the unconnected second task part, and wherein training for the second task includes training the second task part that receives the output of the common part and the unconnected first task part, the method also includes connecting the first task part and the second task part to the common part during testing or deployment, wherein the common part simultaneously performs feature extraction for the first task and the second task and provides output to the first task part and the second task part, wherein the output is the result of feature extraction for the first task and the second task. 4 . The method of claim 1 , further comprising determining which weights are important for a particular task and applying the EWC constraints to those tasks during training for one or more subsequent tasks. 5 . The method according to claim 1 , wherein the first task or the second task comprises one or more of object detection, image segmentation, and lane finding.

6. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Processing a sensor data set using a neural network, the neural network comprising a base part and a plurality of task parts, wherein the base part receives the sensor data set at an input layer and outputs to each of the plurality of task parts, wherein each of the plurality of task parts outputs a result of a task based on the output of the base part; as well as performing an automated or assisted driving task based on the outputs of the plurality of task portions; wherein the base portion comprises layers of a neural network trained sequentially for each of the plurality of tasks, the base portion being trained during training of each of the plurality of tasks; wherein the training of the basis for at least one task is constrained to preserve performance of previous tasks by using elastic weight curing (EWC).

7. The non-transitory computer-readable storage medium of claim 6, wherein the plurality of task portions output results of different tasks in a plurality of tasks, at least one of the plurality of tasks comprising one or more of object detection, image segmentation, or lane finding.

8. The non-transitory computer-readable storage medium of claim 6, wherein the sensor data set comprises a sensor data frame captured by one or more sensors, wherein the sensor data frame comprises at least one of the following: Red-Green-Blue (RBG) images; Infrared images; Light Detection and Ranging (LIDAR) data frames; and Radar data frame.

9. The non-transitory computer-readable storage medium of claim 6, wherein the instructions further cause the one or more processors to sequentially train the base parts for different tasks when connected to corresponding task parts.

10. The non-transitory computer-readable storage medium of claim 6, wherein at least two of the plurality of task parts are trained on different data sets, wherein the base part comprises columns or nodes trained on the different data sets by: training a first task based on a first dataset of the different datasets when connected to a first of the at least two of the plurality of task portions; as well as training a second task based on a second one of the different datasets when connected to a second one of the at least two of the plurality of task portions; Wherein said base portion is constrained by EWC during training of at least one of said tasks.

11. The non-transitory computer-readable storage medium of claim 6, wherein one or more of the following: Performing the automated or assisted driving task includes performing a driving maneuver; performing the automated or assisted driving task includes providing a notification to the driver; or The basic part includes convolutional neural network (CNN).

12. A system comprising a processor for performing one or more autonomous driving or assisted driving tasks based on a neural network, the neural network comprising: A base for performing feature extraction for multiple tasks simultaneously on a single input dataset; performing a plurality of subtask parts of the plurality of tasks based on feature extraction output from the base part; wherein each of the plurality of subtask parts comprises a node or layer of a neural network trained on a different training data set, the base part is trained during the training of each of the plurality of subtask parts, and the output of the base part is set as the input of the plurality of subtask parts; And wherein the base part comprises nodes or layers of a neural network trained using each of the different training data sets constrained by elastic weight solidification to restrict the base part from forgetting previously learned tasks.

13. The system of claim 12, further comprising one or more of the following: A sensor for obtaining sensor data, wherein the sensor comprises one or more of a camera, a LIDAR system, and a radar system; or A control system for controlling one or more actuators, displays, or audio systems of a vehicle.

14. The system of claim 12, wherein one or more of the following: The base portion and the plurality of subtask portions provide outputs based on a single input data set; The base part includes a convolutional neural network (CNN); or The underlying part includes a recurrent neural network (RNN).

15. The system of claim 12, wherein the plurality of subtask parts output results of different tasks among the plurality of tasks, the plurality of tasks comprising two or more of object detection, image segmentation, and lane finding.

Images