Systems and methods for deep multi-task learning for embedded machine vision applications

Abstract

A computer-implemented method includes receiving data generated using at least one sensor of a vehicle; and using a multi-task neural network to simultaneously perform multiple different prediction tasks on the data, wherein the multi-task neural network includes at least one shared parameter inference matrix, the shared parameter inference matrix includes parameters shared between the multiple different prediction tasks, and the at least one shared parameter inference matrix is ​​over-parameterized during training into at least one shared parameter matrix and multiple task-specific parameter matrices, each of the multiple task-specific parameter matrices is associated with a different task in the multiple different tasks.

Description

Technical Field

[0001] The present disclosure relates generally to machine vision and, more particularly, to deep multi-task learning for embedded machine vision applications. Background Art

[0002] The growing interest in autonomous vehicles has led to the development of sophisticated embedded vision systems that can capture large amounts of scene data and then apply machine learning strategies to perform the perception tasks that enable autonomous vehicles to make decisions while driving. The size of state-of-the-art machine learning models has grown significantly in recent years. Large models show great expressiveness but are difficult to deploy in practice for embedded vision applications where the inference system has relatively limited computational power and relatively low memory. Additionally, perception tasks for autonomous driving often require real-time or near real-time inference to ensure safety, but large models are typically slow to infer.

[0003] In order to reduce inference time and improve joint performance with limited computational budget, multi-task learning (MTL) is a common design choice for embedded perception tasks when multiple related tasks are to be learned and reasoned simultaneously. In general, the motivations of MTL can be multifaceted, including: alleviating data sparsity challenges by leveraging useful information from other tasks, reducing inference time with limited computational budget in terms of both memory and computational bandwidth, improving joint performance of all tasks, and reducing training time. However, there is a trade-off between the performance of each individual task and the size of multi-task model parameters. Summary of the invention

[0004] According to various embodiments, the system and method include an MTL neural network architecture that is overparameterized during training for better optimization and generalization and reduced to a compact architecture for inference, thereby providing the benefits associated with a smaller neural network. The MTL neural network includes at least one overparameterized convolutional layer, in which the weight matrix of the shared convolutional layer is factorized into a shared component and a task-specific component associated with each task. During training, each task-specific component is updated using the corresponding task loss function, while the other task-specific components and the shared components remain frozen; then, the shared components are updated using the combined loss function of all tasks, and all task-specific components are frozen (not updated). Once the model is trained, an inference model is generated by shrinking the factorized convolutional layers into a smaller set of weights using matrix multiplication on the shared components and task-specific components. This combines the benefits of improved optimization and generalization provided by a larger number of parameters during training with the faster inference performance associated with a smaller neural network, making the MTL neural network an ideal choice for embedded machine vision applications.

[0005] According to one aspect, a computer-implemented method includes receiving data generated using at least one sensor of a vehicle; and using a multi-task neural network to simultaneously perform multiple different prediction tasks on the data, wherein the multi-task neural network includes at least one shared parameter inference matrix, the shared parameter inference matrix including parameters shared between the multiple different prediction tasks, and the at least one shared parameter inference matrix is ​​over-parameterized during training into at least one shared parameter matrix and multiple task-specific parameter matrices, each of the multiple task-specific parameter matrices being associated with a different task in the multiple different tasks.

[0006] Optionally, the plurality of different prediction tasks comprises semantic segmentation, depth estimation, object detection, instance segmentation or surface normal estimation.

[0007] Optionally, the multi-task neural network comprises task-specific parameter inference matrices associated with the plurality of different prediction tasks.

[0008] Optionally, during training of the multi-task learning model, each of the plurality of task-specific parameter matrices is independently trained using a corresponding task-specific loss function.

[0009] Optionally, during training of the multi-task learning model, the at least one shared parameter matrix is ​​trained independently of the plurality of task-specific parameter matrices using a shared task loss function.

[0010] Optionally, the method further comprises automatically controlling the vehicle based on the results of the plurality of different prediction tasks.

[0011] Optionally, automatically controlling the vehicle based on the results of the plurality of different predictive tasks comprises steering the vehicle, braking the vehicle, or providing an alert to a driver of the vehicle.

[0012] According to one aspect, a computing system includes one or more processors and a memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for: receiving data generated using at least one sensor of a vehicle; and using a multi-task neural network to simultaneously perform multiple different prediction tasks on the data, wherein the multi-task neural network includes at least one shared parameter inference matrix, the shared parameter inference matrix including parameters shared between the multiple different prediction tasks, and the at least one shared parameter inference matrix is ​​over-parameterized during training into at least one shared parameter matrix and multiple task-specific parameter matrices, each of the multiple task-specific parameter matrices being associated with a different task among the multiple different tasks.

[0013] Optionally, the computing system is an embedded computing system of the vehicle.

[0014] Optionally, the plurality of different prediction tasks comprises semantic segmentation, depth estimation, object detection, instance segmentation or surface normal estimation.

[0015] Optionally, the multi-task neural network comprises task-specific parameter inference matrices associated with the plurality of different prediction tasks.

[0016] Optionally, during training of the multi-task learning model, each of the plurality of task-specific parameter matrices is independently trained using a corresponding task-specific loss function.

[0017] Optionally, during training of the multi-task learning model, the at least one shared parameter matrix is ​​trained independently of the plurality of task-specific parameter matrices using a shared task loss function.

[0018] Optionally, the one or more programs include instructions for automatically controlling the vehicle based on the results of the plurality of different predictive tasks.

[0019] According to one aspect, a method for generating a multi-task machine learning neural network includes training the multi-task learning neural network based on multiple training data sets, the multi-task learning neural network including: multiple task-specific modules, each task-specific module is dedicated to a different task, and a shared module, the shared module including at least one over-parameterized convolutional layer, the at least one over-parameterized convolutional layer including at least one shared parameter matrix and multiple task-specific parameter matrices; and generating a multi-task machine learning neural network for simultaneously performing multiple different prediction tasks based on the data sets, the multi-task machine learning neural network including the trained multiple task-specific modules and the trained shared modules, which includes a matrix, which is a combination of the at least one trained shared parameter matrix and the trained multiple task-specific parameter matrices.

[0020] Optionally, a plurality of different prediction tasks include semantic segmentation, depth estimation, object detection, instance segmentation, or surface normal estimation.

[0021] Optionally, training the multi-task learning network comprises updating factors of one parameter matrix among the plurality of task-specific parameter matrices using a task-specific loss function while freezing factors of any other parameter matrices among the plurality of task-specific parameter matrices.

[0022] Optionally, training the multi-task learning network comprises updating factors of the at least one shared parameter matrix using a shared loss function while freezing parameters of the plurality of task-specific parameter matrices.

[0023] Optionally, the multi-task learning model comprises a plurality of over-parameterized convolutional layers.

[0024] Optionally, the matrix multiplication of the trained at least one shared parameter matrix and the trained plurality of task-specific parameter matrices comprises an element-wise product of the trained plurality of task-specific parameter matrices.

[0025] Optionally, the output of the shared module is provided as input to each of the plurality of task-specific modules, and each task-specific module is not interconnected with any other task-specific module.

[0026] Optionally, the matrix being a combination of the at least one trained shared parameter matrix and the multiple trained task-specific parameter matrices is generated by multiplying the at least one shared parameter matrix by a Hadamard product matrix of the multiple trained task-specific parameter matrices.

[0027] Optionally, the multi-task learning neural network is trained based on a jointly labeled training dataset, a non-jointly labeled training dataset, or a combination of jointly labeled and non-jointly labeled training datasets.

[0028] According to one aspect, a computing system includes one or more processors and a memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for: training a multi-task learning neural network based on multiple training data sets, the multi-task learning neural network including multiple task-specific modules, each task-specific module dedicated to a different task, and a shared module, the shared module including at least one over-parameterized convolutional layer, the at least one over-parameterized convolutional layer including at least one shared parameter matrix and multiple task-specific parameter matrices; and generating a multi-task machine learning neural network for simultaneously performing multiple different prediction tasks based on the data set, the multi-task machine learning neural network including multiple trained task-specific modules and trained shared modules, which includes a matrix, which is a combination of the at least one trained shared parameter matrix and the multiple trained task-specific parameter matrices.

[0029] Optionally, the plurality of different prediction tasks comprises semantic segmentation, depth estimation, object detection, instance segmentation or surface normal estimation.

[0030] Optionally, training the multi-task learning network comprises updating factors of one parameter matrix among the plurality of task-specific parameter matrices using a task-specific loss function while freezing factors of any other parameter matrices among the plurality of task-specific parameter matrices.

[0031] Optionally, training the multi-task learning network comprises updating factors of the at least one shared parameter matrix using a shared loss function while freezing parameters of the plurality of task-specific parameter matrices.

[0032] Optionally, the multi-task learning model comprises a plurality of over-parameterized convolutional layers.

[0033] Optionally, the matrix multiplication of the trained at least one shared parameter matrix and the trained plurality of task-specific parameter matrices comprises an element-wise product of the trained plurality of task-specific parameter matrices.

[0034] Optionally, the output of the shared module is provided as input to each of the plurality of task-specific modules, and each task-specific module is not interconnected with any other task-specific module.

[0035] Optionally, the matrix that is a combination of the at least one trained shared parameter matrix and the multiple trained task-specific parameter matrices is a matrix multiplication of the Hadamard product of the at least one shared parameter matrix and the multiple trained task-specific parameter matrices.

[0036] Optionally, the multi-task learning neural network is trained based on a jointly labeled training dataset, a non-jointly labeled training dataset, or a combination of jointly labeled and non-jointly labeled training datasets.

[0037] According to one aspect, a non-transitory computer-readable medium stores a data structure generated by training a multi-task learning neural network according to any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0039] Figure 1 An exemplary MTL neural network that can be used for embedded machine vision applications is shown;

[0040] Figure 2 shows over-parameterization of convolutional layers according to various embodiments;

[0041] Figure 3 is a block diagram of a method for constructing an MTL neural network configured to perform n tasks simultaneously according to various embodiments;

[0042] Figure 4 An embedded system 400 is shown that can implement an MTL neural network according to various embodiments;

[0043] Figure 5is a flowchart of a method for performing multiple tasks on an image using a trained MTL neural network according to various embodiments; and

[0044] Figure 6 An example of a computing system in accordance with various embodiments is shown. DETAILED DESCRIPTION

[0045] Reference will now be made in detail to embodiments and examples of various aspects and variations of the systems and methods described herein. Although several exemplary variations of systems and methods are described herein, other variations of systems and methods may include aspects of the systems and methods described herein combined in any suitable manner with combinations of all or some of the described aspects.

[0046] Systems and methods according to the principles described herein provide deep multi-task learning neural networks with improved performance while maintaining a limited set of inference model parameters that are directly related to the inference computational budget, memory, and inference time. Deep MTL neural networks can be applied to multi-task embedded vision applications, particularly real-time perception tasks for autonomous driving. MTL neural networks are not limited to embedded vision applications and can be applied to other application areas where multiple similar perception tasks need to be performed. According to the principles described herein, MTL neural networks can be used for data sets from different types of perception sensors, including, for example, images from cameras, radar data sets, lidar data sets, ultrasonic data sets, etc. According to the principles described herein, MTL neural networks can be used for any type of autonomous or semi-autonomous driving vehicles, including cars and trucks, passenger aircraft, unmanned aerial vehicles (UAVs), trains, boats, ships, etc.

[0047] According to various embodiments, the fully connected layers and convolutional layers of the MTL neural network are replaced by over-parameterized layers, and the over-parameterized parameters are shared between different tasks to achieve higher performance, thereby reducing the inference parameter size and reducing computational cost. Specifically, tensor factorization is used for model expansion rather than model compression during training. The full-rank diagonal tensor is further expanded to be trained separately for each task, while other tensors are shared between all tasks. For reasoning, the factorized tensors shared between multiple tasks are shrunk back to the compact MTL architecture. The number of weights in the trained MTL neural network can be twice the number of weights in the MTL neural network used for reasoning, preferably at least three times.

[0048] For any shared layer of the deep MTL model, given a weight matrix W shared between t tasks, the weight matrix W is reconstructed into a matrix m×n (output size×kernel size, input size×kernel size), and then directly factorized using singular value factorization (SVD) so that W:=U·M·V, where M is a diagonal matrix, and the sizes of U, M, and V are c respectively. o ×k×r, r×r, r×k×c i , where c o is the output size, c i is the input size, k×k is the kernel size, and r is the rank. The matrix M can be reparameterized as the Hadamard product (element-wise product) of t matrices corresponding to the t tasks to be performed by the MTL model.

[0049] For each shared layer of an MTL model that aims to learn tasks a and b together, the parameters U and V are shared between the two tasks, and M a 、M b is assigned as a task-specific parameter for the corresponding task. a 、M b The ratio of shared parameters U and V is learned as a scaling factor in accordance with each individual task. The Hadamard product is cumulative, associative, and distributive, so the order of tasks does not affect the final product. The MTL model can use parameters U and V updated by the combined task loss L and L updated by the task-specific loss, respectively. a , L b During inference, the weights of the shared MTL model layers are shrunk back to W so that the size of the inference weight matrix does not increase. This is done by adding enough task-specific diagonal matrices M1, …, M t , the sharing mechanism can be naturally extended to multi-task learning with more than 2 tasks.

[0050] During the training phase of the over-parameterized MTL model, the factorization matrices U, M1, …, M t , V for training. The U and V matrices can be initialized using the same initialization method as the unfactored matrix W, and U, M1, …, M t is initialized to the identity matrix. The trained weight matrix is ​​collapsed back to W for inference, providing lower parameter count and computational cost, but with the benefit of expanding parameters during training.

[0051] To train shared parameters and task-specific parameters separately, an alternative training strategy consists of two training processes for each training round. First, a task j is selected from all t tasks. The task-specific parameters M j Use the task loss L j Training. Other parameters U, V, Mk ——where k≠j——and the parameters of the unfactored layers remain frozen. During the entire training process, all tasks t need to have equal opportunities to be trained, and for each epoch, only one task-specific weight matrix is ​​trained. Secondly, all task-specific factors M1,…,M t is frozen, and the other parameters U, V and the parameters of other unfactorized layers are trained by the joint loss of all tasks L, where L = ∑ i∈t α i L i α represents a fixed or adaptive loss weight. During each training process, Frobenius decay can be applied as a penalty to the factorized matrices U, V, and M to regularize the model for better generalization.

[0052] In the following description, it should be understood that the singular forms "a", "an" and "the" used in the following description are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be further understood that when used herein, the terms "includes", "including", "comprises" and / or "comprising" specify the presence of stated features, integers, steps, operations, elements, parts and / or units, but do not exclude the presence or addition of one or more other features, integers, steps, operations, studies, parts, units and / or groups thereof.

[0053] Certain aspects of the present disclosure include process steps and instructions described herein in algorithmic form. It should be noted that the process steps and instructions of the present disclosure can be implemented in software, firmware, or hardware, and when implemented in software, can be downloaded to reside on and operate from different platforms used by various operating systems. Unless otherwise clearly indicated from the following discussion, it should be understood that throughout the specification, discussions using terms such as "processing," "computing," "calculating," "determining," "displaying," "generating," etc. refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in a computer system memory or register or other such information storage, transmission, or display device.

[0054] The present disclosure, in some embodiments, also relates to a device for performing the operations in the text. The device can be specially constructed for the desired purpose, or it can include a general-purpose computer selectively enabled or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a non-temporary computer-readable storage medium, such as but not limited to any type of disk, including a floppy disk, a USB flash drive, an external hard drive, an optical disk, a CD-ROM, a disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card or an optical card, an application-specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, and each is coupled to a computer system bus. In addition, the computer mentioned in the specification may include a single processor or may be an architecture designed with multiple processors to increase computing power. Suitable processors include a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), and an ASIC.

[0055] The methods, apparatus and systems described herein are not essentially related to any particular computer or other device. Various general purpose systems may also be used in conjunction with programs according to the teachings herein, or may prove convenient for constructing more specialized devices to perform the required method steps. The structures required for various of these systems will emerge in the following description. In addition, the present invention is not described with reference to any particular programming language. It should be understood that a variety of programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0056] Figure 1 An exemplary MTL neural network 100 that can be used for embedded machine vision applications is shown. The network 100 includes a shared module 102 for all reasoning tasks and separate task-specific modules 104, 106, and 108, each of which is used for different tasks. The three task-specific modules 104, 106, and 108 correspond to three different reasoning tasks performed by the model. However, this is merely exemplary. In general, the network will have as many task-specific modules as the tasks performed by the MTL network. According to the principles described herein, the number of tasks performed by the MTL network is unlimited. According to various embodiments, the output of the shared module 102 is provided as input to each of the three task-specific modules 104, 106, and 108, and the task-specific modules 104, 106, and 108 are not connected to each other.

[0057] The shared module 102 includes a plurality of convolutional layers. The convolutional layer 110 of the shared module 102 is Figure 1 According to the principles described herein, at least one of the convolutional layers of the shared module 102 is over-factored for training and shrunk back to a compact set for inference, as discussed further below.

[0058] According to various embodiments, a deep multi-task learning training method utilizes over-parameterized convolutional layers to improve model expressiveness and learn task-specific variations. The method increases the parameter set size of the model during training, but the parameter set is shrunk back to a compact set for reasoning. The model can be further fine-tuned to improve the performance of each task without increasing any reasoning parameter size. The deep MTL method according to the principles described herein can be universally applied to any MTL model with convolutional layers to: 1) reduce the engineering effort of designing model-specific solutions; 2) significantly improve joint performance; 3) maintain or even reduce the size of the reasoning model.

[0059] For deep neural network models, the target convolutional layer to be over-factored is c i ×c i-1 ×k×k tensor parameterization, where c i-1 and c i are the corresponding input and output channel sizes (where the input and output sizes are h×w×c i-1 and h×w×c i ), k is the filter size. The tensor can be reshaped c i k×c i-1 k matrix W, which can be factorized into two c-sized i The weight matrix U is k×r and the size is r×c i-1 kV T , where W = UV T For overcomplete factorization, the rank r should not be less than min(c i k|c i-1 k) to avoid loss of information.

[0060] According to the principles described herein, instead of direct factorization of model parameters, three or more parameter matrices are initialized during training, which increases the size of the trained parameters. Then, the inference parameter set W is the product of the trained parameter matrices. For example, if the trained matrix is ​​of size c i The matrix U of size k×r, the matrix M of size r×r, and the matrix R of size r×c i-1 k's matrix V T , then the inference matrix will be of original size c i k×c i-1 k's W = UMV T .

[0061] The shared module 102 includes multiple parameters shared between tasks. Figure 1 In the three-task example, they are represented as tasks A, B, and C. The set of shared parameters (also called weights) is denoted as W ABCThe parameters of the task-specific modules 104, 106 and 108 are denoted as W A , W B and W C The specific convolutional layer of the shared parameters of the shared module 102 is represented as The shared parameters The number of shared layers is n. To simplify the description, Denoted as W - the shared parameter set for a given convolutional layer of the shared module 102. Assuming that the layer is a convolutional layer, the parameter set for a layer can be factorized as:

[0062] W=UMV T

[0063] The size of W is c i k×c i-1 k, the size of matrix U is c i k×r, the size of matrix M is r×r, and the matrix V T The size is r×c i-1 k, and r≥min(c i k, c i-1 k).

[0064] Instead of sharing all parameters W between tasks A, B, and C, which may cause negative transfer between tasks and hurt joint performance, we factorize matrices U and V T is configured to share parameters and uses M to capture task-specific information. According to various embodiments, M is the Hadamard product (element-wise product) of multiple matrices, the number of which is equal to the number of tasks of the same size (in Figure 1 In the example, three), for Figure 1 For example, the matrices are denoted as M1, M2, and M3. These three matrices represent the task-specific factors for each individual task A, B, and C, respectively. The basic principle behind the use of Hadamard products in this knowledge sharing scheme is that the Hadamard product of tensors corresponds to the product of multivariate functions. The matrix M will be:

[0065] M=M1⊙M2⊙M3

[0066] Where ⊙ denotes the Hadamard product. The Hadamard product of the task-specific matrices will produce a single matrix M regardless of the number of tasks.

[0067] Figure 2 An example of over-parameterization of a convolutional layer according to the process described above for a three-task MTL neural network is shown. The weight matrix of a given convolutional layer of the shared module 102 is factorized to train into weight matrices U, V T, M1, M2, and M3. The trained weight matrices are then multiplied together to provide the weight matrix W for the convolutional layer. The weight matrix W is used for inference of the model and can be deployed, for example, in an embedded machine vision system.

[0068] In fact, the shared module W ABC Any number of convolutional layers in a shared module can be factorized using the above principles to improve the performance of the multi-task model. In some variations, only a single convolutional layer in the multiple convolutional layers is factorized. In other variations, all convolutional layers are factorized. It is understood that any share of the convolutional layers of the shared module can be factorized. According to various embodiments, task-specific modules W are not factorized. A , W B and W C The convolutional layers in are factorized.

[0069] Before training, the original initialization of the weight parameter set W can be used to set the size of c i k×r and r×c i-1 k's matrices U and V T Initialize, and you can set U and V T Initialize to the identity matrix of size r×r.

[0070] During training, a task-specific loss L for task t is used i＝t Update the factor set M for a given task t i＝t , while all other factors (M i∈t , U and V T The factor of ) is frozen (i.e., the parameters are not updated). Then, the joint task loss is used Update matrices U and V T , while freezing all task factors M. According to various embodiments, in order to reduce the dynamic conflict between tasks during parameter update, for each round, there is only one task factor M i The training is performed with its task-specific loss. In other words, the three task factors are trained alternately in consecutive training rounds. Therefore, according to the above process, during training, the parameter size of the training model is expanded from W to the matrices U, V T , M1, M2 and M3 (for Figure 1 3-task example).

[0071] During inference, we multiply W by UMV T Restore the size of the convolutional layer to c i k×c i -1k inference weight matrix W, where M is recovered from M1, M2, and M3 by element-wise product M = M1⊙M2⊙M3. The parameter matrix W is then reshaped back to size c i×c i-1 ×k×k tensor to be used as the parameter set for a normal convolutional layer in the inference model. Therefore, according to the above principle, the parameter size of the inference model, such as a model that can be deployed on an embedded device, does not increase.

[0072] Figure 3 is a block diagram of a method 300 for constructing an MTL neural network configured to perform n tasks simultaneously according to the principles discussed above. For example, the method 300 can be used to construct Figure 1 The method 300 is a simplified representation of the training process for training an MTL neural network, focusing on the training aspects related to the overcomplete parameterization of the shared module. Other aspects of training an MTL neural network are well understood by those of ordinary skill in the art and are therefore omitted for brevity.

[0073] In general, a machine learning model that can use method 300 includes a shared module, which includes a plurality of convolutional layers, and a plurality of task-specific modules, each of which includes a plurality of convolutional layers. The parameter set of each of the x convolutional layers of the shared module is factorized into V T ,M1,M2,..,M n , where M i is the parameter matrix for task i among n tasks.

[0074] In step 302, the parameters of the model are initialized. U, V are initialized using the same initialization method as the unfactored matrix W. T Initialize the matrix and set M1, M2, .., M n Initialized to the identity matrix. Training begins at step 302, where the task-specific matrix M1 of the first convolutional layer of the shared module is updated based on training data, such as multiple training images. The update of the matrix M1 is done using the task-specific loss function L1. When updating the parameters of the matrix M1, all other parameters - parameters U, V T , M2, …, M n ——are all frozen (unchanged). In step 306, the matrices U, V are updated using a shared loss function that is a combination of task-specific loss functions. T The shared parameters of the shared module are updated, and the task-specific parameters are frozen. Steps 304 and 306 are repeated for each convolutional layer of the shared module. Note that for simplicity, the training process associated with other structures of the neural network is not shown. Once the convolutional layers of the shared module are updated, the method 300 proceeds to step 308, where one or more layers of the task-specific module are updated.

[0075] Steps 304-308 are repeated for each round. While the first round includes updating the parameters of the first task-specific matrix M1, the second round includes updating the parameters of the second task-specific matrix M2 at step 304. Thus, each round includes a different task-specific matrix M from the previous round. i Steps 304-308 are repeated accordingly for the required number of rounds. Once the last task-specific matrix M of the shared module n is updated, the next round will include updating the first task-specific matrix M1 (i is set to 1).

[0076] After the last round, method 300 proceeds to step 310, where the matrices U, V for each convolutional layer of the shared module are T , M2, …, M n Multiply as described above, thereby obtaining an inference weight matrix W, which is built into the MTL inference model 312 that can be implemented in an embedded system.

[0077] Method 300 is performed using a suitable training data set, such as suitable training images. The training images can be selected and preprocessed to be of the same height and width. The training images can be annotated by a human annotator to include labels for each task to be performed by MTL. For example, for an object detection task, the training image labels include the image file name, bounding box coordinates, and object name. As will be appreciated by those skilled in the art, different tasks may have different labels corresponding to the task.

[0078] The training datasets can be jointly labeled, non-jointly labeled, or some combination of both. This means that a single training dataset (e.g., a single training image) can have labels for a single task (non-jointly labeled) or multiple tasks (jointly labeled) to be trained together simultaneously. For jointly labeled or non-jointly labeled datasets, each training batch can include a mixture of datasets from different tasks. The datasets can be mixed for different tasks using any suitable distribution (e.g., uniform distribution).

[0079] To illustrate the above points, for a jointly labeled training dataset with a batch size of 256 training images for training a 4-task MTL neural network, all 256 training images have labels for all 3 tasks, and the MTL neural network is trained for all 3 tasks simultaneously. In contrast, for a non-jointly labeled training dataset with a batch size of 256 training images for training a 4-task MTL neural network, 64 training images can be labeled for each task (for uniform distribution). During training, the 256 training images are loaded together as a batch, and the MTL neural network is trained for all 3 tasks simultaneously. In this training batch, each task-specific part is updated only by the loss received from its associated 64 images, while the shared part is updated by the loss received from the 256 images together.

[0080] Therefore, due to the over-parameterization of the MTL neural network, according to the principles described therein, the MTL neural network can be trained with jointly or non-jointly labeled datasets. This allows the use of more diverse training datasets, making the training of the MTL neural network easier.

[0081] Figure 4 It is shown that an MTL neural network trained according to the principles discussed above can be implemented - for example Figure 3 The embedded system 400 of the MTL reasoning model 312 can be installed in the vehicle 402 to perform multiple machine vision reasoning tasks simultaneously on data generated using the sensor system 404. The sensor system 404 can include one or more on-board sensors, such as one or more forward sensors, one or more rearward sensors, and / or one or more in-vehicle sensors. The sensor can be any sensor suitable for sensing the vehicle environment, including a visible light camera, a lidar camera, a radar sensor, an ultrasonic sensor, or any other sensor technology.

[0082] The embedded system 400 includes an inference engine 406 that implements an MTL neural network for performing multiple (N) inference tasks on the imaging received from the sensor system 404. The MTL neural network can be configured for any suitable number and combination of inference tasks. Examples of inference tasks include semantic segmentation, depth estimation, object detection, and surface normal estimation.

[0083] The reasoning engine 406 outputs predictions associated with each reasoning task. These predictions can be provided to the vehicle control system 408, which can use the predictions for autonomous or semi-autonomous vehicle control. The vehicle control system 408 can be part of the embedded system 400, or can be a separate system that is communicatively connected to the control system 408. The vehicle control system 408 can include or be communicatively connected to one or more vehicle systems, such as a steering system, a braking system, an acceleration system, and a driver interface system (which can include, for example, an on-board display for communicating with the driver). In some embodiments, the vehicle control system 408 controls at least one of vehicle steering and vehicle braking, and / or provides an alert to the driver of the vehicle based on the prediction from the reasoning engine 406. For example, the vehicle control system 408 can control the braking of the vehicle 402 based on the distance to an obstacle on the road obtained based on, for example, semantic segmentation, object detection, and depth estimation output from the reasoning engine 406.

[0084] Figure 5 is a flow chart of a method 500 for performing multiple tasks on an image using an MTL neural network trained according to the principles discussed above. The method 500 may be, for example, deployed in a vehicle (e.g. Figure 4 An embedded computing system (e.g., Figure 4 In step 502, such as Figure 4 The camera system of the sensor system 404 receives an image. The image can be a single snapshot image or a frame of video. In step 504, an MTL neural network, such as Figure 3 The MTL neural network performs N inference tasks on the image simultaneously. Examples of inference tasks include semantic segmentation, depth estimation, object detection, instance segmentation, and surface normal estimation. Predictions from each task are output at step 506. These predictions can be used for autonomous vehicle control, as described above with respect to Figure 4 The vehicle control system 408 is discussed.

[0085] According to various embodiments, model fine-tuning and / or knowledge distillation can be used to further improve MTL model performance without increasing the size of the inference model, which can be used as post-processing. First, a single-task model of any size (or some best-performing multi-task model) can be used to generate soft targets for the multi-task model to learn from during training. Soft targets can be generated from unsupervised datasets, which are relatively abundant compared to more expensive supervised datasets. Therefore, when both labeled and unlabeled data are available, training can be extended to semi-supervised learning methods.

[0086] Second, for each specific task, the overcomplete factorized multi-task model can be further fine-tuned on its batch normalization layer for a limited number of epochs (usually less than 30 epochs are sufficient for fine-tuning). Since the number of affine parameters of the batch normalization layer is small, fine-tuning on each task in this post-processing is very fast.

[0087] Based on the principles discussed above, the MTL machine learning model includes a knowledge sharing architecture that is configured to improve the model performance of a deep multi-task learning model without increasing the inference cost. This can be used in any embedded vision application where: (1) multiple tasks that may compete for model shared resources are to be learned together, and (2) the inference budget of the deployed model is tight. Compared to the original deep multi-task learning model without the full factorization of the shared modules described herein, the method described herein can achieve significantly higher joint performance by reducing the model parameter size. This can potentially reduce the cost of embedded devices to be used in products without compromising the performance of the deployed deep multi-task learning model.

[0088] Although the above methods are often described for embedded machine vision applications, the same knowledge sharing approach for deep multi-task learning can be applied to any multi-task model with convolutional layers. This approach can potentially be useful for any application involving (1) deep learning models with convolutional layers, (2) multiple tasks that need to be learned simultaneously, and (3) requirements where the inference model parameter size is constrained.

[0089] Figure 6 An example of a computing system 600 according to some embodiments is shown. For example, the computing system 600 can be used, for example, according to Figure 3 The computing system 600 can be used to train the MTL machine learning model. Figure 4 One or more components of the embedded system 400. The system 600 may be a computer connected to a network. The system 600 may be a client or a server. Figure 6 As shown, system 600 can be any suitable type of processor-based system, such as a personal computer, a workstation, a server, a handheld computing device (portable electronic device) such as a phone or a tablet, or an embedded system or other dedicated device. System 600 can include, for example, one or more of an input device 620, an output device 630, one or more processors 610, a memory 640, and a communication device 660. Input device 620 and output device 630 can generally correspond to those described above and can be connected or integrated with computing system 600.

[0090] Input device 620 may be any suitable device that provides input, such as a touch screen, a keyboard or keypad, a mouse, a gesture recognition component of a virtual / augmented reality system, or a voice recognition device. Output device 630 may be or include any suitable device that provides output, such as a display, a touch screen, a haptic technology device, a virtual / augmented reality display, or a speaker.

[0091] Storage device 640 may be any suitable device that provides storage, such as electrical, magnetic, or optical storage including RAM, cache, hard drive, removable storage disk, or other non-transitory computer-readable medium. Communication device 660 may include any suitable device capable of sending and receiving signals over a network, such as a network interface chip or device. The components of computing system 600 may be connected in any suitable manner, such as via a physical bus connection or a wireless connection.

[0092] Processor 610 may be any suitable processor or combination of processors, including a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC). Software 650, which may be stored in storage device 640 and executed by one or more processors 610, may include, for example, programs that implement the functions or portions of the functions of the present disclosure (e.g., implemented in a device as described above). For example, software 650 may include one or more programs executed by one or more processors 610 to perform one or more steps of method 300 and / or method 500.

[0093] The software 650 may also be stored in and / or transmitted in any non-transitory computer-readable storage medium for use by or in conjunction with an instruction execution system, apparatus, or device, such as those described above, which may retrieve instructions associated with the software from the instruction execution system, apparatus, or device and execute those instructions. In the context of the present disclosure, a computer-readable storage medium may be any medium, such as storage device 640, which may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0094] The software 650 may also be transmitted in any transmission medium for use by or in conjunction with an instruction execution system, device, or apparatus such as those described above, which may obtain instructions associated with the software from the instruction execution system, device, or apparatus and execute the instructions. In the context of the present disclosure, a transmission medium may be any medium that may communicate, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or apparatus. Transmission computer-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, or infrared wired or wireless transmission media.

[0095] System 600 can be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communication protocol and can be protected by any suitable security protocol. The network can include any suitable arrangement of network links that can implement the transmission and reception of network signals, such as a wireless network connection, a T1 or T3 line, a cable network, DSL, or a telephone line.

[0096] System 600 may implement any operating system suitable for operating on a network. Software 650 may be written in any suitable programming language, such as C, C++, Java, or Python. In various embodiments, application software implementing the functionality of the present disclosure may be deployed in different configurations, such as in a client / server arrangement or through a web browser as, for example, a web-based application or web service.

[0097] For the purpose of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the specific forms disclosed. In light of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the principles of these techniques and their practical applications. This allows others skilled in the art to best utilize the techniques and various embodiments with various modifications suitable for the intended specific use.

[0098] Although the disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications will be understood to be included within the scope of the disclosure and examples defined by the claims. Finally, the entire disclosures of the patents and publications mentioned in this application are incorporated herein by reference.

Claims

1. A computer-implemented method for automatically controlling a vehicle, comprising: receiving data generated using at least one sensor of the vehicle; performing a plurality of different prediction tasks on the data simultaneously using a multi-task neural network, wherein the multi-task neural network includes at least one shared parameter inference matrix including parameters shared between the plurality of different prediction tasks, and the at least one shared parameter inference matrix is ​​over-parameterized during training into at least one shared parameter matrix and a plurality of task-specific parameter matrices, each of the plurality of task-specific parameter matrices being associated with a different task in the plurality of different prediction tasks; as well as automatically controlling the vehicle based on the results of the plurality of different prediction tasks, During training, the task-specific loss L for task t is used i＝t Update the factor set M for a given task t i＝t , while all other factors are frozen, then, the joint task loss Update matrices U and V T , and freeze all task factors M at the same time.

2. The method according to claim 1, wherein: The different prediction tasks include semantic segmentation, depth estimation, object detection, instance segmentation or surface normal estimation.

3. The method according to claim 1, wherein: The multi-task neural network includes a task-specific parameter inference matrix associated with the plurality of different prediction tasks.

4. The method according to claim 1, wherein: During training of the multi-task neural network, each of the plurality of task-specific parameter matrices is independently trained using a corresponding task-specific loss function.

5. The method according to claim 1, wherein: During training of the multi-task neural network, the at least one shared parameter matrix is ​​trained independently of the plurality of task-specific parameter matrices using a shared task loss function.

6. The method according to claim 1, wherein: Automatically controlling the vehicle based on the results of the plurality of different prediction tasks includes steering the vehicle, braking the vehicle, or providing an alert to a driver of the vehicle.

7. A computing system for automatically controlling a vehicle, comprising one or more processors and a memory storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for: receiving data generated using at least one sensor of the vehicle; performing a plurality of different prediction tasks on the data simultaneously using a multi-task neural network, wherein the multi-task neural network includes at least one shared parameter inference matrix including parameters shared between the plurality of different prediction tasks, and the at least one shared parameter inference matrix is ​​over-parameterized during training into at least one shared parameter matrix and a plurality of task-specific parameter matrices, each of the plurality of task-specific parameter matrices being associated with a different task in the plurality of different prediction tasks; and automatically controlling the vehicle based on the results of the plurality of different prediction tasks, in, During training, a task-specific loss L for task t is used i＝t Update the factor set M for a given task t i＝t , while all other factors are frozen, then, the joint task loss Update matrices U and V T , and freeze all task factors M at the same time.

8. The computing system of claim 7, wherein: The computing system is an embedded computing system of the vehicle.

9. The computing system of claim 7, wherein: The different prediction tasks include semantic segmentation, depth estimation, object detection, instance segmentation or surface normal estimation.

10. The computing system of claim 7, wherein: The multi-task neural network includes a task-specific parameter inference matrix associated with the plurality of different prediction tasks.

11. The computing system of claim 7, wherein: During training of the multi-task neural network, each of the plurality of task-specific parameter matrices is independently trained using a corresponding task-specific loss function.

12. The computing system of claim 7, wherein: During training of the multi-task neural network, the at least one shared parameter matrix is ​​trained independently of the plurality of task-specific parameter matrices using a shared task loss function.

13. A method for generating a multi-task machine learning neural network capable of performing a prediction task based on sensor data of a vehicle to automatically control the vehicle, the method comprising: A multi-task machine learning neural network is trained based on multiple training data sets, and the multi-task machine learning neural network includes: Multiple task-specific modules, each dedicated to a different task, and A sharing module, the sharing module comprising at least one over-parameterized convolutional layer, the at least one over-parameterized convolutional layer comprising at least one shared parameter matrix and a plurality of task-specific parameter matrices; generating a multi-task machine learning neural network for simultaneously performing a plurality of different prediction tasks based on sensor data of the vehicle, the multi-task machine learning neural network comprising a plurality of trained task-specific modules and a trained shared module, the matrix comprising a combination of the at least one trained shared parameter matrix and the plurality of trained task-specific parameter matrices; and embedding the multi-task machine learning neural network into a computing system of the vehicle so that the computing system can perform the plurality of different prediction tasks based on sensor data of the vehicle and automatically control the vehicle based on results of the plurality of different prediction tasks, During training, the task-specific loss L for task t is used i＝t Update the factor set M for a given task t i＝t , while all other factors are frozen, then, the joint task loss Update matrices U and V T , and freeze all task factors M at the same time.

14. The method according to claim 13, wherein: The different prediction tasks include semantic segmentation, depth estimation, object detection, instance segmentation or surface normal estimation.

15. The method according to claim 13, wherein: Training the multi-task machine learning neural network includes updating factors of one parameter matrix in the plurality of task-specific parameter matrices using a task-specific loss function while freezing factors of any other parameter matrices in the plurality of task-specific parameter matrices.

16. The method according to claim 15, wherein: Training the multi-task machine learning neural network includes updating factors of the at least one shared parameter matrix using a shared loss function while freezing parameters of the plurality of task-specific parameter matrices.

17. The method according to claim 13, wherein: The multi-task machine learning neural network includes multiple over-parameterized convolutional layers.

18. The method according to claim 13, wherein: The matrix multiplication of the trained at least one shared parameter matrix and the trained plurality of task-specific parameter matrices comprises an element-wise product of the trained plurality of task-specific parameter matrices.

19. The method according to claim 13, wherein: The output of the shared module is provided as an input to each of the plurality of task-specific modules, and each task-specific module is not interconnected with any other task-specific module.

20. The method according to claim 13, wherein: A matrix that is a combination of the trained at least one shared parameter matrix and the trained plurality of task-specific parameter matrices is generated by multiplying the at least one shared parameter matrix by a Hadamard product matrix of the trained plurality of task-specific parameter matrices.

21. A computing system for generating a multi-task machine learning neural network capable of performing prediction tasks based on sensor data of a vehicle to automatically control the vehicle, the computing system comprising one or more processors and a memory storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for: A multi-task machine learning neural network is trained based on multiple training data sets, wherein the multi-task machine learning neural network comprises: Multiple task-specific modules, each dedicated to a different task, and A sharing module, the sharing module comprising at least one over-parameterized convolutional layer, the at least one over-parameterized convolutional layer comprising at least one shared parameter matrix and a plurality of task-specific parameter matrices; generating a multi-task machine learning neural network for simultaneously performing a plurality of different prediction tasks based on sensor data of the vehicle, the multi-task machine learning neural network comprising a plurality of trained task-specific modules and a trained shared module, the matrix comprising a combination of the at least one trained shared parameter matrix and the plurality of trained task-specific parameter matrices; as well as storing the multi-task machine learning neural network for embedding in a computing system of the vehicle so that the computing system can perform the plurality of different prediction tasks based on sensor data of the vehicle and automatically control the vehicle based on results of the plurality of different prediction tasks, During training, the task-specific loss L for task t is used i＝t Update the factor set M for a given task t i＝t , while all other factors are frozen, then, the joint task loss Update matrices U and V T , and freeze all task factors M at the same time.

22. The computing system of claim 21, wherein: The different prediction tasks include semantic segmentation, depth estimation, object detection, instance segmentation or surface normal estimation.

23. The computing system of claim 21, wherein: Training the multi-task machine learning neural network includes updating factors of one parameter matrix in the plurality of task-specific parameter matrices using a task-specific loss function while freezing factors of any other parameter matrices in the plurality of task-specific parameter matrices.

24. The computing system of claim 23, wherein: Training the multi-task machine learning neural network includes updating factors of the at least one shared parameter matrix using a shared loss function while freezing parameters of the plurality of task-specific parameter matrices.

25. The computing system of claim 21, wherein: The multi-task machine learning neural network includes multiple over-parameterized convolutional layers.

26. The computing system of claim 21, wherein: The matrix multiplication of the trained at least one shared parameter matrix and the trained plurality of task-specific parameter matrices comprises an element-wise product of the trained plurality of task-specific parameter matrices.

27. The computing system of claim 21, wherein: The output of the shared module is provided as an input to each of the plurality of task-specific modules, and each task-specific module is not interconnected with any other task-specific module.

28. The computing system of claim 21, wherein: The matrix that is a combination of the trained at least one shared parameter matrix and the trained multiple task-specific parameter matrices is a matrix multiplication of the Hadamard product of the at least one shared parameter matrix and the trained multiple task-specific parameter matrices.

Images