Training method and target detection method and device of multi-task learning deep network

Abstract

The application provides a training method and a target detection method and device of a multi-task learning deep network, comprising: performing weighted calculation on a loss value corresponding to each task and a dynamic weight corresponding to the loss value to obtain a first total loss of a multi-task learning model; and training the multi-task learning model according to the first total loss, wherein the dynamic weight is continuously updated according to a gradient between the loss value corresponding to each task and a backbone shared network, a learning difficulty weight and an importance weight corresponding to each task. The application improves the training effect of each task by continuously updating the dynamic weight of each loss value.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a training method and object detection method and apparatus for multi-task learning deep networks. Background Technology

[0002] Multi-Task Learning (MTL) is a machine learning method that learns multiple related tasks together based on shared representations. Compared to conventional single-task models, MTL can handle multiple tasks using only a single model. The goal of MTL is to reduce the number of models, thereby improving model speed. Furthermore, it is hoped that during the training of a multi-task model, different tasks can mutually enhance each other, improving the performance of each task.

[0003] Most existing multi-task learning strategies focus on how to set loss weights for different tasks, such as using the homoscedasticity uncertainty of each task to determine the loss weights or dynamically setting the loss weights for each task. However, the above multi-task learning strategies are unstable and may lead to situations where the performance of one task improves while the performance of other tasks deteriorates significantly, ultimately resulting in a training effect that is worse than manual parameter tuning.

[0004] Therefore, there is an urgent need for a training method for multi-task learning deep networks that can improve the training effect of various tasks. Summary of the Invention

[0005] This invention provides a training method and object detection method and apparatus for multi-task learning deep networks to solve the above-mentioned problems.

[0006] This invention provides a training method for a multi-task learning deep network, comprising: acquiring training images for multi-task learning;

[0007] The training images are input into the backbone shared network of the multi-task learning model to extract the feature maps output by the backbone shared network.

[0008] The feature map is input into the network of each branch task of the multi-task learning model to obtain the prediction result for each task;

[0009] Calculate the loss value for each task based on the prediction result for each task and the label for the training image;

[0010] The first total loss of the multi-task learning model is obtained by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value; and the multi-task learning model is trained based on the first total loss.

[0011] The dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weights and importance weights corresponding to each task.

[0012] According to a training method for a multi-task learning deep network provided by the present invention, the dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, including:

[0013] S1, Initialize dynamic weights and obtain initial weight values;

[0014] S2, calculate the weighted loss based on the initial weight value and the loss value corresponding to each task, and calculate the gradient of the weighted loss with respect to the backbone shared network to obtain the weight gradient corresponding to each loss value;

[0015] S3. Based on the learning difficulty weight and importance weight corresponding to each task, calculate the error between the mean of the weight gradient corresponding to all loss values ​​and the weight gradient corresponding to each loss value, and obtain the weight gradient loss of each loss value.

[0016] S4, calculate the gradient of the weight gradient loss relative to the initial weight value, and update the initial weight value according to the calculated gradient value and the preset learning rate to obtain the updated weight value corresponding to each loss value, which is used as the dynamic weight.

[0017] S5, repeat S2-S4 to calculate and update the dynamic weights until the preset number of training iterations is reached.

[0018] According to a training method for a multi-task learning deep network provided by the present invention, step S3, calculating the error between the mean of the weight gradients corresponding to all loss values ​​and the weight gradients corresponding to each loss value based on the learning difficulty weights and importance weights corresponding to each task, to obtain the weight gradient loss of each loss value, includes:

[0019] Calculate the mean G_S_mean of the weight gradients G_S_i corresponding to all loss values:

[0020] G_S_mean=∑w_i*G_S_i / n

[0021] In the formula, w_i is the dynamic weight corresponding to the i-th loss value; G_S_i is the weight gradient corresponding to the i-th loss value; and n is the total number of tasks.

[0022] The weight gradient loss_G_S is calculated based on the mean G_S_mean and the weight gradient G_S_i:

[0023] loss_G_S=MSE(w_i*G_S_i-G_S_mean*V_i*D_i)

[0024] In the formula, MSE is the mean squared error, V_i is the importance weight of the i-th task, with a value range of [0,1], and D_i is the learning difficulty weight of the i-th task, with a value range of [0,1].

[0025] According to a training method for a multi-task learning deep network provided by the present invention, in step S4, the gradient of the weight gradient loss relative to the initial weight value is calculated, and the initial weight value is updated according to the calculated gradient value and a preset learning rate to obtain the updated weight value corresponding to each loss value as dynamic weights. The method further includes:

[0026] The first total loss of the multi-task learning model is obtained by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value.

[0027]

[0028] In the formula, w_i is the dynamic weight corresponding to the i-th loss value, loss_i is the loss value corresponding to the i-th task, and n is the total number of tasks;

[0029] Backpropagation is performed on each network layer in the multi-task learning model based on the first total loss to update the parameters corresponding to each network layer and obtain the updated network parameters. A new loss value is then calculated based on the updated network parameters.

[0030] Accordingly, S5, repeating S2-S4 to calculate and update the dynamic weights until a preset number of training iterations is reached, includes:

[0031] S5, based on the new loss value and the dynamic weights, repeat S2-S4 to update the dynamic weights until the preset number of training iterations is reached.

[0032] According to a training method for a multi-task learning deep network provided by the present invention, before the first total loss of the multi-task learning model is calculated by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and before training the multi-task learning model based on the first total loss, the method further includes:

[0033] Calculate the sum of the loss values ​​corresponding to all tasks to obtain the second total loss of the multi-task learning model;

[0034] A new multi-task learning model is obtained by training the multi-task learning model based on the second total loss.

[0035] Accordingly, the step of calculating the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and training the multi-task learning model based on the first total loss, includes:

[0036] Based on the new multi-task learning model, a new loss value corresponding to each task is calculated;

[0037] The first total loss of the multi-task learning model is obtained by weighting the new loss value corresponding to each task and the dynamic weight corresponding to the loss value, and the new multi-task learning model is trained based on the first total loss.

[0038] According to a training method for a multi-task learning deep network provided by the present invention, after calculating the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and training the multi-task learning model based on the first total loss, the method further includes:

[0039] Based on the multi-task learning model trained according to the first total loss, a new loss value corresponding to each task is calculated.

[0040] Calculate the sum of the new loss values ​​for all tasks to obtain the third total loss of the multi-task learning model;

[0041] The multi-task learning model trained based on the first total loss is trained based on the third total loss.

[0042] The present invention also provides a target detection method, comprising: acquiring an image to be detected in a target detection scene;

[0043] The image to be detected is input into a pre-trained multi-task learning model to obtain the detection results for each task.

[0044] The pre-trained multi-task learning model is trained based on the training method of the multi-task learning deep network described above.

[0045] The present invention also provides a training apparatus for a multi-task learning deep network, comprising: a training image acquisition module for acquiring training images for multi-task learning;

[0046] The feature extraction module is used to input the training image into the backbone shared network of the multi-task learning model and extract the feature map output by the backbone shared network.

[0047] The branch prediction module is used to input the feature map into the network of each branch task of the multi-task learning model to obtain the prediction result corresponding to each task.

[0048] The loss calculation module is used to calculate the loss value for each task based on the prediction result for each task and the label for the training image.

[0049] The model training module is used to perform weighted calculations based on the loss value corresponding to each task and the dynamic weights corresponding to the loss value to obtain the first total loss of the multi-task learning model; and to train the multi-task learning model based on the first total loss.

[0050] The dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weights and importance weights corresponding to each task.

[0051] The present invention also provides a target detection device, comprising: an image acquisition module for acquiring an image to be detected in a target detection scene;

[0052] The target detection module is used to input the image to be detected into a pre-trained multi-task learning model and obtain the detection results corresponding to each task.

[0053] The pre-trained multi-task learning model is trained using the training device for the aforementioned multi-task learning deep network.

[0054] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the above-described multi-task learning deep network training methods or the above-described object detection methods.

[0055] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described multi-task learning deep network training methods or the above-described object detection methods.

[0056] The training method and object detection method and apparatus for multi-task learning deep networks provided by this invention obtain the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value; and train the multi-task learning model based on the first total loss. The dynamic weight of the loss value is continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weight and importance weight corresponding to each task. This ensures that the multi-task learning model fully considers the importance and learning difficulty of different tasks during the training process, thereby ensuring that tasks with different importance and difficulty levels have good training effects. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating the training method for a multi-task learning deep network provided in this embodiment of the invention. Figure 1 ;

[0059] Figure 2 This is a schematic diagram of the network structure of the multi-task learning model provided in an embodiment of the present invention;

[0060] Figure 3 This is a flowchart illustrating a training method for a multi-task learning deep network provided in another embodiment of the present invention. Figure 2 ;

[0061] Figure 4 This is a schematic flowchart of the target detection method provided in an embodiment of the present invention;

[0062] Figure 5 This is a structural block diagram of the training device for a multi-task learning deep network provided in an embodiment of the present invention;

[0063] Figure 6 This is a structural block diagram of the target detection device provided in an embodiment of the present invention;

[0064] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0066] Figure 1 This is a flowchart illustrating the training method for a multi-task learning deep network provided in this embodiment of the invention. Figure 1 ;like Figure 1 As shown, a training method for a multi-task learning deep network includes the following steps:

[0067] S101, Obtain training images for multi-task learning.

[0068] In this embodiment, the training images for multi-task learning refer to the training images used for vehicle detection and lane line detection; that is, multi-task learning in this embodiment refers to vehicle detection and lane line detection. In other embodiments of the present invention, the training images for multi-task learning can be face data used for facial feature point localization, and the corresponding multi-task learning is the localization of each facial part; the training images for multi-task learning can also be training images that combine different perception tasks such as object detection, tracking, semantic segmentation, instance segmentation, clustering, and depth estimation, and the present invention does not limit this.

[0069] S102, the training image is input into the backbone shared network of the multi-task learning model, and the feature map output by the backbone shared network is extracted.

[0070] Based on the above definition of multi-task learning, it can be seen that a multi-task learning model has a backbone network sharing network parameters and branch networks that perform predictions for each task, such as... Figure 2 As shown.

[0071] Feature maps are extracted from training images through a backbone sharing network and then input into the respective branch task networks to achieve prediction.

[0072] It should be noted that the backbone sharing network is not limited in this embodiment; it can be a ResNet series (resnet34, resnet50, resent101, etc.) or a DarkNet series (DarkNet19, darknet53). Furthermore, the backbone sharing network can also be of an appropriate size depending on the application scenario, such as lightweight structures like resnet18, resnet34, and darknet19, medium-sized structures like resnet50 and darknet53, and heavy-duty structures like resent101 and resnet152.

[0073] S103, the feature map is input into the network of each branch task of the multi-task learning model to obtain the prediction result corresponding to each task.

[0074] In this step, the feature maps extracted by the backbone shared network are input into each branch task network. Each branch task network makes predictions based on the feature maps to obtain the corresponding prediction results. Taking the training images used for vehicle detection and lane detection as an example, the branch task networks include a vehicle detection task network and a lane detection task network. The two different branch task networks achieve prediction through corresponding network structures.

[0075] S104, calculate the loss value for each task based on the prediction result for each task and the label corresponding to the training image.

[0076] In this step, the loss value for each task is calculated by using the loss function corresponding to each branch task network to compare the prediction results with the labels (i.e., ground truth) corresponding to the training images.

[0077] S105, the first total loss of the multi-task learning model is obtained by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value; and the multi-task learning model is trained based on the first total loss.

[0078] The dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weights and importance weights corresponding to each task.

[0079] In this step, each training session assigns a different weight to the loss value of each task. This weight is continuously updated based on the gradient of the loss value of each task with respect to the last layer of the backbone shared network (i.e., the shared network layer closest to the branch task network), as well as the importance weight and learning difficulty weight of each task.

[0080] After obtaining the dynamic weights, all loss values ​​are weighted using these dynamic weights to calculate the total loss of the entire multi-task learning model, which serves as the first total loss. Based on this first total loss, the network parameters in the entire multi-task learning model are trained and updated using the backpropagation algorithm, ensuring that the multi-task learning model fully considers the importance and learning difficulty of different tasks during training.

[0081] This invention provides a training method for a multi-task learning deep network. The method calculates a first total loss for the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weights corresponding to those loss values. The multi-task learning model is then trained based on this first total loss. The dynamic weights of the loss values ​​are continuously updated based on the gradient between the loss value for each task and the backbone shared network, as well as the learning difficulty weights and importance weights for each task. This ensures that the multi-task learning model fully considers the importance and learning difficulty of different tasks during training, thereby guaranteeing good training results for tasks with varying importance and difficulty levels.

[0082] Furthermore, the dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, including:

[0083] S1, initialize dynamic weights and obtain initial weight values.

[0084] In this step, all dynamic weights w_i are initialized to 1.

[0085] S2, calculate the weighted loss based on the initial weight values ​​and the loss values ​​corresponding to each task, and calculate the gradient of the weighted loss with respect to the backbone shared network to obtain the weight gradient corresponding to each loss value.

[0086] In this step, the weighted loss w_i*loss_i of each task is first calculated based on the initial weight value and the loss value loss_i corresponding to each task (when the initial weight value is 1, the weighted loss is still loss_i).

[0087] Then, the gradient of the weighted loss w_i*loss_i with respect to the last shared network layer in the backbone shared network is calculated to obtain the weight gradient G_S_i for each task. Specifically, the gradient is calculated according to the chain rule during loss backpropagation, i.e., Where y represents the weighted loss of task i (i.e., w_i*loss_i), x represents the parameters of the layer before the final output of the branch task network corresponding to task i, and s represents the parameters of the last layer in the backbone shared network.

[0088] S3. Based on the learning difficulty weight and importance weight corresponding to each task, calculate the error between the mean of the weight gradients corresponding to all loss values ​​and the weight gradients corresponding to each loss value, and obtain the weight gradient loss of each loss value.

[0089] In this step, the average value G_S_mean of all weight gradients G_S_i is first calculated; then, the weight gradient loss loss_G_S for each task is calculated using the average value G_S_mean, the learning difficulty weight D_i, the importance weight V_i, the weight gradient G_S_i, and the dynamic weight w_i.

[0090] S4, calculate the gradient of the weight gradient loss relative to the initial weight value, and update the initial weight value according to the calculated gradient value and the preset learning rate to obtain the updated weight value corresponding to each loss value, which is used as the dynamic weight.

[0091] In this step, we first calculate the gradient of the weight gradient loss_G_S with respect to the dynamic weights w_i (the initial weight value is 1). Where l is the loss G_S; w_i is the dynamic weight. The initial weight values ​​are then updated based on the gradient G_w_i and the learning rate lr of the multi-task learning model, thus forming the dynamic weight.

[0092] S5, repeat S2-S4 to calculate and update the dynamic weights until the preset number of training iterations is reached.

[0093] In this step, the preset number of training iterations refers to the number of times the multi-task learning model is trained. For example, if the training set has 1000 samples, the batch size is set to 10, and the training epoch is set to 50, then one epoch requires 100 iterations to complete the training of all samples. 50 epochs would require a total of 5000 iterations, which is the preset number of training iterations.

[0094] Each training iteration generates different dynamic weights, which are then used to calculate the first total loss in the next iteration. After a preset number of training iterations, the weight gradients G_S_i of each branch task network in the multi-task learning model converge to "G_S_mean*V_i*D_i after adding the importance weights and difficulty weights of the tasks," thus achieving dynamic balance across all tasks and ensuring the training effectiveness of each task.

[0095] The present invention provides a training method for a multi-task learning deep network, which continuously updates dynamic weights based on the learning difficulty and importance of each task, as well as the gradient between the loss value of each task and the backbone shared network, thereby dynamically balancing the training effects of tasks with different levels of difficulty and importance.

[0096] Further, in step S3, the error between the mean of the weight gradients corresponding to all loss values ​​and the weight gradients corresponding to each loss value is calculated based on the learning difficulty weights and importance weights corresponding to each task, to obtain the weight gradient loss for each loss value, including:

[0097] Calculate the mean G_S_mean of the weight gradients G_S_i corresponding to all loss values:

[0098] G_S_mean=∑w_i*G_S_i / n

[0099] In the formula, w_i is the dynamic weight corresponding to the i-th loss value; G_S_i is the weight gradient corresponding to the i-th loss value; and n is the total number of tasks.

[0100] The weight gradient loss_G_S is calculated based on the mean G_S_mean and the weight gradient G_S_i:

[0101] loss_G_S=MSE(w_i*G_S_i-G_S_mean*V_i*D_i)

[0102] In the formula, MSE is the mean squared error, V_i is the importance weight of the i-th task, with a value range of [0,1], and D_i is the learning difficulty weight of the i-th task, with a value range of [0,1].

[0103] In this embodiment, the mean squared error is used to calculate the error between the weight gradient G_S_i of each task and the mean G_S_mean after adding the importance weight and difficulty weight of the task, and this error is used as the loss value to update the dynamic weight.

[0104] This invention provides a training method for a multi-task learning deep network. By adding the importance weights and difficulty weights of the tasks to the weight gradient loss_G_S, the training effects of different tasks can be balanced.

[0105] Further, in step S4, the gradient of the weight gradient loss relative to the initial weight value is calculated, and the initial weight value is updated according to the calculated gradient value and a preset learning rate to obtain the updated weight value corresponding to each loss value, which is then used as the dynamic weight. The method further includes:

[0106] The first total loss of the multi-task learning model is obtained by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value.

[0107]

[0108] In the formula, w_i is the dynamic weight corresponding to the i-th loss value, loss_i is the loss value corresponding to the i-th task, and n is the total number of tasks.

[0109] Backpropagation is performed on each network layer in the multi-task learning model based on the first total loss to update the parameters corresponding to each network layer and obtain the updated network parameters. A new loss value is then calculated based on the updated network parameters.

[0110] Accordingly, S5, repeating S2-S4 to calculate and update the dynamic weights until a preset number of training iterations is reached, includes:

[0111] S5, based on the new loss value and the dynamic weights, repeat S2-S4 to update the dynamic weights until the preset number of training iterations is reached.

[0112] Specifically, after updating the dynamic weights, the first total loss of the entire multi-task learning model is calculated based on the loss value and its corresponding dynamic weight weights. Then, backpropagation is performed on the multi-task learning model using this first total loss to update the parameters of each network layer, resulting in a new multi-task learning model. Feature extraction and prediction are performed using this new model to generate a new loss value, which is then used to update the dynamic weights. This process continues until a preset number of training iterations is reached, at which point the entire dynamic weight update process ends, and the model training process also concludes. After the dynamic weights stop updating, the final multi-task learning model achieves a dynamic balance between tasks of varying difficulty and importance.

[0113] Further, before calculating the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and training the multi-task learning model based on the first total loss, the method further includes:

[0114] Calculate the sum of the loss values ​​for all tasks to obtain the second total loss of the multi-task learning model.

[0115] The new multi-task learning model is obtained by training the multi-task learning model based on the second total loss.

[0116] Accordingly, the step of calculating the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and training the multi-task learning model based on the first total loss, includes:

[0117] Based on the new multi-task learning model, a new loss value is calculated for each task.

[0118] The first total loss of the multi-task learning model is obtained by weighting the new loss value corresponding to each task and the dynamic weight corresponding to the loss value, and the new multi-task learning model is trained based on the first total loss.

[0119] In this embodiment, before dynamically balancing the weight gradients G_S_i of each task, the multi-task learning model needs to be "warmed up" first, so that each task reaches a convergence state before dynamic balancing. This avoids the problem of model training bias caused by directly adding the learning difficulty weights and importance weights of the tasks for training due to the different convergence states of different tasks in the early stage.

[0120] Specifically, in the warm-up phase, the dynamic weights are directly set to 1, and the total loss of all tasks is calculated by weighting. The total loss of this warm-up phase is the second total loss. The model is continuously trained based on the second total loss to obtain the multi-task learning model output by the warm-up phase.

[0121] Furthermore, based on the multi-task learning model output in the warm-up phase, the first total loss is calculated using the aforementioned dynamic weights. The parameters in the multi-task learning model are then continuously updated based on the first total loss, thereby obtaining the multi-task learning model output in the dynamic balancing phase (i.e., the training phase in which the first total loss is calculated using dynamic weights).

[0122] The present invention provides a training method for a multi-task learning deep network. Before calculating the first total loss using dynamic weights and training the multi-task learning model based on the first total loss, the multi-task learning model is first trained based on the sum of the loss values ​​corresponding to all tasks (i.e., the second total loss). This allows each task to converge to a similar extent before training based on the dynamic weights. This avoids the situation where the model is trained incorrectly due to the different initial convergence states of each task.

[0123] Further, after calculating the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and training the multi-task learning model based on the first total loss, the method further includes:

[0124] Based on the multi-task learning model trained according to the first total loss, a new loss value corresponding to each task is calculated.

[0125] Calculate the sum of the new loss values ​​for all tasks to obtain the third total loss of the multi-task learning model.

[0126] The multi-task learning model trained based on the first total loss is trained based on the third total loss.

[0127] In this embodiment, since the multi-task learning model trained according to the first total loss mainly focuses on tasks with high learning difficulty weights and high importance weights, while tasks with low importance weights and low learning difficulty weights are suppressed to a certain extent during the training process, after obtaining the multi-task learning model trained according to the first total loss, it is necessary to restore the multi-task learning model. This restoration process is the multi-task restoration stage.

[0128] Specifically, in the multi-task recovery phase, all dynamic weights are set to 1, and new loss values ​​are calculated using the multi-task learning model output from the dynamic balancing phase. Then, each new loss value is weighted and calculated to obtain the third total loss. The multi-task learning model output from the dynamic balancing phase is continuously trained and updated based on the third total loss, thus finally obtaining the trained multi-task learning model.

[0129] The present invention provides a training method for a multi-task learning deep network. After calculating the first total loss using dynamic weights and training the multi-task learning model based on the first total loss, the multi-task learning model is further subjected to recovery training. This counteracts the suppression of tasks with low importance and difficulty weights during the training process after the addition of importance and difficulty weights.

[0130] Figure 3 This is a flowchart illustrating a training method for a multi-task learning deep network provided in another embodiment of the present invention. Figure 2 ;like Figure 3 As shown, in another embodiment of the present invention, a training method for a multi-task learning deep network includes the following steps:

[0131] S301, the entire training process of the model is divided into three stages based on the training epoch: the warm-up stage, the dynamic balancing stage, and the multi-task recovery stage. For example, the warm-up stage accounts for 1 / 4 of the total number of training epochs, the dynamic balancing stage accounts for 1 / 2 of the total number of training epochs, and the multi-task recovery stage accounts for 1 / 4 of the total number of training epochs.

[0132] S302, during the warm-up phase (i.e., e < 0.25 * E, where e is the current training epoch and E is the total training epoch), the dynamic weights corresponding to each task are set to 1. Then, the training images are sequentially input into the initial multi-task learning model according to the batch size, thereby calculating the sum of the loss values ​​of each task, i.e., the second total loss. Based on the second total loss, backpropagation is performed on each layer in the initial multi-task learning model to update the parameters of each layer, thereby obtaining the updated multi-task learning model. This updated multi-task learning model is the multi-task learning model output during the warm-up phase.

[0133] S303. In the dynamic balance stage (i.e., 0.25*E ≤ e < 0.75*E), first input the training images into the multi-task learning model output in the warm-up stage in sequence according to the batch size batch-size, calculate the loss values of each task; then update to obtain the dynamic weights, use the dynamic weights and the loss values of each task to calculate the first total loss through weighted calculation, and perform backpropagation on the multi-task learning model output in the warm-up stage according to the first total loss, so as to update the network parameters therein. When the next batch of training images is input into the model, re-determine the dynamic weights to complete a new round of model parameter update. The multi-task learning model obtained after completing the predetermined number of training times in the dynamic balance stage is the multi-task learning model output in the dynamic balance stage.

[0134] Among them, the update process of the dynamic weights is as follows: first initialize all dynamic weights w_i to 1; then calculate the weighted loss of the i-th task (i.e., w_i*loss_i) with respect to the gradient of the S layer (i.e., the last layer of the shared network layer) as the weight gradient G_S_i; calculate the average value of the gradients of the S layer of all tasks G_S_mean = ∑w_i*G_S_i / n (n is the total number of tasks); then calculate the loss function of G_S, loss_G_S = MSE(w_i*G_S_i - G_S_mean*V_i*D_i), where MSE is the mean square error, V_i is the importance weight of the i-th task (the larger V_i is, the more important the task is, and the value is 0 - 1), and D_i is the learning difficulty weight of the i-th task (the larger D_i is, the more difficult the task is to learn, and the value is 0 - 1); then calculate the gradient of loss_G_S with respect to w_i, denoted as G_w_i; then, according to G_w_i and the learning rate lr, update the dynamic weight w_i; finally, normalize the dynamic weight w_i (i.e., add a softmax operation to the dynamic weight w_i), so as to obtain the final dynamic weight for calculating the first total loss.

[0135] S304. In the multi-task recovery stage (i.e., 0.75*E ≤ e < E), the training process of the model is similar to the warm-up stage, but it is trained and updated based on the multi-task learning model output in the dynamic balance stage. Specifically, set the dynamic weights corresponding to each task to 1, and then input the training images into the multi-task learning model output in the dynamic balance stage in sequence according to the batch size batch-size, so as to calculate the sum of the loss values of each task, that is, the third total loss. Perform backpropagation on each layer in the multi-task learning model output in the dynamic balance stage according to the third total loss, and update the parameters of each layer, so as to obtain the final multi-task learning model for deployment and implementation in various terminals.

[0136] This invention provides a training method for a multi-task learning deep network. First, the multi-task learning model is trained and updated based on the sum of the loss values ​​of each task, allowing each task to converge. Then, the model is trained using the total loss calculated based on the loss values ​​of each task and their corresponding dynamic weights. This avoids the possibility of the model being skewed by directly incorporating the difficulty and importance of each task. Furthermore, a multi-task recovery stage is added to retrain and update the multi-task learning model based on the sum of the loss values ​​of each task. This improves the training effect of tasks with lower difficulty and importance weights, resulting in a multi-task learning model with relatively accurate detection performance across all tasks.

[0137] Figure 4 This is a flowchart illustrating the target detection method provided in an embodiment of the present invention; as shown below. Figure 4 As shown, a target detection method includes the following steps:

[0138] S401, acquire the image to be detected in the target detection scene.

[0139] The training images used in the aforementioned multi-task learning refer to training images for vehicle detection and lane detection. In this embodiment, the image to be detected in the target detection scenario is an image used for vehicle detection and lane detection. In other embodiments of the present invention, the image to be detected is adjusted based on the training images.

[0140] S402, input the image to be detected into a pre-trained multi-task learning model to obtain the detection result corresponding to each task.

[0141] The pre-trained multi-task learning model is trained based on the training method of the multi-task learning deep network described above.

[0142] In this step, a pre-trained multi-task learning model is used to detect the image to be detected, thereby obtaining the results of vehicle detection and lane line detection.

[0143] The pre-trained multi-task learning model is trained using the training method for the multi-task learning deep network described above, which will not be elaborated further here.

[0144] The target detection method provided in this embodiment of the invention performs target detection by using a multi-task learning model trained using the training method of the multi-task learning deep network described above, thereby ensuring that the detection results of each task have a high accuracy rate.

[0145] The training apparatus for multi-task learning deep networks provided by the present invention will be described below. The training apparatus for multi-task learning deep networks described below can be referred to in correspondence with the training method for multi-task learning deep networks described above.

[0146] Figure 5 This is a structural block diagram of the training device for a multi-task learning deep network provided in an embodiment of the present invention, as shown below. Figure 5 As shown, a training device for a multi-task learning deep network includes:

[0147] The training image acquisition module 501 is used to acquire training images for multi-task learning.

[0148] In this embodiment, the training images for multi-task learning refer to the training images used for vehicle detection and lane line detection; that is, multi-task learning in this embodiment refers to vehicle detection and lane line detection. In other embodiments of the present invention, the training images for multi-task learning can be face data used for facial feature point localization, and the corresponding multi-task learning is the localization of each facial part; the training images for multi-task learning can also be training images that combine different perception tasks such as object detection, tracking, semantic segmentation, instance segmentation, clustering, and depth estimation, and the present invention does not limit this.

[0149] The feature extraction module 502 is used to input the training image into the backbone shared network of the multi-task learning model and extract the feature map output by the backbone shared network.

[0150] Based on the above definition of multi-task learning, it can be seen that a multi-task learning model has a backbone network sharing network parameters and branch networks that perform predictions for each task, such as... Figure 2 As shown.

[0151] Feature maps are extracted from training images through a backbone sharing network and then input into the respective branch task networks to achieve prediction.

[0152] It should be noted that the backbone sharing network is not limited in this embodiment; it can be a ResNet series (resnet34, resnet50, resent101, etc.) or a DarkNet series (DarkNet19, darknet53). Furthermore, the backbone sharing network can also be of an appropriate size depending on the application scenario, such as lightweight structures like resnet18, resnet34, and darknet19, medium-sized structures like resnet50 and darknet53, and heavy-duty structures like resent101 and resnet152.

[0153] The branch prediction module 503 is used to input the feature map into the network of each branch task of the multi-task learning model to obtain the prediction result corresponding to each task.

[0154] In this module, the feature maps extracted by the backbone shared network are input into each branch task network. Each branch task network makes predictions based on the feature maps to obtain the corresponding prediction results. Taking the training images used for vehicle detection and lane detection as an example, the branch task networks include a vehicle detection task network and a lane detection task network. The two different branch task networks achieve prediction through corresponding network structures.

[0155] The loss calculation module 504 is used to calculate the loss value for each task based on the prediction result for each task and the label for the training image.

[0156] In this module, the loss value for each task is calculated by using the loss function corresponding to each branch task network to compare the prediction result with the label (i.e., the ground truth) corresponding to the training image.

[0157] The model training module 505 is used to perform weighted calculations based on the loss value corresponding to each task and the dynamic weight corresponding to the loss value to obtain the first total loss of the multi-task learning model; and to train the multi-task learning model based on the first total loss.

[0158] The dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weights and importance weights corresponding to each task.

[0159] In this module, each task's loss value has a different weight during each training session. This weight is continuously updated based on the gradient of each task's loss value with respect to the last layer of the backbone shared network (i.e., the shared network layer closest to the branch task network), as well as the importance weight and learning difficulty weight of each task.

[0160] After obtaining the dynamic weights, all loss values ​​are weighted using these dynamic weights to calculate the total loss of the entire multi-task learning model, which serves as the first total loss. Based on this first total loss, the network parameters in the entire multi-task learning model are trained and updated using the backpropagation algorithm, ensuring that the multi-task learning model fully considers the importance and learning difficulty of different tasks during training.

[0161] The training device for a multi-task learning deep network provided in this embodiment of the invention obtains the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value; and trains the multi-task learning model based on the first total loss. The dynamic weight of the loss value is continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weight and importance weight corresponding to each task. This ensures that the multi-task learning model fully considers the importance and learning difficulty of different tasks during the training process, thereby ensuring that tasks with different importance and difficulty levels have good training effects.

[0162] The target detection device provided by the present invention is described below. The target detection device described below and the target detection method described above can be referred to in correspondence.

[0163] Figure 6 This is a structural block diagram of the target detection device provided in an embodiment of the present invention, such as... Figure 6 As shown, a target detection device includes:

[0164] The image acquisition module 601 is used to acquire the image to be detected in the target detection scene.

[0165] The training images used in the aforementioned multi-task learning refer to training images for vehicle detection and lane detection. In this embodiment, the image to be detected in the target detection scenario is an image used for vehicle detection and lane detection. In other embodiments of the present invention, the image to be detected is adjusted based on the training images.

[0166] The target detection module 602 is used to input the image to be detected into a pre-trained multi-task learning model and obtain the detection result corresponding to each task.

[0167] The pre-trained multi-task learning model is trained using the training device for the aforementioned multi-task learning deep network.

[0168] In this module, a pre-trained multi-task learning model is used to detect the image to be detected, thereby obtaining the results of vehicle detection and lane line detection.

[0169] The pre-trained multi-task learning model is trained using the training method for the multi-task learning deep network described above, which will not be elaborated further here.

[0170] The target detection device provided in this embodiment of the invention performs target detection by using a multi-task learning model trained using the training method of the multi-task learning deep network described above, thereby ensuring that the detection results of each task have a high accuracy rate.

[0171] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 730 to execute a training method for a multi-task learning deep network. The training method includes: acquiring training images for multi-task learning; inputting the training images into the backbone shared network of the multi-task learning model to extract feature maps output by the backbone shared network; inputting the feature maps into the branch task networks of the multi-task learning model to obtain prediction results for each task; calculating the loss value for each task based on the prediction results and the labels corresponding to the training images; performing a weighted calculation based on the loss values ​​for each task and the dynamic weights corresponding to the loss values ​​to obtain a first total loss for the multi-task learning model; and training the multi-task learning model based on the first total loss; wherein the dynamic weights are continuously updated based on the gradient between the loss value for each task and the backbone shared network, as well as the learning difficulty weights and importance weights for each task.

[0172] Alternatively, it can be used to perform an object detection method, which includes: acquiring an image to be detected in an object detection scenario; inputting the image to be detected into a pre-trained multi-task learning model to obtain the detection result corresponding to each task; wherein the pre-trained multi-task learning model is trained based on the training method of the multi-task learning deep network described above.

[0173] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a training method for a multi-task learning deep network provided by the above-described method. The training method for the multi-task learning deep network includes: acquiring training images for multi-task learning; inputting the training images into the backbone shared network of the multi-task learning model to extract feature maps output by the backbone shared network; inputting the feature maps into the branch task networks of the multi-task learning model to obtain prediction results corresponding to each task; calculating a loss value corresponding to each task based on the prediction results corresponding to each task and the label corresponding to the training image; performing a weighted calculation based on the loss value corresponding to each task and the dynamic weights corresponding to the loss value to obtain a first total loss of the multi-task learning model; and training the multi-task learning model based on the first total loss; wherein the dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weights and importance weights corresponding to each task.

[0175] Alternatively, it can be used to perform an object detection method, which includes: acquiring an image to be detected in an object detection scenario; inputting the image to be detected into a pre-trained multi-task learning model to obtain the detection result corresponding to each task; wherein the pre-trained multi-task learning model is trained based on the training method of the multi-task learning deep network described above.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a multi-task learning deep network, characterized in that, include: Acquire training images for multi-task learning; The training images are input into the backbone shared network of the multi-task learning model to extract the feature maps output by the backbone shared network. The feature map is input into the network of each branch task of the multi-task learning model to obtain the prediction result for each task; Calculate the loss value for each task based on the prediction result for each task and the label for the training image; The first total loss of the multi-task learning model is obtained by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value; and the multi-task learning model is trained based on the first total loss. The dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weights and importance weights corresponding to each task. The dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, including: S1, Initialize dynamic weights and obtain initial weight values; S2, calculate the weighted loss based on the initial weight value and the loss value corresponding to each task, and calculate the gradient of the weighted loss with respect to the backbone shared network to obtain the weight gradient corresponding to each loss value; S3. Based on the learning difficulty weight and importance weight corresponding to each task, calculate the error between the mean of the weight gradient corresponding to all loss values ​​and the weight gradient corresponding to each loss value, and obtain the weight gradient loss of each loss value. S4, calculate the gradient of the weight gradient loss relative to the initial weight value, and update the initial weight value according to the calculated gradient value and the preset learning rate to obtain the updated weight value corresponding to each loss value, as the dynamic weight. S5, repeat S2-S4 to calculate and update the dynamic weights until the preset number of training iterations is reached.

2. The training method for multi-task learning deep networks according to claim 1, characterized in that, S3 calculates the error between the mean of the weight gradients corresponding to all loss values ​​and the weight gradients corresponding to each loss value based on the learning difficulty weights and importance weights corresponding to each task, thereby obtaining the weight gradient loss for each loss value, including: Calculate the mean G_S_mean of the weight gradients G_S_i corresponding to all loss values: G_S_mean=∑w_i*G_S_i / n In the formula, w_i is the dynamic weight corresponding to the i-th loss value; G_S_i is the weight gradient corresponding to the i-th loss value; and n is the total number of tasks. The weight gradient loss_G_S is calculated based on the mean G_S_mean and the weight gradient G_S_i: loss_G_S=MSE(w_i*G_S_i - G_S_mean*V_i*D_i) In the formula, MSE is the mean squared error, V_i is the importance weight of the i-th task, with a value range of [0,1], and D_i is the learning difficulty weight of the i-th task, with a value range of [0,1].

3. The training method for multi-task learning deep networks according to claim 1, characterized in that, In step S4, the gradient of the weight gradient loss relative to the initial weight value is calculated, and the initial weight value is updated according to the calculated gradient value and a preset learning rate to obtain the updated weight value corresponding to each loss value, which is then used as the dynamic weight. The method further includes: The first total loss of the multi-task learning model is obtained by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value. loss = In the formula, w_i is the dynamic weight corresponding to the i-th loss value, loss_i is the loss value corresponding to the i-th task, and n is the total number of tasks; Backpropagation is performed on each network layer in the multi-task learning model based on the first total loss to update the parameters corresponding to each network layer and obtain the updated network parameters. A new loss value is then calculated based on the updated network parameters. Accordingly, S5, repeating S2-S4 to calculate and update the dynamic weights until a preset number of training iterations is reached, includes: S5, based on the new loss value and the dynamic weights, repeat S2-S4 to update the dynamic weights until the preset number of training iterations is reached.

4. The training method for a multi-task learning deep network according to claim 1, characterized in that, Before calculating the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and training the multi-task learning model based on the first total loss, the method further includes: Calculate the sum of the loss values ​​corresponding to all tasks to obtain the second total loss of the multi-task learning model; A new multi-task learning model is obtained by training the multi-task learning model based on the second total loss. Accordingly, the step of calculating the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and training the multi-task learning model based on the first total loss, includes: Based on the new multi-task learning model, a new loss value corresponding to each task is calculated; The first total loss of the multi-task learning model is obtained by weighting the new loss value corresponding to each task and the dynamic weight corresponding to the loss value, and the new multi-task learning model is trained based on the first total loss.

5. The training method for a multi-task learning deep network according to claim 1 or 4, characterized in that, After calculating the first total loss of the multi-task learning model by weighting the loss value corresponding to each task and the dynamic weight corresponding to the loss value, and training the multi-task learning model based on the first total loss, the method further includes: Based on the multi-task learning model trained according to the first total loss, a new loss value corresponding to each task is calculated. Calculate the sum of the new loss values ​​for all tasks to obtain the third total loss of the multi-task learning model; The multi-task learning model trained based on the first total loss is trained based on the third total loss.

6. A target detection method, characterized in that, include: Acquire the image to be detected in the target detection scene; The image to be detected is input into a pre-trained multi-task learning model to obtain the detection results for each task. The pre-trained multi-task learning model is trained based on the training method of the multi-task learning deep network as described in any one of claims 1 to 5.

7. A training device for a multi-task learning deep network, characterized in that, include: The training image acquisition module is used to acquire training images for multi-task learning. The feature extraction module is used to input the training image into the backbone shared network of the multi-task learning model and extract the feature map output by the backbone shared network. The branch prediction module is used to input the feature map into the network of each branch task of the multi-task learning model to obtain the prediction result corresponding to each task. The loss calculation module is used to calculate the loss value for each task based on the prediction result for each task and the label for the training image. The model training module is used to perform weighted calculations based on the loss value corresponding to each task and the dynamic weights corresponding to the loss value to obtain the first total loss of the multi-task learning model; and to train the multi-task learning model based on the first total loss. The dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, as well as the learning difficulty weights and importance weights corresponding to each task. The dynamic weights are continuously updated based on the gradient between the loss value corresponding to each task and the backbone shared network, including: S1, Initialize dynamic weights and obtain initial weight values; S2, calculate the weighted loss based on the initial weight value and the loss value corresponding to each task, and calculate the gradient of the weighted loss with respect to the backbone shared network to obtain the weight gradient corresponding to each loss value; S3. Based on the learning difficulty weight and importance weight corresponding to each task, calculate the error between the mean of the weight gradient corresponding to all loss values ​​and the weight gradient corresponding to each loss value, and obtain the weight gradient loss of each loss value. S4, calculate the gradient of the weight gradient loss relative to the initial weight value, and update the initial weight value according to the calculated gradient value and the preset learning rate to obtain the updated weight value corresponding to each loss value, as the dynamic weight. S5, repeat S2-S4 to calculate and update the dynamic weights until the preset number of training iterations is reached.

8. A target detection device, characterized in that, include: The image acquisition module is used to acquire the image to be detected in the target detection scene; The target detection module is used to input the image to be detected into a pre-trained multi-task learning model and obtain the detection results corresponding to each task. The pre-trained multi-task learning model is trained using the training device for the multi-task learning deep network as described in claim 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method for the multi-task learning deep network as described in any one of claims 1 to 5 or the object detection method as described in claim 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the multi-task learning deep network as described in any one of claims 1 to 5 or the object detection method as described in claim 6.

Images