A loss self-balancing method, system, device and medium for a multi-task network

Abstract

The application relates to a loss self-balancing method, system, device and medium of a multitask network. The method comprises the following steps: inputting a pre-acquired image dataset into an image multitask network for training; when the training round number does not reach a specified round number, applying an improved homoscedastic uncertainty weighted loss to a subtask loss in each task to obtain a first total loss function for performing a first stage training of task internal loss self-balancing of the image multitask network; when the training round number reaches the specified round number, applying an improved speed balance loss to the first total loss function to obtain a second total loss function for performing a second stage training of loss self-balancing between different tasks of the image multitask network; and inputting an image to be predicted into the trained image multitask network for multitask prediction to output a multitask prediction result. The method can automatically balance the loss functions of different tasks and improve the performance of the multitask network.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a loss self-balancing method, system, device, and medium for multi-task networks. Background Technology

[0002] Multi-task learning refers to using a single deep neural network to simultaneously perform multiple deep learning-related tasks, such as classification, detection, and segmentation. The advantages of multi-task networks include improved generalization ability, leveraging the correlation and complementarity between different tasks to enhance the performance of each task, saving computational resources, and accelerating inference speed. However, multi-task learning also faces challenges, one of which is balancing the loss functions across different tasks. Different tasks may have different magnitudes, speeds, and directions of loss functions. Directly summing them might lead to some tasks' losses dominating while neglecting others, thus affecting the network's learning performance. Therefore, it is necessary to reasonably weight the loss functions of different tasks so that each task receives appropriate attention and optimization.

[0003] Currently, several methods have been proposed for loss weighting in multi-task networks, which can be mainly divided into two categories: static weighting and dynamic weighting. Static weighting involves assigning a fixed weight to each task during training. This weight can be manually set or based on prior knowledge. The advantage of static weighting is its simplicity and ease of implementation, but its disadvantage is that it cannot adapt to changes in different tasks, potentially leading to overfitting or underfitting. Dynamic weighting, on the other hand, involves dynamically adjusting the weights of each task during training based on changes in the loss functions of different tasks, ensuring a balance in the loss functions of each task. The advantage of dynamic weighting is its adaptability to changes in different tasks, but its disadvantage is the need for complex mechanisms to calculate and update the weights of each task, which may increase computational overhead and instability. Summary of the Invention

[0004] Therefore, it is necessary to provide a loss self-balancing method, system, device, and medium for multi-task networks that can handle multiple tasks simultaneously and automatically balance the loss functions of different tasks without increasing additional parameters and computational overhead, thereby achieving a more stable and efficient training process.

[0005] A loss self-balancing method for multi-task networks, the method comprising:

[0006] A deep learning-based image multi-task network is constructed, and a pre-acquired image dataset is input into the image multi-task network for training. The image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor. Each task branch processor handles one task, which includes object detection, semantic segmentation, and instance segmentation.

[0007] When the training epochs of the image multi-task network have not reached the specified number of epochs, the improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task. Then, by combining the homoscedastic uncertainty weighted loss functions of all tasks using the improved homoscedastic uncertainty weighted loss, a first total loss function is obtained. Based on the first total loss function, the image multi-task network is trained in the first stage of self-balancing of intra-task losses. The value of the improved homoscedastic uncertainty weighted loss is a non-negative number.

[0008] When the training epochs of the image multi-task network reach a specified number of epochs, an improved velocity balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task, resulting in a second total loss function. The image multi-task network is then trained in a second stage based on the second total loss function to achieve loss self-balancing between different tasks, until the image multi-task network converges or reaches the preset number of training iterations, resulting in a trained image multi-task network. The improved velocity balancing loss introduces gradient loss.

[0009] The image to be predicted is input into a trained image multi-task network for multi-task prediction, and the multi-task prediction result is output.

[0010] In one embodiment, the data loading and preprocessor is used to perform data augmentation on the input image dataset and load a batch of image data and image labels. The loaded image data and image labels are then concatenated and input into a shared feature extractor. The shared feature extractor is used to extract general features from the input image data and image labels and input the general features into each task branch processor. The task branch processor is used to extract relevant features for the corresponding task from the general features and output the prediction result of the task based on the relevant features.

[0011] In one embodiment, when the number of training epochs of the image multi-task network has not reached a specified number of epochs, an improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task, including:

[0012] If the training epochs of the image multi-task network have not reached the specified number of epochs, the optimizer first adds and updates the weights of the improved homoscedasticity uncertainty-weighted loss during training. The initial weights of the improved homoscedasticity uncertainty-weighted loss are set according to the number of tasks, and the range of the initial weights is [insert range here]. Between these numbers, N represents the number of tasks;

[0013] Then, based on the weights of the improved homoscedasticity uncertainty weighted loss, the improved homoscedasticity uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedasticity uncertainty weighted loss function for each task. Among them, the sub-task loss within each task includes: object detection classification loss, object detection regression loss, and keypoint detection regression loss within the object detection task; semantic segmentation main loss, semantic segmentation first-class auxiliary loss, and semantic segmentation second-class auxiliary loss within the semantic segmentation task; and instance segmentation classification loss and instance segmentation regression loss within the instance segmentation task.

[0014] In one embodiment, the improved homoscedastic uncertainty-weighted loss is expressed as:

[0015]

[0016] Where L1(W) represents the sum of the loss functions of the first task, L2(W) represents the sum of the loss functions of the second task, σ1 represents the learnable loss parameter of the first task, and σ2 represents the learnable loss parameter of the second task.

[0017] In one embodiment, the homoscedastic uncertainty weighted loss function for each task is expressed as follows:

[0018] Loss obj =Loss uwl (Loss obj-cls Loss obj-l1 Loss kpt-l1 );

[0019] Loss sem-seg =Loss uwl (Loss sem-seg-cls Loss sem-seg-aux1 Loss sem-seg-aux2 );

[0020] Loss ins-seg =Loss uwl (Loss ins-seg-cls Loss ins-seg-l1 );

[0021] Among them, Loss objLoss is the homoscedastic uncertainty-weighted loss function for the object detection task. sem-seg Loss is a weighted loss function for homoscedastic uncertainty in semantic segmentation tasks. ins-seg Loss is a weighted loss function for homoscedastic uncertainty in instance segmentation tasks. uwl To improve the homoscedastic uncertainty-weighted loss, Loss obj-cls Loss is the loss for object detection classification. obj-l1 For object detection regression loss, Loss kpt-l1 For keypoint detection regression loss, Loss sem-seg-cls For semantic segmentation, the main loss is Loss. sem-seg-aux1 For semantic segmentation, the first type of auxiliary loss, Loss sem-seg-aux2 Loss is the second type of auxiliary loss for semantic segmentation. ins-seg-cls Loss is the loss for instance segmentation classification. ins-seg-l1 The instance segmentation regression loss is used.

[0022] In one embodiment, a first total loss function is obtained by combining the homoscedastic uncertainty weighted loss functions of all tasks using an improved homoscedastic uncertainty weighted loss, denoted as:

[0023] Loss total-first-stage =Loss uwl (Loss obj Loss sem-seg Loss ins-seg ).

[0024] In one embodiment, the improved velocity balance loss is expressed as

[0025] Loss dwa =sum i (λ i (t)L i (t))+βLossgrad(t;w i (t));

[0026] Where, λ i (t) represents the velocity weight corresponding to task i, and L i (t) represents the loss function currently corresponding to task i, w i (t) represents the learning speed of task i, and Lossgrad(t; w) represents the learning speed of task i. i (t) represents the gradient loss of task i, β is a hyperparameter used to control the weight of the gradient loss, and the optimal value of β is obtained by solving the gradient descent method; the parameter t represents the sum of the subtask losses of task i.

[0027] In one embodiment, when the training epochs of the image multi-task network reach a specified number of epochs, an improved speed-balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task, resulting in a second total loss function, denoted as:

[0028]

[0029] Among them, Loss obj Loss is the homoscedastic uncertainty-weighted loss function for the object detection task. sem-seg Loss is a weighted loss function for homoscedastic uncertainty in semantic segmentation tasks. ins-seg Loss is a weighted loss function for homoscedastic uncertainty in instance segmentation tasks. uwl To improve the weighted loss for homoscedastic uncertainty, λ k1 , λ k2 , λ k3 These are the speed weights for object detection, semantic segmentation, and instance segmentation tasks, respectively.

[0030] In one embodiment, the image to be predicted is input into a trained image multi-task network for multi-task prediction, and the output is a multi-task prediction result, including:

[0031] The image to be predicted is input into the trained image multi-task network to perform object detection, semantic segmentation, and instance segmentation prediction tasks respectively, and the multi-task prediction results are obtained. The multi-task prediction results include the predicted category, detection box coordinates, and key point detection coordinates for object detection, the boundary coordinates for semantic segmentation, and the boundary coordinates and detection box coordinates for instance segmentation.

[0032] A loss self-balancing system for a multi-task network, the system comprising:

[0033] A multi-task network construction module is used to construct a deep learning-based image multi-task network and input a pre-acquired image dataset into the image multi-task network for training; wherein, the image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor, each task branch processor corresponding to a task, the tasks including object detection task, semantic segmentation task, and instance segmentation task;

[0034] The first-stage training module is used to apply an improved homoscedasticity uncertainty weighted loss to the sub-task loss within each task when the training epochs of the image multi-task network have not reached a specified number of epochs, to obtain the homoscedasticity uncertainty weighted loss function for each task, and to combine the homoscedasticity uncertainty weighted loss functions of all tasks using the improved homoscedasticity uncertainty weighted loss to obtain a first total loss function, and to perform a first-stage training of the image multi-task network for task-internal loss self-balancing based on the first total loss function; wherein, the value of the improved homoscedasticity uncertainty weighted loss is a non-negative number;

[0035] The second-stage training module is used to, when the number of training epochs of the image multi-task network reaches a specified number of epochs, apply an improved velocity balancing loss to the first total loss function based on the current learning speed and gradient magnitude of each task to obtain a second total loss function, and then perform a second-stage training of the image multi-task network to self-balance the losses between different tasks based on the second total loss function, until the image multi-task network converges or reaches a preset number of training iterations, thus obtaining a trained image multi-task network; wherein, the improved velocity balancing loss introduces gradient loss;

[0036] The testing module is used to input the image to be predicted into the trained image multi-task network for multi-task prediction and output the multi-task prediction result.

[0037] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0038] A deep learning-based image multi-task network is constructed, and a pre-acquired image dataset is input into the image multi-task network for training. The image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor. Each task branch processor handles one task, which includes object detection, semantic segmentation, and instance segmentation.

[0039] When the training epochs of the image multi-task network have not reached the specified number of epochs, the improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task. Then, by combining the homoscedastic uncertainty weighted loss functions of all tasks using the improved homoscedastic uncertainty weighted loss, a first total loss function is obtained. Based on the first total loss function, the image multi-task network is trained in the first stage of self-balancing of intra-task losses. The value of the improved homoscedastic uncertainty weighted loss is a non-negative number.

[0040] When the training epochs of the image multi-task network reach a specified number of epochs, an improved velocity balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task, resulting in a second total loss function. The image multi-task network is then trained in a second stage based on the second total loss function to achieve loss self-balancing between different tasks, until the image multi-task network converges or reaches the preset number of training iterations, resulting in a trained image multi-task network. The improved velocity balancing loss introduces gradient loss.

[0041] The image to be predicted is input into a trained image multi-task network for multi-task prediction, and the multi-task prediction result is output.

[0042] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0043] A deep learning-based image multi-task network is constructed, and a pre-acquired image dataset is input into the image multi-task network for training. The image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor. Each task branch processor handles one task, which includes object detection, semantic segmentation, and instance segmentation.

[0044] When the training epochs of the image multi-task network have not reached the specified number of epochs, the improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task. Then, by combining the homoscedastic uncertainty weighted loss functions of all tasks using the improved homoscedastic uncertainty weighted loss, a first total loss function is obtained. Based on the first total loss function, the image multi-task network is trained in the first stage of self-balancing of intra-task losses. The value of the improved homoscedastic uncertainty weighted loss is a non-negative number.

[0045] When the training epochs of the image multi-task network reach a specified number of epochs, an improved velocity balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task, resulting in a second total loss function. The image multi-task network is then trained in a second stage based on the second total loss function to achieve loss self-balancing between different tasks, until the image multi-task network converges or reaches the preset number of training iterations, resulting in a trained image multi-task network. The improved velocity balancing loss introduces gradient loss.

[0046] The image to be predicted is input into a trained image multi-task network for multi-task prediction, and the multi-task prediction result is output.

[0047] Compared with existing technologies, the above-mentioned loss self-balancing method for multi-task networks has the following technical advantages:

[0048] 1. Traditional image processing methods can only complete a single task and are difficult to handle two tasks simultaneously: long-distance coarse localization and close-range fine segmentation of the interface. However, this application can use a deep learning-based image multi-task network to perform prediction of multiple visual tasks at the same time, such as instance segmentation, semantic segmentation, object detection, and key point detection. This improves the efficiency and flexibility of the image multi-task network and can meet the real-time requirements of the deployment end.

[0049] 2. When the number of training epochs of the image multi-task network has not reached the specified number of epochs, this application applies the improved homoscedastic uncertainty weighted loss to the sub-task loss within each task. The improved homoscedastic uncertainty weighted loss is a method that uses Bayesian theory to model the uncertainty of a task. It considers the confidence and uncertainty of different tasks, and can achieve self-balancing of the loss within the task during network training. Furthermore, the value of the improved homoscedastic uncertainty weighted loss is non-negative, thereby avoiding the problems of slow convergence and negative loss function in the early stage of training.

[0050] 3. When the training epochs of the image multi-task network reach a specified number, this application applies an improved velocity balancing loss. Velocity balancing loss is a method to dynamically adjust the loss weights of different tasks. It calculates the priority of tasks based on the rate of change of loss for each task, so that tasks with slower training speeds receive greater weights, thereby accelerating their convergence. Furthermore, the improved velocity balancing loss introduces gradient loss. Using gradient loss can balance the gradient magnitude and learning speed of different tasks, thereby avoiding, to some extent, the dimensional differences between different tasks and the problem of the strong getting stronger.

[0051] 4. This application applies the improved velocity balancing loss to the first total loss function to obtain the second total loss function. The second total loss function simultaneously performs adaptive balancing of the loss functions between different tasks from two aspects: loss change rate and homoscedastic uncertainty. This ensures that the loss weight of each task takes into account both the priority of the task and the confidence and uncertainty of the task, thereby improving the performance and generalization ability of the multi-task network. It can reduce the large amount of weight coefficient exploration work in the multi-task network, automatically adjust and balance the loss weight values ​​of each task, and enable end-to-end self-balancing training on different datasets. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a loss self-balancing method for a multi-task network in one embodiment.

[0053] Figure 2 This is a schematic diagram of the structure of an image multitasking network in one embodiment;

[0054] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] In one embodiment, such as Figure 1 As shown, a loss self-balancing method for multi-task networks is provided, which includes the following steps:

[0057] Step S1: Construct a deep learning-based image multi-task network and input the pre-acquired image dataset into the image multi-task network for training; wherein, the image multi-task network includes a data loading and preprocessor, a shared feature extractor and a task branch processor, each task branch processor corresponding to a task, including object detection task, semantic segmentation task and instance segmentation task.

[0058] It is understandable that in an image multi-task network, data acquisition for multiple tasks uses the same data loading and preprocessor for loading and preprocessing. The acquired image data is then input into a shared feature extractor to extract common features. Finally, each task branch processor extracts task-related features from the output of the shared feature extractor and outputs the prediction results for the task, such as masks for instance segmentation and bounding boxes for object detection.

[0059] Step S2: When the number of training rounds of the image multi-task network has not reached the specified number of rounds, the improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task. The improved homoscedastic uncertainty weighted loss is then combined with the homoscedastic uncertainty weighted loss functions of all tasks to obtain the first total loss function. The image multi-task network is then trained in the first stage of task-internal loss self-balancing based on the first total loss function. The value of the improved homoscedastic uncertainty weighted loss is non-negative.

[0060] Homoscedastic uncertainty weighted loss (UWL) is a method that uses Bayesian theory to model the uncertainty of a task. It calculates the confidence level of a task based on its homoscedastic uncertainty, giving tasks with higher uncertainty lower weights to reduce their impact. Furthermore, the improved homoscedastic uncertainty weighted loss designed in this application uses a non-negative value, thus avoiding the problems of slow convergence and negative loss function in the early stages of training.

[0061] Step S3: When the number of training rounds of the image multi-task network reaches a specified number of rounds, the improved velocity balancing loss is applied to the first total loss function according to the current learning speed and gradient magnitude of each task to obtain the second total loss function. The image multi-task network is then trained in the second stage according to the second total loss function to balance the loss between different tasks until the image multi-task network converges or reaches the preset number of training iterations, thus obtaining the trained image multi-task network. The improved velocity balancing loss introduces gradient loss.

[0062] As can be understood, Distributed Loss Allocation (DWA) is a method that dynamically adjusts the weights of losses for different tasks. It calculates task priority based on the rate of change of loss for each task, calculates the loss ratio of adjacent training iterations, and uses softmax normalization to give greater weights to tasks with slower training speeds, thereby accelerating their convergence. Its drawback is that it does not consider the magnitude difference in losses between different tasks when calculating sample weights, lacks a relatively unified standard, and sample weights are easily affected by the magnitude of the loss. Initial weights need to be assigned to the losses of different tasks beforehand; otherwise, the results will be poor, and true end-to-end training cannot be achieved in this case. Therefore, this application introduces gradient loss into DWA, using the gradient loss method to balance the gradient magnitude and learning speed of different tasks, thereby avoiding to some extent the dimensional differences between different tasks and the problem of the strong getting stronger. The goal is to initially solve this magnitude difference problem by minimizing a gradient loss, allowing the loss function to consider both the gradient magnitude and the learning speed of different tasks.

[0063] Furthermore, considering that the initial values ​​of the losses for each task differ significantly in the early stages of training, and that the rates of decrease vary considerably, an improved UWL is applied in the early stages of training to accelerate model convergence. After a period of training, an improved DWA is introduced when the loss changes have stabilized somewhat.

[0064] Step S4: Input the image to be predicted into the trained image multi-task network for multi-task prediction, and output the multi-task prediction result.

[0065] In one embodiment, a data loading and preprocessor is used to augment the input image dataset and load a batch of image data and image labels. The loaded image data and image labels are then concatenated and input into a shared feature extractor. The shared feature extractor can be any convolutional neural network, such as ResNet (Residual Network) or VGG, used to extract general features from the input image data and image labels, and input these general features into each task branch processor. The task branch processor can also be any neural network structure used to extract relevant features for the corresponding task from the general features and output the prediction result for the task based on these features. Furthermore, an attention mechanism can be introduced into the shared feature extractor to determine the importance of each feature for a specific task, and the shared feature extractor itself learns a global feature pool for all tasks.

[0066] Specifically, such as Figure 2 The image multi-task network shown includes a data loading and preprocessor, a backbone network acting as a shared feature extractor, and three task branch processors corresponding to three tasks. Each task branch processor consists of a neck network and a head network. The neck network is used to implement feature fusion for different tasks, including an object detection neck, a semantic segmentation neck, and an instance segmentation neck. The neck network can specifically use structures such as FPN (Feature Pyramid Network) and BiFPN (Weighted Bidirectional Feature Pyramid Network). The head network is used to extract task-related features and output task prediction results. The head network includes an object detection head, a keypoint detection head, a semantic segmentation head, and an instance segmentation head, with a coupling head connecting the object detection head and the keypoint detection head. Specifically, the output dimension of the object detection head is num_classes+5, where num_classes is the main class and 5 is the bounding box and confidence score; the output dimension of the keypoint detection head is 12, representing whether the keypoint is visible and the four coordinate points detected; the output dimension of the semantic segmentation head is num_seg plus the image size, where num_seg is the number of semantic segmentation classes plus the background class; the output dimension of the instance segmentation head is num_ins plus the image size, where num_ins is the number of instance segmentation classes plus the background class.

[0067] In one embodiment, when the number of training epochs of the image multi-task network has not reached a specified number of epochs, an improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task, including:

[0068] If the training epochs of the image multi-task network have not reached the specified number of epochs, the optimizer first adds and updates the weights of the improved homoscedasticity uncertainty-weighted loss during training. The initial weights of the improved homoscedasticity uncertainty-weighted loss are set according to the number of tasks, and the range of the initial weights is [insert range here]. Between, N is the number of tasks. Where, the improved homoscedasticity uncertainty-weighted loss is expressed as...

[0069]

[0070] Where L1(W) represents the sum of the loss functions for the first task, L2(W) represents the sum of the loss functions for the second task, σ1 represents the learnable loss parameters for the first task, and σ2 represents the learnable loss parameters for the second task. The larger σ1 and σ2 are, the greater the uncertainty, and the more the loss will be penalized.

[0071] It is understandable that, compared to the original formula for weighted loss based on homoscedasticity uncertainty, This application modifies the final summation term logσ1σ2 into a regularization term. This is to avoid situations where the loss function value is negative, and the initial weights of the improved homoscedasticity uncertainty-weighted loss are defined in... This can make the network training more stable in the early stages, and by applying the improved homoscedastic uncertainty weighted loss within the task, the network can achieve self-balancing in the first stage of overall training, which can effectively help the network model converge.

[0072] Then, based on the weights of the improved homoscedastic uncertainty weighted loss, it is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task. The sub-task loss within each task can be any loss function suitable for that task, such as the object detection classification loss, object detection regression loss, and keypoint detection regression loss within an object detection task; the semantic segmentation main loss, semantic segmentation first-class auxiliary loss, and semantic segmentation second-class auxiliary loss within a semantic segmentation task; and the instance segmentation classification loss and instance segmentation regression loss within an instance segmentation task. The homoscedastic uncertainty weighted loss function for each task is expressed as follows:

[0073] Loss obj =Loss uwl (Loss obj-cls Loss obj-l1 Loss kpt-l1 );

[0074] Loss sem-seg =Loss uwl (Losssem-seg-cls Loss sem-seg-aux1 Loss sem-seg-aux2 );

[0075] Loss ins-seg =Loss uwl (Loss ins-seg-cls Loss ins-seg-l1 );

[0076] Among them, Loss obj Loss is the homoscedastic uncertainty-weighted loss function for the object detection task. sem-seg Loss is a weighted loss function for homoscedastic uncertainty in semantic segmentation tasks. ins-seg Loss is a weighted loss function for homoscedastic uncertainty in instance segmentation tasks. uwl To improve the homoscedastic uncertainty-weighted loss, Loss obj-cls Loss is the loss for object detection classification. obj-l1 For object detection regression loss, Loss kpt-l1 For keypoint detection regression loss, Loss sem-seg-cls For semantic segmentation, the main loss is Loss. sem-seg-aux1 For semantic segmentation, the first type of auxiliary loss, Loss sem-seg-aux2 Loss is the second type of auxiliary loss for semantic segmentation. ins-seg-cls Loss is the loss for instance segmentation classification. ins-seg-l1 The instance segmentation regression loss is used.

[0077] In one embodiment, a first total loss function is obtained by combining the homoscedastic uncertainty weighted loss functions of all tasks using an improved homoscedastic uncertainty weighted loss, denoted as:

[0078] Loss total-first-stage =Loss uwl (Loss obj Loss sem-seg Loss ins-seg ).

[0079] It is understandable that by combining the weighted loss functions of homoscedasticity uncertainty for all tasks, based on obtaining the homoscedasticity uncertainty weighted loss functions for each task, true end-to-end training can be achieved.

[0080] In one embodiment, the improved velocity balance loss is expressed as

[0081] Loss dwa =sum i (λ i (t)L i(t))+βLossgrad(t;w i (t));

[0082] Where, parameter t represents the sum of subtask losses for task i; in the improved speed-balanced loss formula, speed weights and learning speeds are both relative to the sum of subtask losses for each task; λ i (t) represents the velocity weight corresponding to task i, and L i (t) represents the loss function corresponding to task i, w i (t) represents the learning rate of task i, and its value is the ratio of the loss function of the previous training round to the loss function of the round before that; Lossgrad(t; w) i (t) represents the gradient loss for task i, which is the difference between the gradient of each task and the target value. This target value is determined based on the average gradient of all tasks and the training speed. We want tasks with faster training speeds to have smaller gradients and tasks with slower training speeds to have larger gradients, so that different tasks have equal importance. A larger gradient loss indicates that the network model pays less attention to this task. We want to minimize the gradient loss so that different tasks have equal importance during training. β is a hyperparameter used to control the weight of the gradient loss, and the optimal value of β is obtained by solving the gradient descent method. Gradient descent is a method for updating the β hyperparameter based on gradient information. It can calculate the gradient of the β hyperparameter based on its impact on model performance, and then update the β hyperparameter in the opposite direction of the gradient. This method can directly optimize the objective function of the β hyperparameter, avoiding the blindness of grid search or random search and improving optimization accuracy. This can minimize the gradient loss while optimizing the DWA loss, making the gradient norm of different tasks close to a common target value, thereby avoiding the difference in scale between different tasks and the problem of the strong getting stronger.

[0083] It is understandable that, compared to directly applying the DWA loss, this application uses the gradient loss of different tasks as a regularization term of the DWA loss, thereby maintaining the advantages of the DWA loss while increasing the constraint on the gradient magnitude.

[0084] In one embodiment, when the training epochs of the image multi-task network reach a specified number of epochs, an improved speed-balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task, resulting in a second total loss function, denoted as:

[0085]

[0086] Among them, Loss obj Loss is the homoscedastic uncertainty-weighted loss function for the object detection task. sem-segLoss is a weighted loss function for homoscedastic uncertainty in semantic segmentation tasks. ins-seg Loss is a weighted loss function for homoscedastic uncertainty in instance segmentation tasks. uwl To improve the weighted loss for homoscedastic uncertainty, λ k1 , λ k2 , λ k3 These are the speed weights for object detection, semantic segmentation, and instance segmentation tasks, respectively.

[0087] It is understandable that the role of the second total loss function is to adaptively balance the loss functions among different tasks. This application effectively combines two different loss self-balancing techniques—improved homoscedasticity uncertainty-weighted loss and improved velocity-balanced loss—to improve the model's performance and generalization ability. Using a neural network backpropagation algorithm, the parameters of the image multi-task network are updated according to the total loss function, and the noise parameters for each task are also updated, ensuring that the noise parameters reflect changes in task uncertainty. This process is repeated until the network model converges or reaches the preset number of training iterations, resulting in a trained image multi-task network.

[0088] Furthermore, in the hyperparameter part of the network, the learning rate setting needs to consider both self-balancing losses. Experiments show that UWL requires a smaller initial loss value of 0.001, while DWA requires a larger loss value of 0.01. Since this application uses UWL loss within each task, a larger initial learning rate can be used, so 0.01 is used. Cosine annealing learning rate is used for network training, and the optimizer uses SGD (gradient descent) stochastic gradient descent and applies gradient clipping and other strategies to make the network train more stably.

[0089] In one embodiment, the image to be predicted is input into a trained image multi-task network for multi-task prediction, and the output is a multi-task prediction result, including:

[0090] The image to be predicted is input into the trained image multi-task network to perform object detection, semantic segmentation, and instance segmentation prediction tasks respectively, and the multi-task prediction results are obtained. The multi-task prediction results include the predicted category, detection box coordinates, and key point detection coordinates for object detection, the boundary coordinates for semantic segmentation, and the boundary coordinates and detection box coordinates for instance segmentation.

[0091] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0092] In one embodiment, a loss self-balancing system for a multi-task network is provided, comprising:

[0093] A multi-task network construction module is used to construct a deep learning-based image multi-task network and input a pre-acquired image dataset into the image multi-task network for training; wherein, the image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor, each task branch processor corresponding to a task, the tasks including object detection task, semantic segmentation task, and instance segmentation task;

[0094] The first-stage training module is used to apply an improved homoscedasticity uncertainty weighted loss to the sub-task loss within each task when the training epochs of the image multi-task network have not reached a specified number of epochs, to obtain the homoscedasticity uncertainty weighted loss function for each task, and to combine the homoscedasticity uncertainty weighted loss functions of all tasks using the improved homoscedasticity uncertainty weighted loss to obtain a first total loss function, and to perform a first-stage training of the image multi-task network for task-internal loss self-balancing based on the first total loss function; wherein, the value of the improved homoscedasticity uncertainty weighted loss is a non-negative number;

[0095] The second-stage training module is used to, when the number of training epochs of the image multi-task network reaches a specified number of epochs, apply an improved velocity balancing loss to the first total loss function based on the current learning speed and gradient magnitude of each task to obtain a second total loss function, and then perform a second-stage training of the image multi-task network to self-balance the losses between different tasks based on the second total loss function, until the image multi-task network converges or reaches a preset number of training iterations, thus obtaining a trained image multi-task network; wherein, the improved velocity balancing loss introduces gradient loss;

[0096] The testing module is used to input the image to be predicted into the trained image multi-task network for multi-task prediction and output the multi-task prediction result.

[0097] Specific limitations regarding the loss self-balancing system for multi-task networks can be found in the limitations of the loss self-balancing method for multi-task networks described above, and will not be repeated here. Each module in the aforementioned loss self-balancing system for multi-task networks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0098] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a loss self-balancing method for a multi-tasking network. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0099] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0100] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0101] A deep learning-based image multi-task network is constructed, and a pre-acquired image dataset is input into the image multi-task network for training. The image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor. Each task branch processor handles one task, which includes object detection, semantic segmentation, and instance segmentation.

[0102] When the training epochs of the image multi-task network have not reached the specified number of epochs, the improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task. Then, by combining the homoscedastic uncertainty weighted loss functions of all tasks using the improved homoscedastic uncertainty weighted loss, a first total loss function is obtained. Based on the first total loss function, the image multi-task network is trained in the first stage of self-balancing of intra-task losses. The value of the improved homoscedastic uncertainty weighted loss is a non-negative number.

[0103] When the training epochs of the image multi-task network reach a specified number of epochs, an improved velocity balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task, resulting in a second total loss function. The image multi-task network is then trained in a second stage based on the second total loss function to achieve loss self-balancing between different tasks, until the image multi-task network converges or reaches the preset number of training iterations, resulting in a trained image multi-task network. The improved velocity balancing loss introduces gradient loss.

[0104] The image to be predicted is input into a trained image multi-task network for multi-task prediction, and the multi-task prediction result is output.

[0105] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0106] A deep learning-based image multi-task network is constructed, and a pre-acquired image dataset is input into the image multi-task network for training. The image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor. Each task branch processor handles one task, which includes object detection, semantic segmentation, and instance segmentation.

[0107] When the training epochs of the image multi-task network have not reached the specified number of epochs, the improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task. Then, by combining the homoscedastic uncertainty weighted loss functions of all tasks using the improved homoscedastic uncertainty weighted loss, a first total loss function is obtained. Based on the first total loss function, the image multi-task network is trained in the first stage of self-balancing of intra-task losses. The value of the improved homoscedastic uncertainty weighted loss is a non-negative number.

[0108] When the training epochs of the image multi-task network reach a specified number of epochs, an improved velocity balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task, resulting in a second total loss function. The image multi-task network is then trained in a second stage based on the second total loss function to achieve loss self-balancing between different tasks, until the image multi-task network converges or reaches the preset number of training iterations, resulting in a trained image multi-task network. The improved velocity balancing loss introduces gradient loss.

[0109] The image to be predicted is input into a trained image multi-task network for multi-task prediction, and the multi-task prediction result is output.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A loss self-balancing method for multi-task networks, characterized in that, The method includes: A deep learning-based image multi-task network is constructed, and a pre-acquired image dataset is input into the image multi-task network for training; wherein, the image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor, each task branch processor corresponding to a task, the tasks including object detection task, semantic segmentation task, and instance segmentation task; When the number of training epochs of the image multi-task network has not reached the specified number of epochs, an improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task. The improved homoscedastic uncertainty weighted loss is then combined with the homoscedastic uncertainty weighted loss functions of all tasks to obtain a first total loss function. Based on this first total loss function, the image multi-task network undergoes a first-stage training process of self-balancing the intra-task losses. The value of the improved homoscedastic uncertainty weighted loss is non-negative. When the training epochs of the image multi-task network reach a specified number of epochs, an improved velocity balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task to obtain a second total loss function. Then, the image multi-task network undergoes a second-stage training process based on the second total loss function to balance the losses between different tasks, until the image multi-task network converges or reaches a preset number of training iterations, resulting in a trained image multi-task network. The improved velocity balancing loss incorporates gradient loss. The image to be predicted is input into the trained image multi-task network for multi-task prediction, and the multi-task prediction result is output.

2. The method according to claim 1, characterized in that, The data loading and preprocessor is used to perform data augmentation on the input image dataset and load a batch of image data and image labels. The loaded image data and image labels are then concatenated and input into the shared feature extractor. The shared feature extractor is used to extract general features from the input image data and image labels and input the general features into each task branch processor. The task branch processor is used to extract relevant features of the corresponding task from the general features, and output the prediction result of the task based on the relevant features.

3. The method according to claim 1, characterized in that, When the number of training epochs of the image multi-task network has not reached the specified number of epochs, an improved homoscedastic uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedastic uncertainty weighted loss function for each task, including: If the number of training epochs of the image multi-task network has not reached the specified number of epochs, the optimizer first adds and updates the weights of the improved homoscedasticity uncertainty weighted loss during the training process; wherein, the initial weights of the improved homoscedasticity uncertainty weighted loss are set according to the number of tasks, and the range of the initial weights is within... Between these numbers, N represents the number of tasks; Then, based on the weights of the improved homoscedasticity uncertainty weighted loss, the improved homoscedasticity uncertainty weighted loss is applied to the sub-task loss within each task to obtain the homoscedasticity uncertainty weighted loss function for each task; wherein, the sub-task loss within each task includes: object detection classification loss, object detection regression loss, and keypoint detection regression loss within the object detection task; semantic segmentation main loss, semantic segmentation first-class auxiliary loss, and semantic segmentation second-class auxiliary loss within the semantic segmentation task; and instance segmentation classification loss and instance segmentation regression loss within the instance segmentation task.

4. The method according to claim 3, characterized in that, The improved homoscedastic uncertainty-weighted loss is expressed as: Where L1(W) represents the sum of the loss functions of the first task, L2(W) represents the sum of the loss functions of the second task, σ1 represents the learnable loss parameter of the first task, and σ2 represents the learnable loss parameter of the second task.

5. The method according to claim 4, characterized in that, The homoscedastic uncertainty-weighted loss function for each task is expressed as follows: Loss obj =Loss uwl (Loss obj-cls ,Loss obj-l1 ,Loss kpt-l1 ); Loss sem-seg =Loss uwl (Loss sem-seg-cls ,Loss sem-seg-aux1 ,Loss sem-seg-aux2 ); Loss ins-seg =Loss uwl (Loss ins-seg-cls ,Loss ins-seg-l1 ); Among them, Loss obj Loss is the homoscedastic uncertainty-weighted loss function for the object detection task. sem-seg Loss is a weighted loss function for homoscedastic uncertainty in semantic segmentation tasks. ins-seg Loss is a weighted loss function for homoscedastic uncertainty in instance segmentation tasks. uwl To improve the homoscedastic uncertainty-weighted loss, Loss obj-cls Loss is the loss for object detection classification. obj-l1 For object detection regression loss, Loss kpt-l1 For keypoint detection regression loss, Loss sem-seg-cls For semantic segmentation, the main loss is Loss. sem-seg-aux1 For semantic segmentation, the first type of auxiliary loss, Loss sem-seg-aux2 Loss is the second type of auxiliary loss for semantic segmentation. ins-seg-cls Loss is the loss for instance segmentation classification. ins-seg-l1 The instance segmentation regression loss is used.

6. The method according to claim 5, characterized in that, By combining the improved homoscedastic uncertainty weighted loss functions of all tasks using the aforementioned improved homoscedastic uncertainty weighted loss, a first total loss function is obtained, expressed as follows: Loss total-first-stage =Loss uwl (Loss obj ,Loss sem-seg ,Loss ins-seg )。 7. The method according to claim 1, characterized in that, The improved velocity balance loss is expressed as: Loss dwa =sum i (λ i (t)L i (t))+βLossgrad(t;w i (t)); Where, λ i (t) represents the velocity weight corresponding to task i, and L i (t) represents the loss function currently corresponding to task i, w i (t) represents the learning speed of task i, and Lossgrad(t; w) represents the learning speed of task i. i (t) represents the gradient loss of task i, β is a hyperparameter used to control the weight of the gradient loss, and the optimal value of β is obtained by solving the gradient descent method; the parameter t represents the sum of the subtask losses of task i.

8. The method according to claim 7, characterized in that, When the training epochs of the image multi-task network reach a specified number, an improved speed-balancing loss is applied to the first total loss function based on the current learning speed and gradient magnitude of each task, resulting in a second total loss function, denoted as: Among them, Loss obj Loss is the homoscedastic uncertainty-weighted loss function for the object detection task. sem-seg Loss is a weighted loss function for homoscedastic uncertainty in semantic segmentation tasks. ins-seg Loss is a weighted loss function for homoscedastic uncertainty in instance segmentation tasks. uwl To improve the weighted loss for homoscedastic uncertainty, λ k1 , λ k2 , λ k3 These are the speed weights for object detection, semantic segmentation, and instance segmentation tasks, respectively.

9. The method according to claim 1, characterized in that, The image to be predicted is input into the trained image multi-task network for multi-task prediction, and the multi-task prediction results are output, including: The image to be predicted is input into the trained image multi-task network to perform object detection prediction, semantic segmentation prediction, and instance segmentation prediction respectively, and the multi-task prediction results are obtained. The multi-task prediction results include the predicted category, detection box coordinates, and key point detection coordinates for object detection, the boundary coordinates for semantic segmentation, and the boundary coordinates and detection box coordinates for instance segmentation.

10. A loss self-balancing system for a multi-task network, characterized in that, The system includes: A multi-task network construction module is used to construct a deep learning-based image multi-task network and input a pre-acquired image dataset into the image multi-task network for training; wherein, the image multi-task network includes a data loading and preprocessor, a shared feature extractor, and a task branch processor, each task branch processor corresponding to a task, the tasks including object detection task, semantic segmentation task, and instance segmentation task; The first-stage training module is used to apply an improved homoscedasticity uncertainty weighted loss to the sub-task loss within each task when the training epochs of the image multi-task network have not reached a specified number of epochs, to obtain the homoscedasticity uncertainty weighted loss function for each task, and to combine the homoscedasticity uncertainty weighted loss functions of all tasks using the improved homoscedasticity uncertainty weighted loss to obtain a first total loss function, and to perform a first-stage training of the image multi-task network for task-internal loss self-balancing based on the first total loss function; wherein, the value of the improved homoscedasticity uncertainty weighted loss is a non-negative number; The second-stage training module is used to, when the number of training epochs of the image multi-task network reaches a specified number of epochs, apply an improved velocity balancing loss to the first total loss function based on the current learning speed and gradient magnitude of each task to obtain a second total loss function, and then perform a second-stage training of the image multi-task network to self-balance the losses between different tasks based on the second total loss function, until the image multi-task network converges or reaches a preset number of training iterations, thus obtaining a trained image multi-task network; wherein, the improved velocity balancing loss introduces gradient loss; The testing module is used to input the image to be predicted into the trained image multi-task network for multi-task prediction and output the multi-task prediction result.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images