A multi-task deep learning training method without multi-task joint annotation

Abstract

The present invention relates to a multi-task deep learning training method without multi-task joint labeling, and relates to the field of computer vision. First, a target task branch network is trained using a single-task labeling data set corresponding to a target task, and each auxiliary branch network is trained using each single-task labeling data set corresponding to a number of auxiliary tasks; then, by calculating the Gram matrix of each auxiliary task feature and the target task feature, a channel attention to each auxiliary task feature is formed, and then the weights of each component of the corresponding auxiliary task feature are adjusted, and then the adjusted multiple groups of auxiliary task features are cascaded and fused with the target task features to construct an integrated network for realizing the target task; finally, the integrated network is trained using the single-task labeling data set corresponding to the target task to realize the target task. The present invention adopts feature fusion based on the Gram channel attention mechanism to realize multi-task training for the target task without the need for a multi-task joint labeling training set.

Description

Technical Field

[0001] A multi-task deep learning training method designed by the present invention that does not require multi-task joint annotation belongs to the field of computer vision technology, and specifically relates to a monocular depth estimation method that realizes multi-task training through feature fusion of auxiliary tasks and target tasks. Background Art

[0002] Multi-task training is a research hotspot in the field of deep learning. By sharing features between related tasks, it can alleviate the overfitting phenomenon of deep learning networks on training data, enhance the overall performance of the network and its generalization ability to other data sets, and is widely used in fields such as computer vision.

[0003] Typical multi-task training requires the network to output the prediction results of multiple tasks, compare them with the true annotations of each task, and then superimpose the loss items of all tasks to form the loss function of multi-task training. Back propagation is used to train the multi-task network. For example, there are currently joint training using semantic segmentation, object detection, instance segmentation, optical flow estimation, and two or three tasks in depth estimation. Kokkinos et al. designed a general network that can be used for 7 tasks at the same time. In addition to directly outputting the prediction results of multiple tasks, some other multi-task training methods use the results of other tasks to optimize the prediction effect of the network on a single target task. For example, Yue et al. used the semantic segmentation results to form a weight adjustment for depth and optimized the monocular depth estimation network. These multi-task training methods often require that the training data set also have corresponding multi-task joint annotations, but obtaining such a multi-task joint annotation training data set is more time-consuming and labor-intensive than obtaining a single-task annotation training data set. Therefore, how to effectively reduce the stringent requirements of multi-task deep learning on the training data set is one of the hot issues worth studying.

[0004] At present, there are many research results in feature fusion in deep learning, including: Multi-feature fusion crossneural network (MFC-Net) that integrates, enhances and transmits multi-layer context features in an iterative manner; Multi-scale feature extraction and attention feature fusion network (MSAF-Net) that enhances the joint attention of channels and spaces of features with different resolutions and then fuses low-level features with high-level features; Wu et al. proposed to use canonical correlation analysis to analyze the relationship between feature groups to be fused, hoping to maximize the intra-group correlation and minimize the extra-group correlation. In general, the fusion of features of different resolutions allows the network to take into account both the overall situation and details, while the fusion of features of different tasks allows the network to increase the constraint information of the target task. Yang et al. proposed to learn the screening matrix by minimizing the trace ratio objective function in the multi-task training of action recognition and image classification, so as to increase the intra-group correlation of a certain feature group and reduce the inter-group correlation of different features. Through the optimization operation of the trace ratio function, Yang et al. achieved the dimensionality reduction of the fused features, but how to effectively express the correlation between different feature groups is still a difficult problem in feature fusion in multi-task deep learning training.

[0005] The multi-task training framework proposed in the present invention does not require a multi-task jointly annotated training data set because it adopts the feature fusion of self-created enhanced feature expression and the matching network training strategy, which effectively reduces the stringent requirements of multi-task deep learning on the training data set. In the feature fusion part, the framework uses the Gram matrix to quantitatively represent the nonlinear correlation between different features, forming attention to the auxiliary task features, and realizes multi-task training monocular depth estimation with good accuracy and generalization performance through feature fusion. The feature fusion of self-created enhanced feature expression and the matching network training strategy are also applicable to other multi-task deep learning networks to achieve the target task. Summary of the invention

[0006] Aiming at the existing multiple single-task networks, the present invention proposes a method for realizing multi-task training by fusing auxiliary tasks and target tasks through features. The network task to be optimized is regarded as the target task, and the remaining tasks are regarded as auxiliary tasks. Several auxiliary task features are fused with the target task features, thereby realizing multi-task deep learning training of the target task.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A multi-task deep learning training method without multi-task joint labeling, comprising:

[0009] Step 1: Use the single-task annotated data set corresponding to the target task to train the target task branch network to ensure that the target task branch network can initially extract features related to the target task;

[0010] Step 2: Use the single-task labeled data sets corresponding to several auxiliary tasks to train the auxiliary branch network, ensuring that each auxiliary branch network can extract relevant features of the corresponding auxiliary task;

[0011] Step 3: Calculate the Gram matrix by using the features extracted by each auxiliary branch network (auxiliary task features) and the features extracted by the target task branch network (target task features). The Gram matrix, as a covariance matrix, can quantitatively represent the nonlinear correlation between each set of auxiliary task features and the target task features, thereby forming a channel attention for each set of auxiliary task features to adjust the weights of each component of the corresponding auxiliary task features, and then cascade the adjusted multiple sets of auxiliary task features with the target task features to achieve feature fusion and construct an integrated network that optimizes and achieves the target task, specifically:

[0012] First, the features of all K auxiliary tasks and the features of the target task are batch normalized, and the feature size is specified to be H×W×C, where W, H, and C are the length, width, and number of channels of the feature, respectively.

[0013] For the kth (k=1,…,K) auxiliary branch network, its extracted features are represented as X ki (i=1,2,…,C k1 ), size is H×W×C k1 , where X ki is the 2D feature of the i-th channel of the k-th auxiliary task, with a total of C k1 channels, the feature extracted by the target task branch network is represented as Y j (j=1,2,…,C2), the size is H×W×C2, where Y j is the two-dimensional feature of the j-th channel, with a total of C2 channels. Using the i-th channel two-dimensional feature X of the k-th auxiliary task ki and the target task feature Y of the jth channel j Calculate the Frobenius matrix inner product of the two: Gram k (i,j) can quantitatively represent X ki With Y j Nonlinear correlation between: When Gram k (i,j) is positive, X ki With Y j Positively correlated; when Gram k (i,j) is negative, X ki With Yj Negatively correlated; when Gram k The larger the absolute value of (i,j), the greater the ki With Y j The stronger the correlation is. By k1 The size of the two-dimensional features of the auxiliary tasks is C2, which can be obtained by multiplying the two-dimensional features of the target tasks by C. k1 ×C2 Gram k matrix.

[0014] For the Gram k The matrix, the i-th row represents the correlation between the auxiliary task features of the i-th channel and the target task features of each channel. Therefore, the maximum value of the i-th row represents the maximum correlation between the auxiliary task features of the i-th channel and the target task features. The larger the row maximum value, the stronger the correlation, that is, it can be considered that the auxiliary task features of the channel can be more represented by the target task features. Therefore, the higher the redundancy of the auxiliary task features of the channel for the fused features, the less attention should be paid to them, and vice versa. Therefore, by k The matrix takes the maximum value of each row to form C 1k ×1, reflects the correlation between each channel feature of the kth auxiliary task and the target task feature. The weight factor of each channel of the kth group of auxiliary task features is obtained by performing a fully connected layer calculation on the correlation vector. The adjusted kth group of auxiliary task features, i.e., the kth group of auxiliary task features to be fused, is obtained by weighted scaling of the auxiliary task features of the corresponding channels.

[0015] Perform the above Gram attention adjustment operation on the features of each group of K auxiliary tasks to obtain a total of K groups of auxiliary task features to be fused. Concatenate these K groups of auxiliary task features to be fused and the target task features, a total of K+1 groups of features, in the channel dimension to form fused features, and construct an integrated network.

[0016] Step 4: Migrate the training parameters of all branch networks to the integrated network constructed in step 3. Specifically, migrate the branch network parameters trained in steps 1 and 2 to each branch of the integrated network constructed in step 3 as the initial parameters for training in step 5.

[0017] Step 5, use the single-task annotated data set corresponding to the target task to train the integrated network initialized in step 4, specifically: fix the encoder parameters of each auxiliary branch in the integrated network to ensure that the auxiliary task branch still extracts auxiliary task related features during the training process, and perform Gram attention adjustment and fusion operation with the target task features on each group of auxiliary task features output after each round of training of K auxiliary tasks through step 3, so as to realize multi-task training of the target task.

[0018] The advantages of the present invention compared with the prior art are:

[0019] (1) The present invention provides a multi-task deep learning training method that does not require multi-task joint labeling. First, all auxiliary task branch networks and target task branch networks are trained separately using a training set with corresponding labels for each task, and then the auxiliary task features are fused with the target task features to form an integrated network. The integrated network is trained using the training set corresponding to the target task to achieve the target task. This method does not require a multi-task joint labeling dataset because it uses Gram attention-based feature fusion technology. It only requires single-task labeling datasets corresponding to the target task and several auxiliary tasks to achieve multi-task deep learning training for the target task, effectively reducing the stringent requirements of multi-task deep learning on the training dataset.

[0020] (2) The covariance matrix Gram matrix is ​​used to mine the nonlinear correlation between the auxiliary task features and the target task features, thereby forming channel attention on the auxiliary task features, realizing the effective expression of the correlation between different auxiliary task feature groups and the target features and the fusion operation with the target task features, thereby improving the implementation effect and generalization performance of the multi-task deep learning network on the target task. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 The overall framework diagram of an embodiment of the present invention is as follows: (a) is a monocular depth estimation network, i.e., a depth branch; (b) is a semantic segmentation network, i.e., a semantic branch; and (c) is an integrated network for the depth estimation task;

[0023] Figure 2 This is a schematic diagram of the feature fusion process of the present invention combined with the Gram matrix;

[0024] Figure 3 It is a schematic diagram of calculating the Gram matrix through deep features and semantic features in the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the specific embodiment described herein is only one of the best embodiments for implementing the present invention, and is used to explain the present invention, make the purpose, technical solution and advantages of the present invention clearer and more specific, and is not used to limit the scope of protection of the present invention, but any other improvements and perfections based on the principles of the present invention should be regarded as the scope of protection of the present invention.

[0026] Embodiment 1:

[0027] This embodiment provides a method for realizing monocular depth estimation through multi-task training by fusing depth features and semantic features. This example only uses the semantic segmentation branch as an auxiliary branch, but it can be further expanded to two or more auxiliary branch networks to optimize monocular depth estimation. This embodiment includes:

[0028] Step 101: Use the KITTI stereo image set (stereo image pairs can be considered as a self-supervised annotation of depth) to train Figure 1 The self-supervised monocular depth estimation branch network shown in (a), referred to as the deep branch network, ensures that the deep branch network can initially realize the extraction of depth estimation related features;

[0029] Step 102: Use the Cityscapes semantic segmentation dataset to train Figure 1 The semantic segmentation branch network shown in (b), referred to as the semantic branch network, ensures that the semantic branch network can initially realize the extraction of semantic segmentation related features;

[0030] Step 103: Calculate the Gram matrix through the features extracted by the semantic branch network (semantic features) and the features extracted by the deep branch network (deep features). The Gram matrix, as a covariance matrix, can quantitatively represent the nonlinear correlation between the semantic features and the deep features, and then form a channel attention to the semantic features to adjust the weights of each component of the semantic features. Then, the adjusted semantic features are cascaded with the deep features to achieve feature fusion and construct an integrated network for monocular depth estimation, such as Figure 1 (c) Figure 1 Taking the feature fusion at g1 in (c) as an example, the specific process diagram is as follows Figure 2 As shown, specifically:

[0031] First, the semantic features extracted by the semantic branch network and the deep features extracted by the deep branch network are normalized. The feature sizes extracted by the semantic branch and the deep branch are both 64×128×256, where 128, 64, and 256 are the length, width, and number of channels of the feature, respectively.

[0032] For the semantic branch network, the extracted features are represented as X 1i (i=1,2,…,256), where X1i is the 2D feature of the i-th channel of the first auxiliary task, i.e., the semantic segmentation task, with a total of 256 channels. The feature extracted by the deep branch network is represented as Y j (j=1,2,…,256), where Y j is the two-dimensional feature of the jth channel, with a total of 256 channels. Using the i-th channel two-dimensional feature X of the semantic segmentation task 1i and the depth feature Y of the jth channel j Calculate the Frobenius matrix inner product of the two: Gram 1 (i,j) can quantitatively represent X 1i With Y j Nonlinear correlation between: When Gram 1 (i,j) is positive, X 1i With Y j Positively correlated; when Gram 1 (i,j) is negative, X 1i With Y j Negatively correlated; when Gram 1 The larger the absolute value of (i,j), the greater the 1i With Y j The stronger the correlation, the better. By multiplying the semantic features of the auxiliary branch network, i.e., the semantic branch network, with the deep features of the 256 channels, a 256×256 Gram 1 Matrix, such as Figure 3 shown.

[0033] For the Gram 1 The matrix, the i-th row represents the correlation between the semantic features of the i-th channel and the depth features of all channels. Therefore, the maximum value of the i-th row represents the maximum correlation between the semantic features of the i-th channel and a channel of the depth feature. When the row maximum value is larger, the stronger the correlation is, it can be considered that the semantic features of the channel can be represented by the depth feature. Therefore, the higher the redundancy of the semantic features of the channel for the fused features, the less attention should be paid to them, and vice versa. Therefore, by 1 The matrix takes the maximum value of each row to form a 256×1 correlation vector, which reflects the correlation between the semantic features of each channel and the deep features. The weight factor of each channel of the semantic feature is obtained by performing a fully connected layer calculation on the correlation vector, and the adjusted semantic feature can be obtained by weighted scaling of the semantic feature.

[0034] This embodiment only uses one auxiliary task, so there is only one set of auxiliary task features to be fused. The semantic features and deep features to be fused are directly cascaded in the channel dimension to form fused features. In the feature fusion process at g2, g3, and g4, only the size of the deep features and semantic features is different from the feature size of the fusion process at g1. The fusion process operation can be obtained by analogy to construct an integrated network.

[0035] Step 104: Migrate the training parameters of the deep branch network and the semantic branch network to the integrated network constructed in step 3, specifically: migrate the deep branch network parameters trained in step 101 to the deep branch of the integrated network in step 103, and migrate the semantic branch network parameters trained in step 102 to the semantic branch of the integrated network in step 103 as the initial parameters for training in step 105;

[0036] Step 105: Use the stereo image set of KITTI (stereo image pairs can be considered as a self-supervised annotation of depth) to train the initialized image in step 104. Figure 1 (c) Integrating the network, specifically: fixing the encoder parameters of the semantic branch in the integrated network to ensure that the semantic branch still extracts semantic features during the training process, and performing Gram attention adjustment and fusion operation with the deep features on the semantic features output after each round of training through step 103 to achieve multi-task training for the depth estimation task.

Claims

1. A multi-task deep learning training method without multi-task joint labeling, comprising: Step 1: Use the single-task annotated data set corresponding to the target task to train the target task branch network to ensure that the target task branch network can initially extract features related to the target task; The single-task annotated data set corresponding to the target task is a stereo image set of KITTI; the target task branch network is a self-supervised monocular depth estimation network; the target task is a monocular depth estimation task; Step 2, using the single-task annotated data sets corresponding to several auxiliary tasks to train the auxiliary branch network, ensuring that each auxiliary branch network can extract the relevant features of the corresponding auxiliary task; the auxiliary branch network includes a semantic segmentation branch network; the semantic segmentation branch network is trained by the semantic segmentation data set of Cityscapes, and the auxiliary task includes a semantic segmentation task; Step 3, calculate the Gram matrix through the auxiliary task features extracted by each auxiliary branch network and the target task features extracted by the target task branch network. The Gram matrix, as a covariance matrix, can quantitatively represent the nonlinear correlation between each group of auxiliary task features and the target task features, and then form a channel attention for each group of auxiliary task features to adjust the weights of each component of the corresponding auxiliary task features, and then cascade the adjusted multiple groups of auxiliary task features with the target task features to achieve feature fusion and construct an integrated network to achieve the target task, specifically: First, batch normalize the features of all K auxiliary tasks and the features of the target task, and set the feature size to H×W×C, where W, H, and C are the length, width, and number of channels of the feature respectively; For the kth (k=1,…,K) auxiliary branch network, its extracted features are represented as X ki (i=1,2,…,C k1 ), size is H×W×C k1 , where X ki is the 2D feature of the i-th channel of the k-th auxiliary task, with a total of C k1 channels, the feature extracted by the target task branch network is represented as Y j (j=1,2,…,C2), the size is H×W×C2, where Y j is the two-dimensional feature of the jth channel, with a total of C2 channels; using X ki and Y j Calculate the inner product of the Frobenius matrices of the two as Gram k (i,j), that is Gram k (i,j) can quantitatively represent X ki With Y j Nonlinear correlation between: When Gram k (i,j) is positive, X ki With Y j Positively correlated; when Gram k (i,j) is negative, X ki With Y j Negatively correlated; when Gram k The larger the absolute value of (i,j), the greater the ki With Y j The stronger the correlation is, the better the result will be. k1 The auxiliary task features are multiplied by C2 target task features to obtain a size of C k1 ×C2 Gram k matrix; For the Gram k The matrix, the i-th row represents the correlation between the auxiliary task features of the i-th channel and the target task features of each channel. Therefore, the maximum value of the i-th row represents the maximum correlation between the auxiliary task features of the i-th channel and the target task features. The larger the row maximum value, the stronger the correlation, that is, it can be considered that the auxiliary task features of the channel can be more represented by the target task features. Therefore, the higher the redundancy of the auxiliary task features of the channel for the fused features, the less attention should be paid to them, and vice versa. Therefore, by k The matrix takes the maximum value of each row to form C 1k ×1, which reflects the correlation between each channel feature of the kth auxiliary task and the target task feature; the weight factor of each channel of the kth group of auxiliary task features is obtained by performing a fully connected layer calculation on the correlation vector, and the weighted expansion and contraction of the auxiliary task features of the corresponding channels can obtain the adjusted kth group of auxiliary task features, that is, the kth group of auxiliary task features to be fused; All K groups of auxiliary task features to be fused and target task features, a total of K+1 groups of features, are cascaded in the channel dimension to form fused features and build an integrated network; Step 4: Migrate the training parameters of all branch networks to the integrated network constructed in step 3. Specifically, migrate the branch network parameters trained in steps 1 and 2 to each branch of the integrated network constructed in step 3 as the initial parameters for training in step 5. Step 5, use the single-task annotated data set corresponding to the target task to train the integrated network initialized in step 4, specifically: fix the encoder parameters of each auxiliary branch in the integrated network to ensure that the auxiliary task branch still extracts auxiliary task related features during the training process, and perform Gram attention adjustment and fusion operation with the target task features on each group of auxiliary task features output after each round of training of K auxiliary tasks through step 3, so as to realize multi-task training of the target task.

Images