Multi-task learning method based on image focal region features

Abstract

The present invention discloses a multi-task learning method based on image focus area features, the method steps include: collecting original images to obtain input images; extracting basic features to obtain basic feature images; generating visual interpretation graphs using branch structures; fusing features of focus areas to obtain feature fusion images; using a single decoder to perform feature compression and segmentation result prediction to obtain a mask image; using the mask image to perform region selection to obtain a mask feature image; and superimposing and multiplying the basic feature image, the visual interpretation graph, and the mask feature image to generate a prediction result. The multi-task learning method realizes multi-task joint learning of image classification, segmentation, and positioning tasks by combining the visual interpretation graph of the focus area with the mask image, solves the problem of negative transfer between classification, segmentation, and positioning tasks in the current multi-task learning field, realizes consistency of multiple tasks at the semantic level, and improves the effects of image classification, segmentation, and positioning tasks.

Description

Technical Field

[0001] The present invention relates to the field of deep learning image multi-task learning, and in particular to a multi-task learning method based on image focus area features. Background Art

[0002] With the development of deep learning technology, the efficiency of multi-task learning methods has been significantly improved. Multi-task learning can achieve multiple tasks through end-to-end training of a single model, and improve learning efficiency by sharing information between tasks. Traditional image multi-task learning methods often use the method of sharing basic backbone networks to achieve multi-task knowledge sharing. This method easily causes the model to pay more attention to a certain task, and the correlation is not strong. It may lead to the inability to complement the information between multiple tasks, and even affect the performance of the original single-task network. Modeling the correlation between multiple tasks is still a difficult problem in this field.

[0003] Therefore, existing image multi-task learning methods often have negative transfer problems between multiple tasks in image vision tasks such as classification and segmentation, which brings challenges to image multi-task learning. Summary of the invention

[0004] The purpose of the invention is to provide a multi-task learning method based on image focal area features, which can solve the problem of negative transfer between classification, segmentation and positioning tasks in the current multi-task learning field, achieve consistency of multiple tasks at the semantic level, and improve the effects of image classification, segmentation and positioning tasks.

[0005] Technical solution: The multi-task learning method based on image focus area features of the present invention comprises the following steps:

[0006] Step 1: collect a variety of annotated original images and preprocess them to obtain input images for multi-task learning;

[0007] Step 2, using a feature encoder to extract basic features of the input image to obtain a basic feature image;

[0008] Step 3: Generate a visual explanation image based on the basic feature image using a branch structure with an attention mechanism;

[0009] Step 4: Use a dual encoder network to fuse the features of the focus area of ​​the input image and the visual explanation map to obtain a feature fused image;

[0010] Step 5, using a single decoder to perform feature compression on the feature fusion image and predict the segmentation result to obtain a mask image for segmentation prediction;

[0011] Step 6: Use the mask image to select a region on the visual interpretation image to obtain a mask feature map;

[0012] Step 7: superimpose and multiply the basic feature image, the visual explanation image, and the mask feature image, and generate a prediction result by superimposing and multiplying the focal area of ​​the prediction feature.

[0013] Furthermore, in step 1, the specific steps of collecting a variety of annotated original images and preprocessing them are as follows:

[0014] Step 1.1, collect a variety of annotated original images, and then form an image dataset from the collected original images;

[0015] Step 1.2, unify the annotations in the image dataset into JSON format;

[0016] Step 1.3, perform random data enhancement processing on the contrast, saturation and brightness of the original images in the image dataset.

[0017] Furthermore, in step 1.2, the specific steps for unifying the annotations in the image dataset into JSON format are as follows:

[0018] Step 1.2.1, generating the contour of the segmented object on the corresponding original image according to the annotations in the image data set;

[0019] Step 1.2.2, take points at intervals of pixels on the contour line to obtain segmentation contour points;

[0020] Step 1.2.3, find the circumscribed rectangle bbox of each segmentation contour point distribution area, and bind the segmentation contour points, circumscribed rectangle bbox and the corresponding original image in the image dataset to generate a unified JSON format file.

[0021] Furthermore, in step 3, the branch structure with the attention mechanism includes an auxiliary classifier and a visual explanation branch. The specific steps of generating a visual explanation graph according to the basic feature image are as follows:

[0022] Step 3.1, using the auxiliary classifier to predict the basic feature image through multi-scale transformer and softmax prediction classification to obtain the predicted feature image;

[0023] In step 3.2, the visual explanation branch is used to perform feature focusing on the predicted feature image to generate a feature weight map, and then the feature weight map is visualized to generate a visual explanation map.

[0024] Furthermore, in step 3.2, the expression for generating the visual explanation graph is:

[0025] M = sigmoid(BN(PointConv(F)))

[0026] Where M represents the visual explanation graph, sigmoid represents the activation function, BN represents batch normalization, PointConv represents point-by-point convolution, and F represents the basic feature image.

[0027] Furthermore, in step 4, the dual encoder network includes an interpreted image branch and an original image branch, the interpreted image branch includes a multi-level convolutional layer and a mixed pooling layer, and the original image branch includes a feature extraction layer.

[0028] Furthermore, in step 4, the specific steps of performing feature fusion of the focus area are:

[0029] Step 4.1, use a multi-level convolutional layer to perform multi-level feature extraction on the visual explanation map to obtain a multi-level feature map, and then use a mixed pooling layer to perform multiple pooling operations on the multi-level feature map to obtain a multi-level visual explanation map;

[0030] Step 4.2, using the feature extraction layer to perform multi-level feature extraction on the original image to obtain a multi-level extracted feature map;

[0031] In step 4.3, the multi-level extracted feature map and the multi-level visual interpretation map are fused with the focus area features through the feature concatenation layer to obtain a feature fused image.

[0032] Furthermore, in step 5, the specific steps of using a single decoder to perform feature compression on the feature fusion image and predict the segmentation result are as follows:

[0033] Step 5.1, compress the feature fusion image by point-by-point convolution to obtain a compressed feature image. The expression for feature compression by point-by-point convolution is:

[0034] PointConv(G)

[0035] Where G represents the feature matrix vector of the compressed feature fusion image;

[0036] Step 5.2, upsample the compressed feature image to increase the resolution, and then perform a convolution operation with the output channel being half of the input channel to reduce the number of channels. At the same time, use the jump connection of the low-level features with the same resolution to reduce the loss of information. That is, when the image passes through the upsampling layer, the spatial dimension becomes larger. At the same time, since the output image channel should be 1, the network needs to continuously reduce the output channel dimension of each layer;

[0037] Step 5.3, map each point in the compressed feature image to a probability value in the range of [0,1] through an activation function, and then compare the probability value of each point with a preset threshold. If it is greater than the threshold, it is determined to be a point on the segmentation target. After all points are compared, a single-channel feature matrix with the same height and width as the original image is finally obtained. A mask image for segmentation prediction is generated based on the single-channel feature matrix, and the mask image is output by a single decoder;

[0038] Step 5.4, select the leftmost point, rightmost point, topmost point and bottommost point of the white pixel points of the mask image to generate a mask circumscribed rectangle.

[0039] Furthermore, in step 6, the specific steps of using the mask image to select the region on the visual interpretation image are as follows:

[0040] Step 6.1, use the black pixels of the mask image to mask out the invalid information on the visual interpretation image;

[0041] In step 6.2, the white pixels of the mask image are used to select important information on the visual explanation image as the mask feature image. At the same time, the loss function of the focus loss of the attention area is used to calculate the loss between the mask image and the visual explanation image. By calculating the loss between the mask image and the visual explanation image, the subsequent prediction results pay more attention to the focus area. The loss function is expressed as:

[0042] Loss = (M × Q) 2

[0043] Where M represents the visual interpretation map and Q represents the pixel matrix of the mask map.

[0044] Furthermore, in step 7, the expression for generating the prediction result by superimposing the focus area of ​​the multiplication-enhanced prediction feature is:

[0045] F c =F×M×Q+F

[0046] In the formula, F c represents the prediction result feature image, F represents the basic feature image, M represents the visual explanation image, and Q represents the pixel matrix of the mask image.

[0047] Compared with the prior art, the present invention has the following beneficial effects: the features of the focus area of ​​the input image and the visual interpretation map are fused through a dual encoder network, the key information of the multi-level features is enhanced, and a segmentation result based on focus prior is achieved; the information interaction between multiple tasks is achieved by selecting the area on the visual interpretation map through a mask map, which can effectively improve the correlation between image recognition tasks, solve the negative transfer problem existing in multi-task learning, and improve the effect of image multi-task learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the method of the present invention;

[0049] Figure 2 is a network structure diagram of the present invention;

[0050] Figure 3 This is a visualization diagram of the prediction results of the present invention. DETAILED DESCRIPTION

[0051] The following specific embodiments illustrate the implementation of the present application. People familiar with this technology can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. The technical features involved in the different implementations of the present application described can be combined with each other as long as they do not conflict with each other.

[0052] Embodiment 1:

[0053] like Figure 1-3 As shown, the multi-task learning method based on image focus area features disclosed in the present invention includes the following steps:

[0054] Step 1: collect a variety of annotated original images and preprocess them to obtain input images for multi-task learning;

[0055] Step 2: Use a feature encoder to extract basic features of the input image to obtain a basic feature image. The feature encoder consists of three convolutional layers and is used to extract basic features of the input image, including texture features, color features, etc.

[0056] Step 3: Generate a visual explanation image based on the basic feature image using a branch structure with an attention mechanism;

[0057] Step 4: Use a dual encoder network to fuse the features of the focus area of ​​the input image and the visual explanation map to obtain a feature fused image;

[0058] Step 5, using a single decoder to perform feature compression on the feature fusion image and predict the segmentation result to obtain a mask image for segmentation prediction;

[0059] Step 6: Use the mask image to select a region on the visual interpretation image to obtain a mask feature map;

[0060] Step 7: superimpose and multiply the basic feature image, the visual explanation image, and the mask feature image, and generate a prediction result by superimposing and multiplying the focal area of ​​the prediction feature.

[0061] Through the dual encoder network, the features of the focus area of ​​the input image and the visual explanation map are fused to enhance the key information of the multi-level features and achieve segmentation results based on focus priors. Through the mask map, region selection is performed on the visual explanation map to achieve information interaction between multiple tasks, which can effectively improve the correlation between image recognition tasks, solve the negative transfer problem in multi-task learning, and improve the effect of image multi-task learning.

[0062] Furthermore, in step 1, the specific steps of collecting a variety of annotated original images and preprocessing them are as follows:

[0063] Step 1.1, collect a variety of annotated original images, including images with instance annotations and pixel-level annotations, and then form an image dataset with the collected original images;

[0064] Step 1.2, unify the annotations in the image dataset into JSON format;

[0065] Step 1.3, performing random data enhancement processing of contrast, saturation and brightness on the original images in the image dataset;

[0066] In step 1.4, the preprocessed images in the image dataset are used as input images for multi-task learning.

[0067] Furthermore, in step 1.2, the specific steps for unifying the annotations in the image dataset into JSON format are as follows:

[0068] Step 1.2.1, generating the contour of the segmented object on the corresponding original image according to the annotations in the image data set;

[0069] Step 1.2.2, take points at intervals of pixels on the contour line to obtain segmentation contour points;

[0070] Step 1.2.3, find the circumscribed rectangle bbox of each segmentation contour point distribution area, and bind the segmentation contour points, circumscribed rectangle bbox and the corresponding original image in the image dataset to generate a unified JSON format file.

[0071] Furthermore, in step 3, the branch structure with the attention mechanism includes an auxiliary classifier and a visual explanation branch. The specific steps of generating a visual explanation graph according to the basic feature image are as follows:

[0072] Step 3.1, using the auxiliary classifier to predict the basic feature image through multi-scale transformer and softmax prediction classification to obtain the predicted feature image;

[0073] In step 3.2, the visual explanation branch is used to perform feature focusing on the predicted feature image to generate a feature weight map, and then the feature weight map is visualized to generate a visual explanation map.

[0074] Furthermore, in step 3.2, the expression for generating the visual explanation graph is:

[0075] M = sigmoid(BN(PointConv(F)))

[0076] Where M represents the visual explanation graph, sigmoid represents the activation function, BN represents batch normalization, PointConv represents point-by-point convolution, and F represents the basic feature image.

[0077] Furthermore, in step 4, the dual encoder network includes an interpreted image branch and an original image branch, the interpreted image branch includes a multi-level convolutional layer and a mixed pooling layer, and the original image branch includes a feature extraction layer.

[0078] Furthermore, in step 4, the specific steps of performing feature fusion of the focus area are:

[0079] Step 4.1, use a multi-level convolutional layer to perform multi-level feature extraction on the visual explanation map to obtain a multi-level feature map, and then use a mixed pooling layer to perform multiple pooling operations on the multi-level feature map to obtain a multi-level visual explanation map;

[0080] Step 4.2, using the feature extraction layer to perform multi-level feature extraction on the original image to obtain a multi-level extracted feature map;

[0081] In step 4.3, the multi-level extracted feature map and the multi-level visual interpretation map are fused with the focus area features through the feature concatenation layer to obtain a feature fused image.

[0082] Furthermore, in step 5, the specific steps of using a single decoder to perform feature compression on the feature fusion image and predict the segmentation result are as follows:

[0083] Step 5.1, compress the feature fusion image by point-by-point convolution to obtain a compressed feature image. The expression for feature compression by point-by-point convolution is:

[0084] PointConv(G)

[0085] Where G represents the feature matrix vector of the compressed feature fusion image;

[0086] Step 5.2, upsample the compressed feature image to increase the resolution, and then perform a convolution operation with the output channel being half of the input channel to reduce the number of channels, while using the skip connection of the low-level features of the same resolution to reduce the loss of information;

[0087] Step 5.3, map each point in the compressed feature image to a probability value in the range of [0,1] through an activation function, and then compare the probability value of each point with a preset threshold. If it is greater than the threshold, it is determined to be a point on the segmentation target. After all points are compared, a single-channel feature matrix with the same height and width as the original image is finally obtained. A mask image for segmentation prediction is generated based on the single-channel feature matrix, and the mask image is output by a single decoder;

[0088] Step 5.4, select the leftmost point, rightmost point, topmost point and bottommost point of the white pixel points of the mask image to generate a mask circumscribed rectangle, and the mask circumscribed rectangle is used as part of the classification result and output as a learning result.

[0089] Furthermore, in step 6, the specific steps of using the mask image to select the region on the visual interpretation image are as follows:

[0090] Step 6.1, use the black pixels of the mask image to mask out the invalid information on the visual interpretation image;

[0091] In step 6.2, the white pixels of the mask image are used to select important information on the visual explanation image as the mask feature image. At the same time, the loss function of the focus loss of the attention area is used to calculate the loss of the mask image and the visual explanation image. The loss function is expressed as:

[0092] Loss = (M × Q) 2

[0093] Where M represents the visual interpretation map and Q represents the pixel matrix of the mask map.

[0094] Furthermore, in step 7, the expression for generating the prediction result by superimposing the focus area of ​​the multiplication-enhanced prediction feature is:

[0095] F c =F×M×Q+F

[0096] In the formula, F c represents the prediction result feature image, F represents the basic feature image, M represents the visual explanation image, and Q represents the pixel matrix of the mask image.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-task learning method based on image focus area features, characterized in that: The steps include: Step 1: collect a variety of annotated original images and preprocess them to obtain input images for multi-task learning; Step 2, using a feature encoder to extract basic features of the input image to obtain a basic feature image; Step 3: Generate a visual explanation image based on the basic feature image using a branch structure with an attention mechanism; Step 4, using a dual encoder network to fuse the features of the focus area of ​​the input image and the visual interpretation image to obtain a feature fused image; Step 5, using a single decoder to perform feature compression on the feature fusion image and predict the segmentation result to obtain a mask image for segmentation prediction; Step 6: Use the mask image to select a region on the visual interpretation image to obtain a mask feature map; Step 7, superimposing and multiplying the basic feature image, the visual interpretation image, and the mask feature map, and generating a prediction result by superimposing and multiplying the focus area of ​​the enhanced prediction feature; In step 4, the dual encoder network includes an interpretation image branch and an original image branch, the interpretation image branch includes a multi-level convolution layer and a mixed pooling layer, and the original image branch includes a feature extraction layer; In step 4, the specific steps of feature fusion of the focus area are: Step 4.1, use a multi-level convolutional layer to perform multi-level feature extraction on the visual explanation map to obtain a multi-level feature map, and then use a mixed pooling layer to perform multiple pooling operations on the multi-level feature map to obtain a multi-level visual explanation map; Step 4.2, using the feature extraction layer to perform multi-level feature extraction on the original image to obtain a multi-level extracted feature map; In step 4.3, the multi-level extracted feature map and the multi-level visual interpretation map are fused with the focus area features through the feature concatenation layer to obtain a feature fused image.

2. The multi-task learning method based on image focus area features according to claim 1, characterized in that: In step 1, the specific steps of collecting multiple annotated original images and preprocessing them are as follows: Step 1.1, collect a variety of annotated original images, and then form an image dataset from the collected original images; Step 1.2, unify the annotations in the image dataset into JSON format; Step 1.3, perform random data enhancement processing on the contrast, saturation and brightness of the original images in the image dataset.

3. The multi-task learning method based on image focus area features according to claim 2, characterized in that: In step 1.2, the specific steps for unifying the annotations in the image dataset into JSON format are as follows: Step 1.2.1, generating the contour of the segmented object on the corresponding original image according to the annotations in the image data set; Step 1.2.2, take points at intervals of pixels on the contour line to obtain segmentation contour points; Step 1.2.3, find the circumscribed rectangle bbox of each segmentation contour point distribution area, and bind the segmentation contour points, circumscribed rectangle bbox and the corresponding original image in the image dataset to generate a unified JSON format file.

4. The multi-task learning method based on image focus area features according to claim 1, characterized in that: In step 3, the branch structure with attention mechanism includes auxiliary classifier and visual explanation branch. The specific steps of generating visual explanation map according to basic feature image are as follows: Step 3.1, using the auxiliary classifier to predict the basic feature image through multi-scale transformer and softmax prediction classification to obtain the predicted feature image; In step 3.2, the visual explanation branch is used to perform feature focusing on the predicted feature image to generate a feature weight map, and then the feature weight map is visualized to generate a visual explanation map.

5. The multi-task learning method based on image focus area features according to claim 4, characterized in that: In step 3.2, the expression for generating the visual explanation graph is: M = sigmoid(BN(PointConv(F))) Where M represents the visual explanation graph, sigmoid represents the activation function, BN represents batch normalization, PointConv represents point-by-point convolution, and F represents the basic feature image.

6. The multi-task learning method based on image focus area features according to claim 1, characterized in that: In step 5, the specific steps of using a single decoder to perform feature compression on the feature fusion image and predict the segmentation result are as follows: Step 5.1, compress the feature fusion image by point-by-point convolution to obtain a compressed feature image. The expression for feature compression by point-by-point convolution is: PointConv(G) Where G represents the feature matrix vector of the compressed feature fusion image; Step 5.2, upsample the compressed feature image to increase the resolution, and then perform a convolution operation with the output channel being half of the input channel to reduce the number of channels, while using the skip connection of the low-level features of the same resolution to reduce the loss of information; Step 5.3, through the activation function, each point in the compressed feature image is mapped to a probability value in the range of [0,1], indicating the probability of the point being a point on the segmentation target, and then the probability value of each point is compared with the preset threshold. If it is greater than the threshold, it is determined to be a point on the segmentation target, and the point is set to a white pixel. After all points are compared, a single-channel feature matrix with the same height and width as the original image is finally obtained. A mask image for segmentation prediction is generated based on the single-channel feature matrix, and the mask image is output by a single decoder; Step 5.4, select the leftmost point, rightmost point, topmost point and bottommost point of the white pixel points of the mask image to generate a mask circumscribed rectangle.

7. The multi-task learning method based on image focus area features according to claim 1, characterized in that: In step 6, the specific steps of using the mask image to select the region on the visual interpretation image are: Step 6.1, use the black pixels of the mask image to mask out the invalid information on the visual interpretation image; In step 6.2, the white pixels of the mask image are used to select important information on the visual explanation image as the mask feature image. At the same time, the loss function of the focus loss of the attention area is used to calculate the loss of the mask image and the visual explanation image. The loss function is expressed as: Loss=(M×Q) 2 Where M represents the visual interpretation map and Q represents the pixel matrix of the mask map.

8. The multi-task learning method based on image focus area features according to claim 1, characterized in that: In step 7, the expression for generating the prediction result by superimposing the focus area of ​​the multiplication enhancement prediction feature is: F c =F×M×Q+F In the formula, F c represents the prediction result feature image, F represents the basic feature image, M represents the visual explanation image, and Q represents the pixel matrix of the mask image.

Images