Small sample segmentation method facing test scene without mask supervision

Abstract

The application discloses a kind of test scene unmasked supervision-oriented small sample segmentation method, and the method specifically includes the following steps: obtaining the image dataset for training deep neural network model;Image pair and its mask constructed using image dataset are used as supervision signal, and the pre-designed iterative optimization deep neural network model without mask is trained by cross-entropy loss;The trained deep neural network model is used to output the prediction mask of the image to be segmented without mask supervision.The method of the application is suitable for training to provide large-scale labeled data, but when testing, there is no any dense label information except other images belonging to the same category as the image to be segmented in small sample segmentation task, solve the current small sample segmentation task in inference still needs to densely mark new class image demand, obtain good segmentation performance under the condition of only using image pair, especially for single target segmentation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a small sample segmentation method for test scene unmasked supervision, a computer device and a storage medium. BACKGROUND

[0002] Semantic segmentation is one of the important basic tasks in computer vision, and can be widely applied to automatic driving, medical image segmentation and other tasks. With continuous improvement of network structure and model, the semantic segmentation task has achieved excellent performance. However, the semantic segmentation model is data-driven, and needs large-scale pixel-level dense labeled image data for training, and the model does not have the ability to generalize to new class data. At present, the semantic segmentation model can only segment the target classes appearing in the training process, and when new class images appear, large-scale and large number of new class dense labeled samples need to be collected to retrain the model, which is not practical in real application, thus limiting their further popularization and application.

[0003] Unlike existing machine learning models relying on large data training, humans can use accumulated knowledge in the past to quickly recognize new concepts through one or a few new class samples. Inspired by human learning of new knowledge and narrowing the gap between artificial intelligence and human learning, people have proposed small sample learning. Small sample learning reduces the demand for large-scale labeled data sets, and has attracted a lot of attention in recent years. Inspired by small sample learning, people have proposed small sample semantic segmentation task, which extends the idea of small sample learning to the segmentation task. Small sample segmentation realizes the segmentation of new class targets with only one or a few dense labeled samples, reducing the demand for large-scale dense labeled data in the semantic segmentation task.

[0004] In the small sample segmentation task, one or a few new class dense labeled reference samples are still needed to segment new class images at the test time. Although the dependence on large-scale data has been greatly reduced compared to the traditional semantic segmentation task, there is still a certain demand for labeled data. SUMMARY

[0005] To solve the above problems, the purpose of the present application is to provide a small sample segmentation method for test scene without mask supervision, which does not need to provide any dense labeled sample information during inference, and only uses another image containing the same category target as the image to be segmented to provide guidance information, further reducing the demand for labeled data. Compared with dense labeled mask information, images are very cheap and easy to obtain. A large number of similar images can be obtained in the way of image search to form the input image pair of the model, and the model is trained relying on the existing large-scale dense labeled data set, so that the model learns to mine the common category features in the image. The model can segment the target to be segmented only by using the image pair containing the same category, greatly reducing the time-consuming and laborious labeling process, reducing human interaction, and showing strong generalization ability.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is:

[0007] The small sample segmentation method for test scene without mask supervision comprises the following steps:

[0008] S1, obtaining an image data set for training a deep neural network model;

[0009] S2, using the image pair and its mask constructed by the image data set as a supervision signal, training a pre-designed iterative optimization deep neural network model without mask through cross-entropy loss;

[0010] S3, using the trained deep neural network model to output the predicted mask of the image to be segmented without mask supervision.

[0011] Further, in step S1, the training image data set is where N is the number of constructed tasks (episodes) in the training set, and respectively represent the i-th support image providing category guidance information and its corresponding mask, and respectively represent the i-th query image to be segmented and its corresponding mask, M i ∈{1,…,C}, C represents the total number of categories, and each category contains multiple images. The obtained mask is pre-processed, and all category target objects appearing in the support and query images are set as foreground, and the rest are set as background.

[0012] Further, the deep neural network model comprises three components: a feature extractor, a mask generation module and a decoder, wherein the feature extractor is a ResNet-50 pre-trained using an ImageNet data set, and the mask generation module and the decoder are parts that need to be trained.

[0013] Further, step S2 specifically comprises:

[0014] S21, in the process of training the deep neural network model, first randomly sample a batch of image pairs from the image dataset to obtain a sample set Wherein the batch size N bs is given in advance;

[0015] S22, input the images in the image pair sample set B into the feature extractor to obtain the feature maps of the support image and the query image Wherein L is the total number of layers in each module of the feature extractor;

[0016] S23, pass the high-level feature maps of the support image and the query image from the last module of the feature extractor through the mask generation module, and obtain the initial support prediction mask by using similarity calculation and cross attention mechanism For subsequent segmentation tasks, and calculate the cross entropy loss between the real support mask value;

[0017] S24, input the feature maps of the support and query features from the last two modules of the feature extractor into the feature enhancement module in the decoder, and use the self-attention mechanism to capture global information and enhance the support and query features;

[0018] S25, input the initial support prediction mask and the support and query features output by the feature enhancement module into the Transformer structure in the decoder, wherein the query feature is taken as Query, the support feature is taken as Key, The Value is calculated, and the similarity between the query feature and the support feature is weighted to obtain the initial prediction mask of the query image Fuse the bottom-level features of the query and support images from the first two modules of the feature extractor, and obtain the query prediction mask through the decoder

[0019] S26, repeat step S24, and then take the query mask obtained in step S25 as the Value of the Transformer module in the decoder, take the support feature as Query, and take the query feature as Key, to calculate the initial prediction mask of the support image, and fuse the bottom-level features of the query and support images from the first two modules of the feature extractor, and obtain the support prediction mask through the decoder

[0020] S27, repeat steps S24 to S26 for T times, and calculate the cross entropy loss between the query and support prediction masks obtained in the last iteration and the real mask value after the iteration is completed;

[0021] S28, the total loss function is obtained based on the cross-entropy loss between each prediction mask and the real mask:

[0022]

[0023] Wherein, beta, gamma are preset weight parameters, used to represent the contribution of the three prediction masks;

[0024] S29, according to the total loss function obtained, using Adam optimizer or momentum stochastic gradient descent optimizer, and back propagation algorithm training deep neural network model;

[0025] Further, in step S28, the cross-entropy loss function L is:

[0026]

[0027] Wherein The real label of image i at position j, The output prediction result of model for image i at position j, and HW is the pixel number of the whole image.

[0028] Further, in step S3, the operation without mask supervision during inference is: during inference, for the image to be segmented, optionally one image belonging to the same category as the target to be segmented forms an image pair, and the image pair is input into the trained deep neural network model for prediction, and the prediction mask of the query image to be segmented generated by the last iteration is output As the final segmentation prediction result.

[0029] The test scene mask-free supervision small sample segmentation method of the application has the following contributions compared with the existing method:

[0030] Firstly, the application proposes a new scene setting, which only uses image pairs to perform small sample segmentation tasks during inference, and proposes a test scene mask-free supervision small sample segmentation method.

[0031] Secondly, the application designs a mask generation module, which generates an initial support prediction mask using the high-level semantic features of the support image and the query image, and provides the class information of the target to be segmented for the segmentation task; a feature enhancement module is proposed, which learns the global features of the features using the self-attention mechanism, and the module is parameter shared for support and query features, which can better mine the class information of the common target in the two images; an alternating iterative optimization module is proposed, which alternately optimizes the query and support prediction masks by alternating the positions of the support features and the query features in the input of the Transformer module, and then optimizes the current iteration using the prediction mask output by the last iteration.

[0032] Finally, the present application does not need to provide any dense labeled reference samples for segmentation during inference, and only uses easily obtained images to input the network for inference prediction, greatly reducing the demand for dense labeled samples in the segmentation task, obtaining competitive performance in the single target segmentation task, and having certain generalization ability.

[0033] The test scene oriented small sample segmentation method without mask supervision of the present application has good application value, and the whole process realizes end-to-end automatic processing, and no additional labeled data is needed except training data. For example, in some fields with less labeled samples, the method can be used to roughly label them first, and then manually correct the label. For some images without semantic labels, as long as the same category of pictures can be retrieved, the segmentation can also be performed. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The flowchart of the test scene oriented small sample segmentation method without mask supervision of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0036] On the contrary, the present application covers any alternative, modification, equivalent method and scheme made on the essence and scope of the present application defined by the claims. Further, in order to make the public better understand the present application, some specific details are described in detail in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0037] REFERENCE Figure 1 In the preferred embodiment of the present application, the test scene oriented small sample segmentation method without mask supervision comprises the following steps:

[0038] First, an image data set for training a deep neural network model is obtained. The image data set is where N is the number of tasks (episodes) constructed in the image data set, and respectively represent the i-th support image providing category guidance information (support) and its corresponding mask, and respectively represent the i-th query image to be segmented (query) and its corresponding mask, M iC represents the total number of categories, and each category contains multiple images. For the mask obtained by sampling, all category target objects that appear in both the support image and the query image are set as foreground, and the rest is set as background.

[0039] Secondly, the image pair and its mask constructed by the image dataset are used as a supervision signal to train the pre-designed iterative optimization deep neural network model without a mask through cross-entropy loss. The deep neural network model includes three components: a feature extractor, a mask generation module, and a decoder. The feature extractor is mainly a ResNet-50 pre-trained on ImageNet, and the network parameters are frozen during the training process. The mask generation module and the decoder are the parts that need to be trained. The specific steps include the following:

[0040] Firstly, in the training of the deep neural network model, the batch processing method is adopted. First, a batch of image pair sample set B is randomly sampled from the image dataset Where the batch size N bs is pre-defined.

[0041] Secondly, the images in the image pair sample set B are input into the feature extractor to obtain the feature maps of the support image and the query image Where L is the total number of layers in each module in the feature extractor.

[0042] Thirdly, the high-level feature maps of the support image and the query image from the last module of the feature extractor are input into the mask generation module to obtain the initial support prediction mask for the subsequent segmentation task, and the cross-entropy loss between the real support mask value and the initial support prediction mask value is calculated:

[0043]

[0044] Fourthly, the feature maps of the support and query features from the last two modules of the feature extractor are input into the feature enhancement module in the decoder to capture global information and enhance the support and query features using the self-attention mechanism.

[0045] Fifthly, the initial support prediction mask and the support and query features output by the feature enhancement module are input into the Transformer module, where the query feature is taken as Query, the support feature is taken as Key, and the support feature is taken as Value, the similarity between the query feature and the support feature is calculated, and the The initial prediction mask of the query image is obtained by weighting, and the query prediction mask is obtained by fusing the underlying features of the query and support images from the first two modules of the feature extractor through the decoder

[0046] In the sixth step, the fourth step is repeated, and then the query mask obtained in the fifth step is taken as the Value of the Transformer module, the support features are taken as the Query, and the query features are taken as the Key to calculate the initial prediction mask of the support image, and the support prediction mask is obtained by fusing the underlying features of the query and support images from the first two modules of the feature extractor through the decoder

[0047] In the seventh step, the fourth step, the fifth step and the sixth step are repeated for T times, and the cross-entropy loss between the query and support prediction masks obtained in the last iteration and the real mask value is calculated after the iteration is completed.

[0048]

[0049]

[0050] In the eighth step, the total loss function is obtained based on the cross-entropy loss between each prediction mask and the real mask, and is as follows:

[0051]

[0052] Wherein, alpha, beta and gamma are weight parameters set in advance, and are used to represent the contribution of the three prediction masks.

[0053] In the ninth step, the deep neural network model is trained using the Adam optimizer or the stochastic gradient descent optimizer with momentum and the back propagation algorithm based on the obtained total loss function.

[0054] Finally, the trained deep neural network model is used to output the prediction mask of the image to be segmented without mask supervision. In inference, for an image to be segmented, an image pair is selected, in which the image to be segmented and the image belong to the same category, and the image pair is input into the trained deep neural network model for prediction, and the prediction mask of the query image to be segmented obtained in the last iteration is output as the final segmentation prediction result.

[0055] Through the above technical solutions, the small sample segmentation method without mask supervision for the test scene is provided. In inference, the present application does not need to provide any densely labeled reference sample for segmentation, but only uses the easily obtained image pair to input into the network for inference and prediction, thereby greatly reducing the demand for densely labeled samples in the segmentation task, and having a certain generalization ability.

[0056] ​The embodiment of the present application further provides a computer, comprising a memory and a processor, which are connected in communication with each other, the memory stores computer instructions, and the processor executes the steps of the small sample segmentation method for test scenario oriented unmasked supervision shown in the above embodiment by executing the computer instructions.

[0057] The embodiment of the present application further provides a computer readable storage medium, which comprises a stored program, wherein the computer readable storage medium stores computer instructions for making a computer execute the steps of the small sample segmentation method for test scenario oriented unmasked supervision shown in the above embodiment when the computer runs.

[0058] Obviously, those skilled in the art should understand that each module or each step of the present application can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and optionally, each module or each step can be realized by program codes executable by a computing device, so that each module or each step can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different sequences, or each module can be manufactured into an integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0059] The above only describes the preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A small sample segmentation method for test scene unmasked supervision, characterized in that, The method comprises the following steps: S1, obtaining an image data set for training a deep neural network model; S2, using the image pairs and their masks constructed by the image data set as a supervision signal, training a pre-designed iterative optimization deep neural network model without a mask through cross-entropy loss, specifically comprising: S21, in training the deep neural network model, a batch processing manner is adopted, first, a batch of image samples are randomly sampled from the image dataset wherein the batch size predefined; S22, inputting the image pairs in the image pair sample set into a feature extractor to obtain feature maps of the support images and the query images wherein L is the total number of layers of each module in the feature extractor​ S23, pass the high-level semantic feature maps of the support image and the query image from the last module of the feature extractor through the mask generation module, and obtain an initial support prediction mask by using similarity calculation and cross attention mechanism for subsequent segmentation tasks, and calculate the cross-entropy loss between the support prediction mask and the real support mask value; S24, inputting the feature maps of the support and query features from the last two modules of the feature extractor into the feature enhancement module in the decoder, capturing global information and enhancing the support and query features using a self-attention mechanism; S25, obtaining the initial support prediction mask and the support and query features output by the feature enhancement module are input into the Transformer module in the decoder, where the query features are as Query, the support features are as Key, the similarity between the query features and the support features is calculated as Value, and the similarity is weighted to obtain the initial prediction mask of the query image After the initial prediction mask is fused with the underlying features of the query and support images from the first two modules of the feature extractor, the query prediction mask is obtained through the decoder ; S26, repeating step S24, and then obtaining the query prediction mask from step S25 As the Value of the Transformer module in the decoder, the support features are taken as the Query, the query features are taken as the Key, and the initial prediction mask of the support image is calculated. After fusion with the underlying features of the query and support images from the first two modules of the feature extractor, the support prediction mask is obtained through the decoder ; S27, repeat iteration Sub-step S24 to S26, after the iteration, the cross-entropy loss between the query and support prediction mask obtained in the last iteration and the real mask value is calculated. S28, obtaining a total loss function based on the cross-entropy loss between each predicted mask and the real mask as: wherein is a pre-set weight parameter to represent the contribution of the three prediction masks; S29, training the deep neural network model using the Adam optimizer or the stochastic gradient descent optimizer with momentum, and the back propagation algorithm according to the obtained total loss function; S3, using the trained deep neural network model to output a predicted mask of the image to be segmented without mask supervision.

2. The test scenario oriented small sample segmentation method without mask supervision of claim 1, wherein, In step S1, the image dataset is ,in The number of tasks constructed for the image dataset. and They represent the first Zhang provides supporting images and their corresponding masks for category guidance information. and They represent the first Zhang is the query image to be segmented and its corresponding mask. , represents the total number of classes, each of which contains multiple images.

3. The test scenario oriented small sample segmentation method without mask supervision of claim 2, wherein, In step S1, the obtained mask is further processed, and all class target objects appearing in the support image and the query image at the same time are set as foreground, and the rest is set as background.

4. The test scenario oriented small sample segmentation method without mask supervision of claim 3, wherein, The deep neural network model comprises three components: a feature extractor, a mask generation module and a decoder, wherein the feature extractor is a pre-trained ResNet-50 using an ImageNet data set, and the mask generation module and the decoder are the parts to be trained.

5. The test scenario oriented small sample segmentation method without mask supervision of claim 1, wherein, In step S28, the cross-entropy loss function L is: wherein is an image at a location of a true label, is a model that outputs a prediction result for an image at a location 6. The test scenario oriented small sample segmentation method without mask supervision of claim 1-5, wherein, In step S3, the operation without mask supervision during inference is: during inference, for the image to be segmented, optionally one image belonging to the same category as the target to be segmented constitutes an image pair, which is input into the trained deep neural network model for prediction, and the last iteration produces a prediction mask of the query image to be segmented as the final segmentation prediction result.

Images