A saliency-based multimodal small-shot learning method

Abstract

The invention discloses a saliency-based multi-modal small-sample learning method, which includes two parts: multi-modal combination and label propagation, specifically as follows: in the process of multi-modal combination, the pre-trained saliency detection network is first used to support The saliency map is extracted from the sample image of the sample image, and the foreground and background of the sample image are separated; secondly, the semantic word embedding of the foreground and background regions of the sample image of the support set is obtained through the GloVe model, which is used as semantic information to assist visual information classification; finally, for the obtained support set The foreground, background and semantic information of the sample image are adaptively combined through the modal mixing mechanism to obtain the sample feature representation with multi-modal information; The sample is constructed according to the K-nearest neighbor method; finally, the category of the unlabeled query set sample is predicted through the labeled support set sample.

Description

technical field [0001] The invention relates to a small-sample image classification method, in particular to a small-sample learning method based on salient multimodal data processing. Background technique [0002] With the introduction of convolutional neural networks, deep learning has made breakthroughs in image classification, speech recognition, and object detection. However, these studies usually require a large amount of labeled data for training, such as ImageNet, etc., but in reality In daily life, it is very difficult to obtain a large amount of data, such as photos of endangered species, medical images, etc., which seriously limits the applicability of the model in the real world, and the labeling of images will also consume a lot of manpower and financial resources. It can identify a new object with a very small number of samples, use previously learned knowledge to help quickly learn new content, and integrate new concepts into existing concept networks to learn...

AI Technical Summary

Problems solved by technology

[0002] With the introduction of convolutional neural networks, deep learning has made breakthroughs in image classification, speech recognition, and object detection. However, these studies usually require a large amount of labeled data for training, such as ImageNet, etc., but in reality In daily life, it is very difficult to obtain a large amount of data, such as photos of endangered species, medical images, etc., which seriously limits the applicability of the model in the real world, and the labeling of images will also consume a lot of manpower and financial resources. Able to recognize a new object through a very small number of samples, help to quickly learn new content through previously learned knowledge, and integrate new concepts into existing concept networks to learn new concepts

Images