Multi-label image classification method based on multi-scale and cross-modal attention mechanism

Abstract

The invention discloses a multi-label image classification method based on a multi-scale and cross-modal attention mechanism. The method comprises the following steps: constructing a label graph, and learning label features through a graph convolutional neural network; obtaining a to-be-classified image, and extracting image features from a pre-trained convolutional neural network; constructing a classification model, and respectively inputting the obtained label features and image features into an MSML-GCN module and a GCN-SGA module for feature fusion calculation; fusing the obtained prediction results to obtain a final prediction label, and performing iterative training on the classification model by using a multi-label classification loss function to obtain a trained classification model; and inputting the extracted image features of the to-be-classified image into the trained classification model to obtain a multi-label image classification result. The technical problem that the image classification effect is poor due to the fact that an existing image classification method fully learns the dependency relationship between labels every year can be solved.

Description

technical field [0001] The invention relates to the field of multi-label image classification, in particular to a multi-label image classification method based on a multi-scale and cross-modal attention mechanism. Background technique [0002] Nowadays, Multi-label image classification (Multi-label image classification) has been widely used in the field of computer vision, including multi-target recognition, sentiment analysis, medical diagnosis recognition, etc. Since each image contains multiple objects, how to effectively learn the relationship between these objects and how to fuse these relationships with image features is still full of challenges. Regarding how to learn label features, the mainstream method is mainly through simple fully connected network learning and the graph neural network that has become popular in recent years. Fully connected network learning has a weak ability to represent label dependencies, while graph neural networks are not capable of label d...

AI Technical Summary

Problems solved by technology

[0005] 1. Since the image size of the input model is fixed, there will be restrictions on feature extraction

[0006] 2. In a single model, the fusion angle of image features and label features is single, and the utilization of label features and image features is not sufficient

[0007] For the establishment of the relationship between the image local area and the label feature, there is a shortcoming of insufficient label feature representation ability. At the same time, there are more ways to explore how to use the learned semantic attention (CN2020111001588: A cross-modal based State-of-the-art fast multi-label image classification method and system)

Images