A method for fine-grained scene image classification

Abstract

The present application belongs to the technical field of computer multimedia, and particularly relates to a fine-grained scene image classification method. The present application is based on a new multi-modal feature fusion method, also known as a modal-independent adapter MAA, which enables the model to adaptively learn the importance of different modalities under different conditions without pre-setting in the model architecture. Specifically, the present application is based on a multi-layer perceptron MLP to eliminate the differences in distribution of each modality, which eliminates the data distribution difference between different modalities, and then uses a modal-independent Transformer encoder to perform semantic-level feature fusion, and combines an average pooling operation and a full connection layer to realize fine-grained scene image classification. The method proposed in the present application can accurately realize fine-grained scene image classification, and the model can be well adapted to new modalities and further improve the performance.

Description

Technical Field

[0001] This invention relates to a method for fine-grained scene image classification, belonging to the field of computer multimedia technology. Background Technology

[0002] Unlike general image classification, fine-grained image classification tasks require models to perform more refined classification of images within a broad class, thus demanding stronger feature encoding capabilities. Within different broad classes, fine-grained scene image classification deals with scene images, which contain more information than images of individual objects. Since images from the same broad class are often visually very similar, fine-grained scene image classification models often use features from different modalities, including objects and text within the image, to leverage various types of information. Recent research has focused primarily on using better multimodal feature encoding to improve model performance.

[0003] After obtaining the aforementioned multimodal feature encodings, the model proceeds to multimodal feature fusion. Generally, most existing models design multimodal feature fusion methods based on prior intuition, assuming that global visual features are always the most discriminative and representative features among all modalities. For example, many methods use global visual features as query vectors in attention mechanisms or concatenate single global visual feature vectors with inference results from other modalities. However, in addition to global visual features, objects and text in the image are also very important, especially when the images have similar global layouts. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention proposes a fine-grained scene image classification method. This method utilizes a modality-agnostic adapter (MAA) for multimodal feature fusion, enabling the model to adaptively learn the importance of different modalities under varying conditions without requiring pre-setting in the model architecture. Specifically, this invention recognizes that the importance of different modal features varies under different conditions, and that there is no constant modality preference in fine-grained image classification tasks. Therefore, this invention eliminates the data distribution differences between different modalities and then uses a modality-agnostic Transformer encoder for semantic-level feature fusion. The proposed method can accurately achieve fine-grained scene image classification, and the model can adapt well to new modalities, further improving its performance.

[0005] The technical solution of the present invention is described in detail below.

[0006] This invention provides a method for fine-grained scene image classification, which employs a fine-grained scene image classification model. The fine-grained scene image classification model comprises a pre-trained model designed in parallel, several multi-layer perceptrons designed in parallel, a Transformer encoder, an average pooling layer, and a fully connected layer, all connected sequentially. The specific steps are as follows:

[0007] For a given input image, first use different pre-trained models to obtain multimodal feature encodings, denoted as:

[0008]

[0009] In the formula N m D represents the length of the feature sequence of a single modality. m The dimension of the feature is represented by m∈{G,L,T}, which represent global visual features, local visual features, and text features, respectively.

[0010] Next, multiple multilayer perceptrons (MLPs) are used to eliminate the differences in the distribution of various modalities, that is, to unify the dimension of the features, denoted as D. This results in modality-independent encodings. The input is fed into a modality-independent Transformer encoder for semantic-level fusion;

[0011] Then, given the feature sequence A single feature vector for each input image is obtained using average pooling.

[0012] Finally, the probability distribution of the classification results is obtained using a fully connected layer, thus achieving fine-grained scene image classification.

[0013] In this invention, the modal features extracted by different pre-trained models can be global visual features, local visual features, and text features, as well as other modal features.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] 1. This invention treats all modalities equally to achieve better multimodal feature fusion.

[0016] 2. This invention is applicable to various modal combinations, eliminating the need to design new fusion methods for each modal combination, and the performance of the model can be further improved by adding new modalities. Attached Figure Description

[0017] Figure 1 This is a flowchart of the fine-grained scene image classification method based on a modality-independent adapter according to the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0019] like Figure 1 As shown, this invention proposes a fine-grained scene image classification method based on a modality-independent adapter (MAA). The MAA is used to achieve multimodal feature fusion for fine-grained scene image classification, treating all modalities equally without setting any prior preferences in the model architecture. Specifically, we first eliminate the data distribution differences between different modalities, and then use a modality-independent Transformer encoder for feature fusion. In this way, the model can adaptively learn the importance of different modalities in different situations, rather than pre-setting them in the model architecture.

[0020] Specifically, given an input image First, different pre-trained models are used to obtain multimodal feature encodings, denoted as... Where N m D represents the length of the feature sequence of a single modality. m Let m ∈ {G, L, T} represent the dimension of the features, where m ∈ {G, L, T} represents global visual features, local visual features, and text features, respectively. Before using a modality-independent Transformer, the model first uses multiple Multilayer Perceptrons (MLPs) to eliminate the differences in distribution among the various modalities, i.e., to unify the dimension of the features, denoted as D. Thus, the resulting modality-independent encoding can be... The input is fed into a modality-independent Transformer encoder for semantic-level fusion. Finally, given the feature sequence... A single feature vector for each input image is obtained using average pooling. Finally, the probability distribution of the classification results is obtained using a fully connected layer. During model training, the cross-entropy loss function is used as shown in formula (1):

[0021]

[0022] Where y is the index of the target class, and C is the number of classes in the dataset. and These are the parameters of the fully connected layer. The following content is organized according to the three key points mentioned above: multimodal feature encoding, modality dissimilarity elimination, and modality-independent feature fusion:

[0023] I. Multimodal Feature Encoding

[0024] In this invention, ViT is used. [1] The output is used as a global visual encoding, using KnowBert. [2]The output is used as text encoding. For local visual encoding, this invention crops the top left, top right, bottom left, bottom right, and center portions of the original image and inputs these five cropped images into the ViT model to obtain the local visual encoding. It can be seen that the local image encoding branch is very simple because it does not involve additional parameters or another complex pre-trained model.

[0025] II. Elimination of Modal Differences

[0026] After obtaining multimodal feature encodings from different pre-trained models, this invention divides the multimodal feature fusion of the modality-independent adapter (MAA) into two steps: modality difference elimination and modality-independent feature fusion. Modality difference elimination will be introduced first.

[0027] On the one hand, since the current feature encodings belong to different modalities, the feature fusion process will be hindered if the distribution differences between them are not eliminated. In other words, the feature fusion module will focus more on modal differences than semantic relationships due to distribution differences, so it is unreasonable to directly perform feature fusion. On the other hand, feature encodings of different modalities may have different dimensions, so multiple MLPs are needed to eliminate dimensional differences. Based on the above two reasons, at this stage, the feature encodings of different modalities will generate modality-independent encodings through their respective MLPs, which can be expressed as formula (2):

[0028] Z′ m =Act(FC(LN(Z) m )))#(2)

[0029] Where Act, FC, and LN represent activation function, fully connected layer, and layer normalization, respectively. All feature encodings eventually have the same dimension, D.

[0030] III. Modality-independent feature fusion

[0031] The modality-independent encoding obtained above is fused with features by a modality-independent Transformer encoder, which can be expressed as formulas (3) and (4):

[0032] Z0 = Concat(Z′) G Z L ′,Z′ T )#(3)

[0033] Z i+1 =Transformer i (Z i )#(4)

[0034] Where i = 1, 2, ..., L represents the number of layers in the Transformer encoder, and L represents the total number of layers.

[0035] This invention does not use positional encoding in the Transformer encoder; instead, it uses modal encoding to convey information about the modal sources. As can be seen, the encoding order of different modalities in formula (2) does not affect the final calculation result after the average pooling operation, ensuring that different modalities have the same prior importance. Furthermore, since the differences in modal distribution have been eliminated in the previous step, the feature fusion operation can focus more on semantic-level relationships. In this way, this invention can adaptively learn the importance of different information under different circumstances.

[0036] Furthermore, this invention is adaptable to various modal combinations. If different modalities are treated differently based on prior intuition, then a completely new model must be redesigned whenever a new modality needs to be added, which is quite complex. In contrast, because this invention treats all modalities equally, it is easy to add new modalities and further improve the performance of the model, which is another benefit of this invention.

[0037] In a specific implementation, two datasets are used: a building dataset (Con-Text) and a crowd activity dataset (Crowd Activity). The building dataset... [3] This is a subset of the ImageNet dataset, further subdividing the "Building" and "Place of Business" categories into 28 finer-grained categories, such as bakeries, coffee shops, and pizza shops. The dataset contains 24,255 images, with 16,179 in the training set and 8,076 in the test set. It's worth noting that not all images in this dataset contain text; therefore, in addition to the text within the images, more information needs to be encoded to complete the fine-grained image classification task. (Crowd Activity Dataset) [4] The dataset contains 8,785 images across 21 categories, with 5,770 images in the training set and 3,015 in the test set. These categories encompass not only everyday life scenes such as Christmas celebrations, celebrity speeches, and graduation ceremonies, but also crowd parades. Unlike the building dataset, each image in this dataset includes text, and the text often incorporates significant background information.

[0038] The experimental results are shown in Table 1. This method achieves fine-grained classification of scene images, reaching a mean average precision (mAP) of 90.87% on the building dataset and 88.29% on the crowd activity dataset. It can be seen that adding new modalities consistently improves model performance. However, adding text encoding yields a more significant improvement compared to global and local visual encoding, which contradicts the prior assumptions of previous models. Furthermore, the relative importance of different modalities is inconsistent between the two datasets. For example, global visual encoding is slightly more important than local visual encoding for the building dataset, but this is not the case for the crowd activity dataset, possibly because the latter contains more local details. Additionally, the interactions between different modalities are quite complex. For instance, on the building dataset, using local visual encoding improves mAP by approximately 0.74% regardless of whether text encoding is used, while on the crowd activity dataset, local visual encoding only brings a small improvement when text encoding already provides sufficient information to the model. These observations support the claim of this invention that the importance of different modalities varies under different circumstances, and that the model should not have a specific modality preference. This is why the present invention enables the model to adaptively learn the importance of different modalities under different conditions without requiring prior settings in the model architecture.

[0039] Table 1 Experimental Results

[0040]

[0041]

[0042] References

[0043] [1]Dosovitskiy A,Beyer L,Kolesnikov A,et al.An image is worth 16x16words:Transformers for image recognition at scale[C] / / InternationalConference on Learning Representations.2021. [2]Peters M E,Neumann M,Logan IV R L,et al.Knowledge enhancedcontextual word representations[C] / / Proceedings of the 2019Conference onEmpirical Methods in Natural Language Processing and the 9th InternationalJoint Conference on Natural Language Processing.2019.

[0044] [3]Karaoglu S,van Gemert J C,Gevers T.Con-text:text detection usingbackground connectivity for fine-grained object classification[C] / / Proceedings of the 21st ACM international conference on Multimedia.2013:757-760.

[0045] [4]Wang H,Liao J,Cheng T,et al.Knowledge mining with scene text forfine-grained recognition[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2022:4624-4633。

Claims

1. A method for fine-grained scene image classification, characterized in that, It employs a fine-grained scene image classification model for fine-grained scene image classification. This model comprises a pre-trained model designed in parallel, several multi-layer perceptrons designed in parallel, a Transformer encoder, an average pooling layer, and a fully connected layer, all connected sequentially. The specific steps are as follows: For a given input image, first use different pre-trained models to obtain multimodal feature encodings, denoted as: In the formula, N m D represents the length of the feature sequence of a single modality. m The dimension of the feature is represented by m∈{G,L,T}, which represent global visual features, local visual features, and text features, respectively. Next, multiple multilayer perceptrons (MLPs) are used to eliminate the differences in the distribution of various modalities, that is, to unify the dimension of the features, denoted as D. This results in modality-independent encodings. The input is fed into a modality-independent Transformer encoder for semantic-level fusion; Then, given the feature sequence A single feature vector for each input image is obtained using average pooling. Finally, the probability distribution of the classification results is obtained using a fully connected layer, thus achieving fine-grained scene image classification.

2. The method for fine-grained scene image classification according to claim 1, characterized in that, The output of ViT is used as the global visual encoding, and the output of KnowBert is used as the text encoding. The top left, top right, bottom left, bottom right and center portions of the original image are cropped, and the five cropped images are input into the ViT model to obtain the local visual encoding.

3. The method for fine-grained scene image classification according to claim 1, characterized in that, The feature encodings of different modalities will generate modality-independent encodings through their respective MLPs, as shown in Equation (2): Z′ m =Act(FC(LN(Z m )))#(2) Where Act, FC, and LN represent activation function, fully connected layer, and layer normalization, respectively.

4. The method for fine-grained scene image classification according to claim 1, characterized in that, Feature fusion is performed using a modality-independent Transformer encoder, as expressed in equations (3) and (4): Z0=Concat(Z′ G ,Z′ L ,Z′ T )#(3) WITH i+1 =Transformer i (WITH i )#(4) Where i = 1, 2, ..., L represents the number of layers in the Transformer encoder, and L represents the total number of layers.

5. The method for fine-grained scene image classification according to claim 1, characterized in that, In the training of the fine-grained scene image classification model, the cross-entropy loss function shown in equation (1) is used: Where y is the index of the target class, and C is the number of classes in the dataset. and These are the parameters of the fully connected layer.

Images