Multi-modal image classification method based on vision-table feature alignment

Abstract

The invention provides a multi-modal image classification method based on vision-table feature alignment. Firstly, in order to solve the problem that classification features and continuous features in heterogeneous table data are incompatible in a semantic space, a table encoder based on a Transform architecture is designed, and the table encoder captures a complex dependency relationship between the features by using a self-attention mechanism and provides structured semantic representation for cross-modal alignment; then, aiming at the semantic diversity of the image channel and the non-uniform distribution of the table attribute on the visual saliency, designing a multi-metric optimal transmission alignment module, and establishing a structured mapping relation between the image channel and the table attribute through an optimal transmission theory; and finally, designing a double-branch contrast learning module which enhances the consistency of cross-modal representation at a sample level through symmetric contrast loss and learns modal-invariant feature representation. According to the method, through collaborative optimization of the multi-task learning framework, the image classification performance can be remarkably improved, and the robustness and interpretability of the model are enhanced.

Description

Technical Field

[0001] This invention relates to image processing technology, specifically to a multimodal image classification method based on visual-table feature alignment. Background Technology

[0002] Image classification is a crucial task in computer vision, aiming to automatically identify and categorize objects and scenes in input images using algorithms. With the development of deep learning technology, Convolutional Neural Networks (CNNs) have achieved significant breakthroughs in image classification, becoming one of the most successful and widely used methods. Networks such as AlexNet, GoogLeNet, ResNet, and DenseNet are often used as backbone networks, autonomously learning image features through training data. However, most existing methods design complex deep neural networks through convolutional operations to extract and select key features, potentially neglecting domain information of the target region when processing images. To address this, many researchers have introduced attention mechanisms into image classification tasks, using them to focus on important features and suppress unimportant ones, thereby improving model performance. Networks such as VisionTransformer and Swin Transformer capture global dependencies through self-attention mechanisms, enhancing feature representation capabilities. Furthermore, to fully leverage the advantages of CNNs in local feature extraction and Transformers in capturing global dependencies, hybrid models combining CNNs and Transformers have emerged. These hybrid models can better utilize the powerful feature extraction capabilities of CNNs and the ability of Transformers to capture long-range dependencies. Hybrid Attention Networks, for example, combines convolutional neural networks and attention mechanisms, using attention modules at different levels to enhance the model's representational capabilities.

[0003] However, the aforementioned deep learning-based unimodal image classification methods rely solely on the feature representations of the images themselves, making them susceptible to the "black box" problem in practical applications. Even the designers of the network models often cannot explain the internal workings of the model or the reasons behind its specific decisions, as well as the features used to classify any given image. This lack of interpretability hinders the practical application and development of deep learning tools. Furthermore, with the rapid growth of data resources and types brought about by the era of big data, multimodal learning has achieved significant breakthroughs in image classification tasks. By jointly training with information such as images and text, visual language models can learn rich visual and semantic representations. However, due to the significant differences between the structured characteristics of tabular data and unstructured data such as images, existing cross-modal transfer methods are not suitable for tabular data.

[0004] To address the aforementioned problems in multimodal image classification, MMCL recently leveraged the high interpretability of tabular data, combining two leading contrastive learning strategies, Simplified and SCARF, to train a unimodal encoder using image and tabular data. Charms further utilized multimodal alignment to facilitate the transfer of expert knowledge from tables to images. While existing techniques have made substantial progress, semantic incompatibility exists between categorical and continuous features in heterogeneous tabular data, making traditional feature concatenation or embedding ineffective in capturing the complex dependencies between features. Secondly, image channels exhibit semantic diversity, and tabular attributes are unevenly distributed in terms of visual saliency; existing methods lack effective cross-modal deep alignment mechanisms. Finally, the lack of fine-grained modal consistency constraints at the instance level leads to insufficient cross-modal representation learning. Therefore, designing a multimodal image classification model based on visual-tabular feature alignment is of great significance. Summary of the Invention

[0005] The purpose of this invention is to address the problem of insufficient cross-modal alignment between visual and tabular features in multimodal image classification. It provides a multimodal image classification method based on visual-tabular feature alignment. Through the collaborative optimization of Transformer tabular encoder, multi-metric optimal transport alignment, and bi-branch contrastive learning, it achieves deep feature fusion and modal consistency learning of image and tabular data, thereby improving the accuracy and interpretability of image classification.

[0006] To achieve the above objectives, this invention provides a multimodal image classification method based on visual-table feature alignment, mainly comprising the following six parts: the first part is image and table preprocessing of the multimodal dataset; the second part is visual feature extraction; the third part is structured table semantic encoding; the fourth part is cross-modal multi-metric optimal transport alignment; the fifth part is instance-level representation consistency enhancement; and the sixth part is the training and classification inference of the multimodal image classification network model based on visual-table feature alignment, specifically: The first part is multimodal data preprocessing: Step 1: Obtain a multimodal dataset containing image and tabular data, and perform sample balancing and image sample size standardization. Step 2 involves data augmentation of the image samples processed in Step 1 to avoid overfitting. This includes random scaling and cropping, random horizontal flipping, and standardization using the mean and standard deviation of ImageNet. Step 3: Divide the attribute features of the table into continuous features and categorical features, and form image-table pairs with the image samples obtained in Step 2. Together, they constitute the final training set required. The second part is visual feature extraction: Step 4: Extract global image features F1 from the training set image samples I obtained in Step 2 using an image encoder; Step 5: Use the global image features F1 obtained in Step 4 as input, and input them into the image classification head H. cls And comparative learning projector H proj This yields the image classification output F1.1 and the image contrast features F1.2. The third part is the semantic encoding of structured tables: Step 6, use the table classification features T obtained in Step 3. cat and continuous features of the table T con As input, a Transformer-based table encoder is used to capture complex dependencies between features through a self-attention mechanism, generating a table representation F2 that preserves structural information between features; Step 7: Using the table representation F2 obtained in Step 6 as input, and applying the table classification header H... cls And comparative learning projector H proj This yields the table category features F2.1 and the table contrast features F2.2. Part Four is Cross-Modal Multi-Metric Optimal Transmission Alignment: Step 8: Using the image classification features F1.1 and table classification features F2.1 obtained in Steps 5 and 7 as input, the Multi-Metric Optimal Transmission Alignment Module (MOTA) is used to perform feature alignment between image channels and table attributes to obtain the mask matrix Mask with optimal cross-modal association. Step 9: Use the mask matrix Mask obtained in Step 8 to perform attribute-level hard mask semantic alignment on the image classification features F1.1 obtained in Step 5, and establish a cross-modal structured mapping. Part Five focuses on enhancing instance-level representation consistency: Step 10: The image contrast feature F1.2 and table contrast feature F2.2 obtained in Steps 5 and 7 are used as inputs and optimized using the symmetric contrastive learning loss function to align the multimodal representations of the same sample at the instance level, thereby enhancing cross-modal representation consistency. Part VI is about model training and classification reasoning: Step 11: A multi-task learning strategy that integrates image classification loss, table classification loss, attribute prediction loss and contrastive learning loss is adopted to collaboratively train the network and obtain an image classification pre-trained model. Step 12: Use the image classification pre-trained model obtained in Step 11 to perform inference on the test set images, and output the image category prediction and accuracy.

[0007] This invention presents a multimodal image classification method based on visual-table feature alignment. First, to address the semantic incompatibility between categorical and continuous features in heterogeneous tabular data, a table encoder based on the Transformer architecture is designed. This module utilizes a self-attention mechanism to capture complex dependencies between features, providing structured semantic representations for cross-modal alignment. Second, considering the semantic diversity of image channels and the non-uniform distribution of table attributes in terms of visual saliency, a multi-metric optimal transport alignment module is designed. This module establishes a structured mapping relationship between image channels and table attributes through optimal transport theory. Finally, a bi-branch contrastive learning module is designed. This module enhances the consistency of cross-modal representations at the sample level through symmetric contrastive loss, learning modality-invariant feature representations. This invention, through collaborative optimization using a multi-task learning framework, can significantly improve the performance of image classification and enhance the robustness and interpretability of the model. Attached Figure Description

[0008] Figure 1 This is a diagram of the overall network framework of the present invention; Figure 2 This invention relates to a table encoder based on the Transformer architecture. Figure 3 This is the multi-metric optimal transmission module of the present invention; Figure 4 Example images for the top five categories in terms of accuracy on the public test set; Figure 5 This is the confusion matrix of the top five categories with the highest accuracy on a public test set using the method of this invention. Detailed Implementation

[0009] To better understand this invention, the multimodal image classification method based on visual-table feature alignment of this invention will be described in more detail below with reference to preferred embodiments. In the following description, detailed descriptions of existing technologies may obscure the subject matter of this invention, and such descriptions will be omitted here.

[0010] Example: In a preferred embodiment of the present invention, the specific implementation steps are as follows: Step 1: Obtain the multimodal dataset. Preferably, the publicly available DVM car dataset is used. This dataset contains 1,451,784 car images and corresponding car parameter tables. The original image size is 300×300, and the table data has 16 attribute features. In order to achieve sample balance, car models with fewer than 700 samples are removed, and the image sample resolution is uniformly set to 224×224. Step 2 involves data augmentation of the image samples processed in Step 1 to avoid overfitting. This includes random scaling and cropping, random horizontal flipping, and standardization using the mean and standard deviation of ImageNet. Step 3: Divide the attribute features of the table. In this embodiment, it is divided into 13 continuous features and 3 classification features, which together with the image samples obtained in Step 2 form image-table pairs. The two together constitute the final training set samples required. Figure 1 This is a diagram of the overall network framework of the multimodal image classification method based on visual-table feature alignment of the present invention. In this embodiment, it is carried out according to the following steps: Step 4: Extract global image features F1 from the training set image samples I obtained in Step 2 using an image encoder. Preferably, the image encoder uses ResNet50 as the main framework. The specific implementation is as follows: Step 4.1, the image encoder as follows Figure 1 As shown in (a), the input image is a 224×224×3 image. The following layers are applied sequentially: an initial convolutional layer Conv1 with 64 7×7 convolutional kernels, a max pooling layer MaxPooling with a 3×3 pooling kernel, Layer1 consisting of 3 residual blocks with 256 output channels, Layer2 consisting of 4 residual blocks with 512 output channels, Layer3 consisting of 6 residual blocks with 1024 output channels, and Layer4 consisting of 3 residual blocks with 2048 output channels. The final output is a 7×7×2048 feature map. Step 4.2: Using the feature map obtained in Step 4.1 as input, global average pooling and fully connected layers are applied to obtain the global image feature F1. Step 5: Using the global image features F1 obtained in Step 4 as input, and employing the image classification head H... cls And comparative learning projector H proj This yields the image classification output F1.1 and the image contrast features F1.2. Step 6, use the table classification features T obtained in Step 3. cat and continuous features of the table T con As input, a table encoder based on the Transformer architecture is used to obtain a table representation F2 that preserves the structural information between features. The specific implementation is as follows; Step 6.1, the table encoder based on the Transformer architecture, as follows: Figure 2 As shown, the input table classification features T cat Discrete categories are mapped to dense vectors through embedding; simultaneously, the continuous features T of the input table are... conThe transformation to the same vector space is performed using linear projection; then, all feature embeddings are concatenated with learnable [CLS] tokens to obtain table embedding features T1. Step 6.2: Take the table embedding feature T1 obtained in step 6.1 as input, use the self-attention mechanism to capture the complex dependencies between table features, and then obtain the table representation F2 through pre-layer normalization and residual connection. Step 7: Using the table representation F2 obtained in Step 6.2 as input, and applying the table classification header H... cls And comparative learning projector H proj The table classification features F2.1 and table contrast features F2.2 are obtained. Step 8: Using the image classification features F1.1 and table classification features F2.1 obtained in steps 5 and 7 as input, the multi-metric optimal transmission alignment module MOTA is used to perform feature alignment between image channels and table attributes to obtain the mask matrix Mask with optimal cross-modal association. In this embodiment, the specific implementation of this module is as follows; Step 8.1, the multi-metric optimal transmission alignment module, as follows: Figure 3 As shown, the image classification feature F1.1 channel obtained in step 5 is C. By using K-Means clustering, similar image channels in F1.1 are grouped together to obtain fewer channels C1, so that each channel captures different aspects of the image features. Step 8.2: Using the image channel features F1.1.1 clustered in Step 8.1 and the table attribute features F2.1 obtained in Step 7 as input, calculate the Frobenius norm, cosine distance, and MMD distance on three dimensions: feature structure similarity, orientation consistency, and distribution matching degree, respectively, and construct the multi-metric cost matrix Cost; then, use the EMD algorithm in optimal transmission theory to calculate the optimal coupling matrix, and generate a mask matrix Mask for attribute-level feature selection based on this matrix; Step 9: Using F1.1 obtained in Step 5 and the mask matrix Mask obtained in Step 8 as input, the attribute-level semantic alignment is achieved by using a masking mechanism based on optimal transmission. Step 10: Using the contrastive features F1.2 and F2.2 obtained in Steps 5 and 7 as input, the symmetric InfoNCE loss function is used to equalize the multimodal representation of the same sample at the instance level; Step 11: Use a multi-task learning strategy to collaboratively train the network to obtain the final image classification pre-trained model. The specific training settings in this embodiment are as follows; Step 11.1: Input the training set samples from step 3 into the network from step 4 to step 10, and set the network hyperparameters: learning rate 0.001, early stopping strategy for the number of iterations, maximum number of iterations Max epochs 200, batch size 32, and SGD optimizer; Step 11.2, the total loss function of the network is Loss = L img + L tab + a·L attribute + b·L contra The image classification loss L img And table classification loss L tab Cross-entropy loss is used, and attribute prediction loss L is applied. attribute The mean of the mean squared error loss and the cross-entropy loss is L, and the two-way contrast loss is L. contra A symmetric InfoNCE loss is used, and then the network is trained to obtain the final image classification pre-trained model; Step 12: Input the public test set into the pre-trained model obtained in step 11. The network can then output the predicted image category and accuracy.

[0011] This invention addresses the challenge of heterogeneous alignment between visual and tabular features in multimodal data, presenting a multimodal image classification method based on visual-tabular feature alignment. First, a Transformer-based tabular encoder is employed at the feature encoding level. This module models the complex dependencies between categorical features and continuous features through a self-attention mechanism, effectively enhancing structured semantic representation capabilities. Second, to address the issues of semantic diversity in image channels and uneven distribution of visual saliency of tabular attributes, a multi-metric optimal transport alignment module is designed. This module integrates multiple similarity metrics to construct a cost matrix and establishes a structured mapping relationship between image channels and tabular attributes through optimal transport theory. Furthermore, to enhance the consistency of cross-modal representations, a dual-branch contrastive learning module is designed. This module achieves modality-invariant feature learning at the sample level through symmetric contrastive loss, improving the robustness of feature representation. A multi-task learning framework introduced into the network collaboratively optimizes image classification, attribute prediction, and contrastive learning objectives, using semantic alignment as a structural constraint for cross-modal learning, prompting the model to generate more discriminative fused features. This invention demonstrates excellent performance in cross-modal alignment and feature fusion, and is suitable for complex classification tasks under visual-tabular heterogeneous data.

[0012] Although specific embodiments of the present invention have been described above, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

Claims

1. A multimodal image classification method based on visual-table feature alignment, characterized in that, It includes the following six parts: The first part is multimodal data preprocessing: Step 1: Obtain a multimodal dataset containing image and tabular data, and perform sample balancing and image sample size standardization. Step 2 involves data augmentation of the image samples processed in Step 1 to avoid overfitting. This includes random scaling and cropping, random horizontal flipping, and standardization using the mean and standard deviation of ImageNet. Step 3: Divide the attribute features of the table into continuous features and categorical features, and form image-table pairs with the image samples obtained in Step 2. Together, they constitute the final training set required. The second part is visual feature extraction: Step 4: Extract global image features F1 from the training set image samples I obtained in Step 2 using an image encoder; Step 5: Use the global image features F1 obtained in Step 4 as input, and input them into the image classification head H. cls And comparative learning projector H proj The image classification features F1.1 and image contrast features F1.2 are obtained. The third part is the semantic encoding of structured tables: Step 6, use the table classification features T obtained in Step 3. cat and continuous features of the table T con As input, a Transformer-based table encoder is used to capture complex dependencies between features through a self-attention mechanism, generating a table representation F2 that preserves structural information between features; Step 7: Take the table representation F2 as input and use the table classification header H. cls And comparative learning projector H proj This yields the table category features F2.1 and the table contrast features F2.

2. Part Four is Cross-Modal Multi-Metric Optimal Transmission Alignment: Step 8: Using the image classification features F1.1 and table classification features F2.1 obtained in Steps 5 and 7 as input, the Multi-Metric Optimal Transmission Alignment Module (MOTA) is used to perform feature alignment between image channels and table attributes to obtain the mask matrix Mask with optimal cross-modal association. Step 9: Use the mask matrix Mask obtained in Step 8 to perform attribute-level hard mask semantic alignment on the image classification features F1.1 obtained in Step 5, and establish a cross-modal structured mapping. Part Five focuses on enhancing instance-level representation consistency: Step 10: The image contrast feature F1.2 and table contrast feature F2.2 obtained in Steps 5 and 7 are used as inputs and optimized using the symmetric contrastive learning loss function to align the multimodal representations of the same sample at the instance level, thereby enhancing cross-modal representation consistency. Part VI is about model training and classification reasoning: Step 11: A multi-task learning strategy that integrates image classification loss, table classification loss, attribute prediction loss and contrastive learning loss is adopted to collaboratively train the network and obtain an image classification pre-trained model. Step 12: Use the image classification pre-trained model obtained in step 11 to perform inference on the test set images, and output the image category prediction and accuracy.

2. The multimodal image classification method based on visual-table feature alignment according to claim 1, characterized in that, The image encoder in step 4 is a residual network-based coding structure.

3. The multimodal image classification method based on visual-table feature alignment according to claim 1, characterized in that, Step 6 uses a table encoder based on the Transformer architecture to obtain the structured semantic representation F2 of the heterogeneous table data. The specific implementation is as follows: (1) In the table encoder based on the Transformer architecture, the input table classification features T are processed. cat Discrete categories are mapped to dense vectors through embedding; simultaneously, the continuous features T of the input table are... con The transformation to the same vector space is performed using linear projection; then, all feature embeddings are concatenated with learnable [CLS] tokens to obtain table embedding features T1. (2) The table embedding feature T1 is used as input, and a self-attention mechanism is used to capture the complex dependencies between table features. Then, the table representation F2 is obtained through pre-layer normalization and residual connection.

4. The multimodal image classification method based on visual-table feature alignment according to claim 1, characterized in that, Step 8 uses a multi-metric optimal transport alignment module (MOTA) to obtain a mask matrix (Mask) that represents the optimal correlation between features of image channels and table attributes. The specific implementation is as follows: (1) In the multi-metric optimal transmission alignment module, the image classification feature F1.1 channel obtained in step 5 is C. By clustering, similar image channels in F1.1 are grouped together to obtain fewer channels C1, so that each channel captures different aspects of the image features. (2) The clustered image channel features F1.1.1 and the table attribute features F2.1 obtained in step 7 are used as inputs. By fusing multiple dimensions of structural similarity, directional consistency and distribution matching degree between features, a multi-metric cost matrix Cost is constructed. Then, the optimal transmission theory is used to solve the correlation between image channels and table attributes, and then a mask matrix Mask for attribute-level feature screening is generated.

5. The multimodal image classification method based on visual-table feature alignment according to claim 1, characterized in that, Step 11 employs a multi-task learning strategy, where the network's total loss function is composed of the image classification loss L. img Table classification loss L tab Attribute prediction loss L attribute and symmetric contrast loss L contra The weighted summation is used, i.e., Loss = L img + L tab + a·L attribute + b·L contra Among them, the symmetric contrast loss L contra The total loss function, calculated in step 10, is used to enhance cross-modal representation consistency. By jointly optimizing this total loss function, the multimodal alignment framework is collaboratively trained to obtain the final image classification pre-trained model.

Images