A fine-grained image classification method based on multi-modal learning

Abstract

The application discloses a fine-grained image classification method based on multi-modal learning, and first downloads original pictures of different species and corresponding additional information files from a known data set, and then uses the additional information files after preprocessing to train a neural network for extracting multi-modal features and fusion features and to converge, and then uses the converged neural network to perform label probability prediction on fine-grained images, and then performs decision correction on the prediction probabilities of the two neural networks, and finally outputs the category of the species in the image according to the correction result.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and more specifically, relates to a fine-grained image classification method based on multimodal learning. Background Technology

[0002] In recent years, image classification has become an important research direction in the field of computer vision and has been applied in many real-world scenarios such as autonomous driving and facial recognition. Research on image classification is mainly based on supervised learning methods, that is, using machine learning or deep learning methods to learn, classify, and recognize manually labeled datasets. Traditional image classification methods focus only on classifying highly distinguishable objects such as cats, dogs, and scenes. However, large categories of objects can still be subdivided into more subcategories. For example, in bird image classification, birds can be subdivided into different bird species, and traditional methods struggle to identify these fine-grained differences. Fine-grained image classification refers to the identification of fine-grained subclasses. Fine-grained datasets are characterized by large intra-class differences and small inter-class differences. For example, images of two species belonging to similar subcategories may look very similar, but species belonging to the same category may look different under different lighting conditions or in different poses. The challenge of fine-grained image classification lies in uncovering subtle visual differences between species with high visual similarity. Therefore, compared to traditional image classification, fine-grained image classification requires classification models to learn subtle features in images more efficiently.

[0003] Besides maximizing the model's ability to learn subtle differences between images of different categories, introducing image-related multimodal information to assist classification is also an effective way to improve the accuracy of fine-grained image classification. Some public datasets not only contain a large number of images but also information about the photographer at the time of taking the photos. For example, the latitude and longitude of the photo's location can reflect the habitat distribution of a species, and the photo's shooting time can reflect the species' activity period. Scientifically utilizing this data can distinguish between species that look very similar but have different habitats and habits, thereby improving the accuracy of fine-grained image classification. Authoritative publicly available datasets such as iNaturalist2018 and iNaturalist2021 not only contain a large number of images but also image-related multimodal information. This paper conducts a series of experiments based on these datasets to verify the effectiveness of the proposed method.

[0004] Currently, several methods apply multimodal information to fine-grained image classification. Kevin Tang et al. first introduced multimodal features into fine-grained image classification, extracting features such as age and date using an MLP (Multilayer Perceptron) network and concatenating them with image features to achieve category prediction. Oisin Mac Aodha et al. developed a geographic prior strategy to fine-tune predictions based on image features with geographic information. Grace Chu et al. used a geographic location prior model, a post-processing model, and a feature modulation model to utilize multimodal information. J. Christopher et al. used a multiplication strategy to integrate prediction results. These fine-grained image classification methods based on multimodal information have achieved excellent classification accuracy. However, they have shortcomings in multimodal data preprocessing methods, multimodal feature extraction methods, feature fusion methods, and decision-making strategies. This paper proposes an improved method based on learning from and referencing these methods. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fine-grained image classification method based on multimodal learning, which enables the effective integration of metropolitan area networks and access networks, while having scalable, highly flexible and low-cost performance.

[0006] To achieve the above-mentioned objectives, the present invention provides a fine-grained image classification method based on multimodal learning, characterized by comprising the following steps;

[0007] (1) Data collection;

[0008] Download the original images of different species and their corresponding supplementary information files from the known dataset. The supplementary information files contain the latitude information (lat), longitude information (lon), time information (data) of the original images when they were taken, as well as the species category labels in the images.

[0009] (2) Preprocessing of additional information files;

[0010] (2.1) Connect the latitude information lat and the longitude information lon to obtain the location information loc, and then perform nonlinear processing on the location information loc;

[0011] loc = concat(lat, lon)

[0012]

[0013] (2.2) Perform nonlinear processing on the time information data;

[0014]

[0015] (3) The nonlinearized position information and time information The spatiotemporal information X is obtained by passing each element through a trainable linear layer and then concatenating the layers. m :

[0016]

[0017] Where f(·) represents a trainable linear layer;

[0018] (4) Add multimodal information labels to the original image;

[0019] (4.1) Generate multimodal information tags;

[0020] Spatiotemporal information X m Directly used as multimodal information tags or using single-layer or multi-layer MLPs to analyze spatiotemporal information X m Preliminary feature extraction is then performed before it is used as a multimodal information label;

[0021] (4.2) Add multimodal information labels to the original image;

[0022] The multimodal information label is directly replaced with the pigment point at the corresponding size at the edge of the original image; or the multimodal information label is multiplied by the pigment point at the corresponding size at the edge of the original image.

[0023] (5) Extracting multimodal features Z by constructing a self-attention multilayer perceptron (SAMLP) network. m ;

[0024] The SAMLP network consists of four SAMLP modules connected in series. Each SAMLP module includes, in sequence, a self-attention mechanism module, a multilayer perceptron (MLP), and a feedforward network.

[0025] Spatiotemporal information X m The input is fed into a SAMLP network, and similarity features are extracted through a self-attention mechanism module. Then extract features using MLP Here, Relu() represents the activation function, and LN() represents layer regularization; then, features are extracted through a feedforward network. Where Dropou(t) represents random deactivation, and f1() and f2() both represent trainable linear layers; ultimately, the SAMLP network outputs multimodal features Z. m ;

[0026] (6) Extracting visual features Z i ;

[0027] The mix-up method is used to enhance the original image with excessive modal information labels, resulting in an enhanced new image. Then, the Res2Net network is used to extract the visual features Z of the new image. i ;

[0028] (7) Feature fusion;

[0029] (7.1) Utilizing multimodal features Z m For visual feature Z i Enhance attention;

[0030] Calculate multimodal features Z m With visual feature Z i The vector similarity is then used as a weight to evaluate the visual feature Z. i Enhance attention;

[0031] (7.2) Feature fusion of multi-cascaded dynamic MLP;

[0032] In a single dynamic MLP, multimodal features Z m With visual feature Z i Feature concatenation is performed, followed by dynamic filtering and resizing using different MLPs. Finally, the two resized features are multiplied to obtain a new visual feature Z. i This is used as the visual feature input for the next level of dynamic MLP, and so on, until the final dynamic MLP outputs the fused features.

[0033]

[0034] Among them, MLP 1 With MLP 2 This indicates two MLP networks with different output sizes, and concat() represents the feature concatenation process;

[0035] (8) Train the network until it converges;

[0036] (8.1) Calculate the predicted probabilities of multimodal features and fusion features corresponding to species category labels by using the normalized exponential function softmax;

[0037] (8.2) Calculate the loss function value;

[0038] loss = loss1 + loss2

[0039]

[0040]

[0041] Where loss1 is the loss value of the Res2Net network, loss2 is the loss value of the SAMLP network, λ is the label smoothing coefficient, K is the number of species categories, p represents the true label value of the input image, and q i,j To predict the probability of the j-th class by fusing features, q m,j Predict the probability of the j-th class for multimodal features;

[0042] (8.3) Repeatedly train until the loss function value converges;

[0043] (9) Real-time classification of fine-grained images;

[0044] (9.1) Process the image to be classified according to steps (2)-(8.1) to calculate the predicted probabilities of multimodal features and fusion features corresponding to species category labels;

[0045] (9.2) Select the top n maximum probability values ​​from the probabilities predicted by the fusion features, and then form a label set Pred1 = {pred1_k} corresponding to these n probability values; select the top n maximum probability values ​​from the probabilities predicted by the multimodal features, and then form a label set Pred2 = {pred2_k} corresponding to these n probability values, where k = 1, 2, ..., n;

[0046] (9.3) Calculate the total probability of the label: total_prob

[0047] Let the probabilities corresponding to labels pred1_k and pred2_k be pred1_k_prob and pred2_k_prob, respectively.

[0048]

[0049] Condition 1 states that if a tag exists in both tag set Pred1 and tag set Pred2, then the total probability of that tag, total_prob, is the sum of pred1_k_prob and pred2_k_prob. Condition 2 states that if a tag appears only in tag set Pred1 and is ranked first, then the total probability of that tag, total_prob, is the sum of the probability of it being ranked first in Pred1 and the probability of it being ranked last in Pred2. Condition 3 states that if a tag appears only in tag set Pred1 and is not ranked first, then the total probability of that tag, total_prob, is pred1_k_prob.

[0050] (9.4) If the first-ranked label in the label set Pred1 appears in Pred2, then the first-ranked label in Pred1 is directly used as the output species category; otherwise, total_prob is used as the final predicted probability, and then the corresponding species category is output.

[0051] The objective of this invention is achieved as follows:

[0052] This invention presents a fine-grained image classification method based on multimodal learning. First, it downloads original images of different species and corresponding supplementary information files from a known dataset. After preprocessing the supplementary information files, it trains and converges a neural network that extracts and fuses multimodal features. Then, it uses the converged neural network to predict the label probability of the corresponding fine-grained image. Next, it makes a decision correction on the predicted probabilities of the two neural networks. Finally, it outputs the category of the species in the image based on the correction result.

[0053] Furthermore, the fine-grained image classification method based on multimodal learning in this invention also has the following beneficial effects:

[0054] (1) This invention proposes a multimodal information preprocessing method, which solves the problems of complex multimodal information processing and different distributions, and provides a fixed and effective data preprocessing method.

[0055] (2) This invention proposes a multi-time period feature fusion method, which includes two parts: early-stage feature fusion and late-stage feature fusion. The multi-time period fusion method fuses multimodal features with visual features in high dimension, different time periods and different depths, and has excellent feature fusion effect.

[0056] (3) This invention proposes a SAMLP module to improve the ability to extract multimodal features. In existing methods, the feature extraction of multimodal information is mainly based on MLP, which has a simple structure and insufficient feature extraction capability. SAMLP establishes a correlation between multiple multimodal data inputs, enhances effective data based on the internal correlation of input vectors, and improves the utilization rate of multimodal data.

[0057] (4) This invention proposes a decision correction strategy that directly corrects the final predicted label based on the prediction results of multimodal features. The decision correction strategy does not overturn the original prediction results on a large scale, but directly intervenes in the final prediction results to a certain extent based on the prediction results of multimodal features. In the decision stage, the parts that cannot be fused by feature fusion are complemented to a certain extent. Attached Figure Description

[0058] Figure 1 This is a flowchart of the fine-grained image classification method based on multimodal learning of this invention;

[0059] Figure 2This is a schematic diagram of multimodal information preprocessing;

[0060] Figure 3 This is a diagram illustrating the addition of multimodal information tags;

[0061] Figure 4 This is a schematic diagram of the SAMLP structure;

[0062] Figure 5 It is a dynamic MLP structure diagram;

[0063] Figure 6 It is a multi-level dynamic MLP feature fusion structure diagram;

[0064] Figure 7 This is a diagram illustrating the addition of multimodal information tags. Detailed Implementation

[0065] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0066] Example

[0067] Figure 1 This is a flowchart of the fine-grained image classification method based on multimodal learning of this invention.

[0068] In this embodiment, as Figure 1 As shown, the present invention provides a fine-grained image classification method based on multimodal learning, comprising the following steps;

[0069] (1) Data collection;

[0070] Download the original images of different species and their corresponding supplementary information files from the known dataset. The supplementary information files contain the latitude information (lat), longitude information (lon), time information (data) of the original images when they were taken, as well as the species category labels in the images.

[0071] (2) Preprocessing of additional information files;

[0072] In multimodal information, longitude data is distributed in the range [0, 180], latitude data in the range [0, 90], and time data is in the format of year-month-day. Therefore, as shown in the example... Figure 2 As shown, we need to complete the normalization process first, then connect the latitude information lat and the longitude information lon to obtain the location information loc, and then perform nonlinear processing on the location information loc.

[0073] loc = concat(lat, lon)

[0074]

[0075] Finally, the time information data is processed to be non-linear;

[0076]

[0077] (3) The nonlinearized position information and time information The spatiotemporal information X is obtained by passing each element through a trainable linear layer and then concatenating the layers. m :

[0078]

[0079] Here, represents a trainable linear layer. This is because time information and location information have different degrees of influence on the model's prediction results, and time and space information should be adaptively weighted during fusion.

[0080] (4) Add multimodal information labels to the original image;

[0081] (4.1) Generate multimodal information tags;

[0082] Spatiotemporal information X m It can be used directly as a multimodal information label, or a single-layer or multi-layer MLP can be used to extract the initial features of spatiotemporal information before using it as a multimodal information label;

[0083] (4.2) Add multimodal information labels to the original image;

[0084] The multimodal information label is directly replaced with the pigment point at the corresponding size at the edge of the original image; or the multimodal information label is multiplied by the pigment point at the corresponding size at the edge of the original image.

[0085] (5) Extracting multimodal features Z by constructing a self-attention multilayer perceptron (SAMLP) network. m ;

[0086] In this embodiment, the impact of different multimodal information label generation methods, addition methods, and sizes on classification accuracy was tested. The optimal way to add multimodal information labels was to use a four-layer MLP to extract the initial features of the multimodal information and convert them to a size of 24*24. These features were then modulated with pigment point data values ​​from specific edge regions of the image. Too few MLP layers resulted in insufficient feature extraction capability, while too many layers led to structural redundancy and overfitting. Furthermore, labels that were too small contributed almost nothing to classification, while labels that were too large damaged the visual information of the original image.

[0087] Based on the above analysis, we construct a self-attention multilayer perceptron (SAMLP) network, as follows:

[0088] The SAMLP network consists of four SAMLP modules connected in series. Within each SAMLP module, such as... Figure 4 As shown, it includes, in sequence, a self-attention mechanism module, a multilayer perceptron (MLP), and a feedforward network;

[0089] Spatiotemporal information X m The input is fed into a SAMLP network, and similarity features are extracted through a self-attention mechanism module. Then extract features using MLP Here, Relu() represents the activation function, and LN() represents layer regularization; then, features are extracted through a feedforward network. Where Dropou(t) represents random deactivation, and f1() and f2() both represent trainable linear layers; ultimately, the SAMLP network outputs multimodal features Z. m ;

[0090] (6) Extracting visual features Z i ;

[0091] The mix-up method is used to enhance the original image with excessive modal information labels, resulting in an enhanced new image. Then, the Res2Net network is used to extract the visual features Z of the new image. i ;

[0092] (7) Feature fusion;

[0093] Current research primarily employs the following methods to fuse image features with multimodal information features:

[0094] Spatiotemporal prior strategy: Given the geographical location and time of an image, first estimate the probability that the object category in the image occurs at that location, and use it as the spatiotemporal prior probability.

[0095] Feature connection strategy: Based on the potential for mutual information complementarity between image features and multimodal spatiotemporal information features, concatenating image features and multimodal features before the classifier can combine the two features to generate better weights and predict the final category of the image.

[0096] In this invention, we effectively combine the two fusion methods in (1) and (2) to achieve a very good feature fusion effect.

[0097] (7.1) Utilizing multimodal features Z m For visual feature Z i Enhance attention;

[0098] Calculate multimodal features Z m With visual feature Z i The vector similarity is then used as a weight to evaluate the visual feature Z. i Enhance attention;

[0099] (7.2) Feature fusion of multi-cascaded dynamic MLP;

[0100] In a single dynamic MLP, such as Figure 5 As shown, the multimodal feature Z m With visual feature Z i Feature concatenation is performed, followed by dynamic filtering and resizing using different MLPs. Finally, the two resized features are multiplied to obtain a new visual feature Z. i This serves as the visual feature input for the next level of dynamic MLP, and so on, such as... Figure 6 As shown, the final fused features are output through the last stage of dynamic MLP;

[0101]

[0102] Among them, MLP 1 With MLP 2 This indicates two MLP networks with different output sizes, and concat() represents the feature concatenation process;

[0103] In this embodiment, as Figure 6 As shown, a multi-cascaded structure is used for later feature fusion, fusing image features and multimodal features at different levels. Its main function is to map features with similar image features but different multimodal features to different categories. While maintaining the advantages of dynamic MLP, the multi-cascaded fusion structure increases the number of rounds of feature fusion, enabling visual features and multimodal features to adaptively fuse at different depths and dimensions. Simultaneously, as the fusion structure deepens, MLP processing is applied to multimodal features, increasing their feature depth, reshaping their feature dimensions, and promoting fusion in the next stage.

[0104] (8) Train the network until it converges;

[0105] (8.1) Calculate the predicted probabilities of multimodal features and fusion features corresponding to species category labels by using the normalized exponential function softmax;

[0106] (8.2) Calculate the loss function value;

[0107] loss = loss1 + loss2

[0108]

[0109]

[0110] Where loss1 is the loss value of the Res2Net network, loss2 is the loss value of the SAMLP network, λ is the label smoothing coefficient, K is the number of species categories, p represents the true label value of the input image, and q i,j To predict the probability of the j-th class by fusing features, q m,j Predict the probability of the j-th class for multimodal features;

[0111] (8.3) Repeatedly train until the loss function value converges;

[0112] After the model is trained, the heatmap drawn using the Grad-Cam method can reflect the regions in the image that contribute to classification, thereby verifying the effectiveness of multimodal information labeling.

[0113] First, the output feature map of the last layer and the model's output class are obtained through forward propagation. The classes not mapped by the Softmax layer are then used for backpropagation to calculate the gradient of the last layer's output feature map. Based on the calculated gradient, the weight of each channel can be obtained. Finally, the calculated weights are visualized as a heatmap, with different colors representing regions of varying classification contributions. Therefore, comparing the heatmaps before and after adding multimodal information labeling verifies the effectiveness of the multimodal information labeling.

[0114] (9) Real-time classification of fine-grained images;

[0115] (9.1) Process the image to be classified according to steps (2)-(8.1) to calculate the predicted probabilities of multimodal features and fusion features corresponding to species category labels;

[0116] Multimodal features and fused features are two types of features with different meanings. No matter how efficient the fusion method is, there will inevitably be differences between them that cannot be complemented. Therefore, using feature fusion methods alone to improve the accuracy of classification tasks has limitations. In determining the predicted category, this invention uses the category predicted by multimodal features to make certain corrections to the final prediction result. The specific process is as follows:

[0117] (9.2) Select the top n maximum probability values ​​from the probabilities predicted by the fusion features, and then form a label set Pred1 = {pred1_k} corresponding to these n probability values; select the top n maximum probability values ​​from the probabilities predicted by the multimodal features, and then form a label set Pred2 = {pred2_k} corresponding to these n probability values, where k = 1, 2, ..., n;

[0118] (9.3) Calculate the total probability of the label: total_prob

[0119] Let the probabilities corresponding to labels pred1_k and pred2_k be pred1_k_prob and pred2_k_prob, respectively.

[0120]

[0121] Condition 1 states that if a tag exists in both tag set Pred1 and tag set Pred2, then the total probability of that tag, total_prob, is the sum of pred1_k_prob and pred2_k_prob. Condition 2 states that if a tag appears only in tag set Pred1 and is ranked first, then the total probability of that tag, total_prob, is the sum of the probability of it being ranked first in Pred1 and the probability of it being ranked last in Pred2. Condition 3 states that if a tag appears only in tag set Pred1 and is not ranked first, then the total probability of that tag, total_prob, is pred1_k_prob.

[0122] (9.4) If the first-ranked label in the label set Pred1 appears in Pred2, then the first-ranked label in Pred1 is directly used as the output species category; otherwise, total_prob is used as the final predicted probability, and then the corresponding species category is output.

[0123] Experiment and Results Analysis

[0124] This embodiment uses the iNaturalist 2018 and iNaturalist 2021 datasets, two fine-grained datasets with multimodal information. Due to the large number of categories in the iNaturalist dataset, training and testing on limited devices is extremely time-consuming, hindering model debugging and improvement. Therefore, this embodiment uniformly samples the iNaturalist 2018 and iNaturalist 2021 datasets to obtain INAT18-1K and INAT21-1K, each with 1000 categories. A series of experiments were conducted using these two datasets to illustrate and analyze the effectiveness of the network components used in this invention. Finally, this embodiment compares the method of this invention with the Dynamic MLP method on the INAT2021_mini full dataset with 10,000 categories.

[0125] The experimental setup used ResNet-50, ResNet-101, and Res2Net-101 as the backbone networks for image feature extraction. To ensure fair comparison of results, the different methods differed only in the processing and fusion of multimodal features. In terms of configuration, this embodiment used a 3090ti GPU, with a batch size of 32, an initial learning rate of 0.001, and trained using a stochastic gradient descent algorithm for 90 training epochs. Finally, the experiment consistently employed mix-up data augmentation.

[0126] (1) Early Feature Fusion Experiment

[0127] This paper achieves early feature fusion by adding multimodal information markers to the initial image. This section presents a series of experiments on the size setting, generation method, and addition method of the multimodal information markers. First, the input image is uniformly cropped to 224*224, and multimodal information markers are added to the edge regions of the image. This paper only tests three sizes: 12, 24, and 48, to avoid larger markers causing serious interference to the image. Regarding the marker generation method, one approach is to use MLP structures with different layers to extract features and resize multimodal spatiotemporal information to obtain multimodal information markers; another approach is to directly use multimodal information data as the data for multimodal information markers. For the addition method, the direct replacement method first adjusts the size of the preprocessed multimodal information and then directly replaces the pigment point data in the target region; the modulation method adjusts the size of the preprocessed multimodal information and then performs modulation operations with the pigment point data in the target region. The experimental results of the size setting, generation method, and addition method of the multimodal information markers are shown in Table 1.

[0128]

[0129]

[0130] Table 1 Experimental Table for Adding Multimodal Spatiotemporal Markers

[0131] In this experiment, the baseline network used for comparison was a Dynamic MLP network with a Res2Net101 backbone, and the dataset was INAT21-1K. Experimental results show that the best label addition method is to use a 4-layer MLP structure to extract multimodal information features, convert the label size to 24*24, and then add it to the edge regions of the image through modulation operations. Choosing an appropriate method to add multimodal information labels and injecting multimodal features into the original image can achieve early feature fusion, improving the fine-grained image classification accuracy by 0.39% compared to the baseline network.

[0132] To explore the principles of early fusion, this paper uses the GRAD-CAM method to create a heatmap of the feature maps in the last layer of a visual feature extraction network. For example... Figure 7 As shown, this species is PANCHLORA NIVEA (green banana cockroach). The original image is shown below. Figure 7 As shown in (a), without the addition of multimodal information labels, as shown in (b), the colored parts of the heatmap are accurately distributed on the insect body, while the remaining dark areas are areas that contribute no or no contribution. After adding multimodal information labels, as shown in (c), a color-changing effect is formed in the lower left corner of the heatmap, indicating that this area has a positive contribution to the final prediction result. This further shows that the addition of multimodal information labels transforms some noise information into information that is helpful for classification.

[0133] (2) Ablation test

[0134] In this embodiment, the ablation experiment uses methods including early feature fusion and the following series of methods:

[0135] Preprocessing of multimodal information. Before inputting multimodal information, this embodiment first separates information with different time and geographical locations. Then, trainable weights are assigned to each type of information. As the weights are trained and changed, the model's utilization of this information is improved. Simultaneously, these normalization processes provide a fixed format and method for adding other types of multimodal information, facilitating subsequent research.

[0136] This invention utilizes SAMLP to extract multimodal information features. It improves upon a multi-layer MLP structure based on a self-attention mechanism by adding a feedforward network at the end to form a SAMLP module. The SAMLP module exhibits stronger feature extraction capabilities.

[0137] Attention-based feature enhancement. Inspired by image question answering research, this approach enhances spatiotemporally relevant regions in the feature map (especially regions with added multimodal information labels) by calculating the similarity between multimodal information feature vectors and partial image feature vectors.

[0138] Decision correction. The prediction results based on multimodal information features are matched with the prediction results based on fused features. Decision correction is performed according to the strategy designed in this invention, and the loss function is modified to ensure the convergence of the two prediction models, ultimately achieving complementarity between visual features and multimodal features.

[0139] The ablation experiment was conducted on the INAT21-1K dataset, using a Dynamic MLP network as the baseline. The results of the ablation experiment are shown in Table 2. The accuracy of the fine-grained classification task gradually improved with the use of a series of methods. Compared to the baseline, the comprehensive method incorporating all the improved methods achieved an accuracy improvement of 0.51%.

[0140]

[0141] Table 2 Ablation Experiment Results

[0142] (3) Comparative experiments with advanced methods

[0143] This embodiment reproduces ConcatNet, GeoawareNet, EnsembleNet, and Dynamic MLP networks based on existing papers or open-source code. Experiments were conducted using the INAT18-1K and INAT21-1K datasets for training and validation, comparing results with ResNet-50, ResNet-101, and Res2Net-101 as backbone networks. The baseline network used here is a fine-grained image classification model that does not utilize multimodal information, while "Ours" is a synthesis of all the methods proposed in this paper. After comparison, "Ours" achieved the best results. Experimental results are shown in Table 3.

[0144]

[0145]

[0146] Table 3 compares the methods with state-of-the-art methods on the INAT18-1K and INAT21-1K datasets;

[0147] Clearly, the use of multimodal data significantly improves classification accuracy. The results show that when the backbone network is the best-performing Res2Net101, our method achieves the greatest improvement compared to Dynamic MLP methods, reaching 0.47% and 0.51% on INAT18-1K and INAT21-1K, respectively. Obviously, a superior backbone network can unleash the potential of "Ours," fusing visual and multimodal information to achieve the best classification results. Therefore, for fine-grained image classification methods improved using multimodal information, the performance of visual feature extraction networks, multimodal feature extraction networks, and feature fusion methods is crucial.

[0148] To further demonstrate the effectiveness of the proposed method, this paper compares the accuracy and training time of "Ours" and Dynamic MLP methods on the INAT2021_mini all-class dataset, as shown in Tables 4 and 5. Using Res2Net101 as the backbone network, "Ours" shows a significant improvement in accuracy compared to Dynamic MLP. Meanwhile, the training time of "Ours" only increases by 0.3%, indicating that the increase in parameters caused by our method is negligible.

[0149] Backbone Method Accuracy (%) Res2Net-101 Dynamic MLP 83.99 Res2Net-101 Ours 84.56

[0150] Table 4 compares the accuracy of Dynamic MLP on the INAT2021_mini dataset;

[0151]

[0152]

[0153] Table 5 compares the training time of Dynamic MLP on the INAT2021_mini dataset;

[0154] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A fine-grained image classification method based on multi-modal learning, characterized in that, Comprising the following steps: (1) Data collection; Download the original pictures of different species and the corresponding additional information files from a known data set, and the additional information files contain the latitude information lat, longitude information lon, time information data, and the corresponding species category label in the picture when the original picture is taken; (2) Additional information file preprocessing; (2.1) Concatenate the latitude information lat and the longitude information lon to obtain the location information loc, and then perform nonlinear processing on the location information loc; loc = concat(lat, lon) (2.2) Perform nonlinear processing on the time information data; (3) the position information after nonlinear processing and time information respectively through the trainable linear layer and then spliced to obtain the space-time information X m : Wherein, represents a trainable linear layer; (4) Add multi-modal information labels to the original image; (4.1) Generate multi-modal information labels; spatiotemporal information X m directly as a multi-modal information label or after preliminary feature extraction of the spatiotemporal information using a single-layer or multi-layer MLP (4.2) Add multi-modal information labels to the original image; Directly replace the pigment points of the multi-modal information labels with the corresponding size at the edge of the original image; or multiply the multi-modal information labels with the pigment points of the corresponding size at the edge of the original image; (5), by building a self-attention multi-layer perception SAMLP network to extract multi-modal features Z m ; The SAMLP network is composed of four SAMLP modules connected in series, and each SAMLP module includes a self-attention mechanism module, a multi-layer perception MLP, and a feedforward network connected in series; Temporal-spatial information X m Input into the SAMLP network, and extract similarity features through the self-attention mechanism module And then extract features through the MLP Wherein, Relu() represents an activation function, and LN() represents layer normalization; then extract features through a feedforward network Wherein, Dropou(t) represents random inactivation, and f1(), f2() both represent trainable linear layers; finally, make the SAMLP network output multi-modal features Z m ; (6) extracting visual features Z i ; The original image with excessive modal information labels is enhanced by using a mixed data enhancement mix-up method to obtain a new enhanced image, and a Res2Net network is used to extract visual features Z of the new image i ; (7) Feature fusion; (7.1) using the multi-modal features Z m on the visual features Z i attention enhancement; Compute multimodal feature Z m Vector similarity with visual feature Z i Attention enhancement on visual feature Z i with similarity as weight; (7.2) Feature fusion of multi-cascade dynamic MLP; In a single dynamic MLP, the multi-modal feature Z m is concatenated with the visual feature Z i , and then respectively filtered and resized by different MLPs, and finally the two adjusted features are multiplied to obtain a new visual feature Z i , which is input as the visual feature of the next level dynamic MLP, and so on, and finally the fusion feature is output by the last level dynamic MLP. where MLP 1 and MLP 2 denotes the MLP network with different output sizes, and concat() denotes the feature concatenation process; (8) Train the network to convergence; (8.1) Calculate the prediction probability of the multi-modal feature and the fusion feature corresponding to the species category label respectively through the normalization exponential function softmax; (8.2) Calculate the loss function value loss; loss = loss1 + loss2 Wherein, loss1 is the loss value of the Res2Net network, loss2 is the loss value of the SAMLP network, λ is a label smoothing coefficient, K is the number of species categories, p represents the true label value of the input picture, q i,j is the probability of the jth category predicted by the fusion feature, q m,j is the probability of the jth category predicted by the multi-modal feature. (8.3) Through repeated training until the loss function value loss converges; (9) Real-time classification of fine-grained images; (9.1) Process the image to be classified according to steps (2)-(8.1), thereby calculating the prediction probability of the multi-modal feature and the fusion feature corresponding to the species category label; (9.2) Select the top n largest probability values in the predicted probability of the fusion feature, and then form a label set Pred1 = {pred1_k} from the n probability values corresponding to the labels pred1_k; Select the top n largest probability values in the predicted probability of the multi-modal feature, and then form a label set Pred2 = {pred2_k}, k = 1, 2, …, n from the n probability values corresponding to the labels pred2_k; (9.3) Calculate the comprehensive probability total_prob of the label: Let the probabilities corresponding to the labels pred1_k and pred2_k be pred1_k_prob and pred2_k_prob, respectively; Wherein, condition 1 refers to: if a label exists in both label set Pred1 and label set Pred2, then the comprehensive probability total_prob of the label is the sum of pred1_k_prob and pred2_k_prob; condition 2 refers to: if a label only exists in label set Pred1 and ranks first, then the comprehensive probability total_prob of the label is the sum of the probability value ranking first in Pred1 and the probability value ranking last in Pred2; condition 3 refers to: if a label only exists in label set Pred1 and does not rank first, then the comprehensive probability total_prob of the label is pred1_k_prob; (9.4), if the label ranking first in label set Pred1 exists in Pred2, then directly taking the label ranking first in Pred1 as the output species category; otherwise, taking total_prob as the final prediction probability; (9.5), outputting the corresponding species category according to the final prediction probability.

Images