A fine-grained image classification method based on information interaction

By constructing the CB-ViT model and combining CNN and ViT networks, information interaction between local and global features is achieved, which solves the shortcomings of local and global features in fine-grained image classification and improves detection accuracy and applicability.

CN118470370BActive Publication Date: 2026-11-17WUXI UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410310575.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-19
Publication Date
2026-11-17
Estimated Expiration
2044-03-19

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively distinguish images with similar appearances but belonging to different fine-grained categories. Furthermore, CNN networks excel at extracting local features but lack global information, while ViT modules are adept at capturing global information but lack local features, resulting in insufficient detection accuracy.

Method used

A CB-ViT model is constructed, which combines a CNN branch network and a ViT branch network. The feature enhancement modules FEM and LFE are used to realize the information interaction between local and global features. The downsampling module and upsampling module are used to perform dual-branch information fusion.

Benefits of technology

It improves the detection accuracy and versatility of fine-grained image classification, enhances the global perception capability of local features and the local details of global representation, and achieves high-precision fine-grained detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470370B_ABST
    Figure CN118470370B_ABST
Patent Text Reader

Abstract

This invention discloses a fine-grained image classification method based on information interaction, comprising: inputting an input image into the ResNet module of a CNN branch network and the LFE module of a CB-ViT model respectively; the ResNet module extracts features from the input image to obtain feature maps, and inputs the feature maps into a cascaded CNN module to obtain local features of the input image; the LFE module obtains low-level features of the input image, and inputs the low-level features of the input image into the ViT branch network to extract global features to obtain global image information of the input image; the patch embedding module converts the input image data to be classified into a sequence of feature maps, projects the feature map sequence into a high-dimensional space through linear mapping, adds a classification label vector for the final output probability distribution, and adds a learnable position encoding matrix for additional position information; the Transformer Encoder module combines the classification label vector with a linear transformation and activation function to obtain the fine-grained classification of the input image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a fine-grained image classification method based on information interaction. Background Technology

[0002] Fine-grained image classification, in the field of computer vision, refers to the task of classifying images with similar appearances but belonging to different subcategories. In fine-grained image classification, the appearance features between different fine-grained categories are very similar, with small differences, characterized by small inter-class differences and large intra-class differences. Traditional image classification methods struggle to accurately distinguish these different fine-grained categories. Therefore, computer vision techniques are used to extract more discriminative features to better differentiate fine-grained categories, such as beak shape and feather color in birds. These feature differences are then observed and analyzed more carefully. However, fine-grained classification faces challenges such as high appearance similarity, large feature dimensions, and difficult data annotation. The fine-grained image classification problem challenges the theories of feature representation, feature discrimination, and pattern recognition in the field of computer vision. Researching new algorithms and networks to address these challenges will advance the theoretical development of computer vision. In-depth research into fine-grained image classification can improve the computer's ability to recognize small objects or object subclasses, improving the accuracy and precision of image classification. This will help enhance the performance of computer vision systems in real-world scenarios, bringing more convenience and benefits to people's lives and work.

[0003] Chinese patent disclosure "A Fine-Grained Clothing Retrieval Method Based on CNN-Transformer Dual-Stream Network" (CN202211014352.3). This method inputs an image into a CNN branch network to extract features; maps clothing attributes to feature vectors, guiding the attention module to extract coarse-grained image features related to the attributes, where the generated attention matrix is ​​used to locate local regions in the image related to the attributes; inputs the local regions in the image and clothing attributes into a Transformer network to extract fine-grained image features; fuses the features from the two branches to obtain a robust representation of the clothing image and uses similarity calculation for clothing retrieval; introduces a DilatedTransformer variant based on the original Transformer to reduce network computation and accelerate training and inference speed. This invention uses a novel CNN-Transformer dual-stream structure, leveraging the complementarity of the two network structures to perform coarse-to-fine feature representation of clothing images, and finally fuses coarse-grained and fine-grained features to achieve high-performance retrieval. This method proposes a CNN-Transformer dual-stream network for fine-grained clothing retrieval. However, this method and device are not fully applicable to fine-grained detection of other types of images. The applicability and versatility of this method are insufficient when facing the fine-grained detection needs of different types of target images. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, this invention provides a fine-grained image classification method based on information interaction. The fine-grained image classification method based on information interaction proposed in this invention is suitable for testing needs involving fine-grained detection of different types of targets, and has high testing accuracy and strong versatility.

[0005] To achieve the above effects, the technical solution of the present invention is as follows: This invention provides a fine-grained image classification method based on information interaction, comprising the following steps: Step 1: Construct the CB-ViT model, which includes a CNN branch network and a ViT branch network; wherein, the CNN branch network includes a ResNet module and a cascaded CNN module connected in sequence, and the ViT branch network includes an LFE module and a cascaded ViT module connected in sequence; the cascaded CNN module is composed of n CNN modules connected in sequence, and the cascaded ViT module is composed of n ViT modules connected in sequence. Step 2: Input the input image into the ResNet module of the CNN branch network and the LFE module of the CB-ViT model respectively. The ResNet module extracts features from the input image to obtain feature maps, and then inputs the feature maps into the cascaded CNN module to obtain the local features of the input image. Step 3: Obtain low-level features of the input image through the LFE module, and input the low-level features of the input image into the ViT branch network for global feature extraction to obtain global image information of the input image; In the CB-ViT model, the ViT module includes a patch embedding module and a Transformer Encoder module connected in sequence; it continuously feeds the global image information of the input image in the ViT branch network into the CNN branch network; at the same time, it gradually feeds the local features of the input image in the CNN branch network into the patch embedding module in the ViT module. Step 4: The patch embedding module transforms the input image data to be classified into a sequence of feature maps, projects the sequence of feature maps into a high-dimensional space through linear mapping, adds a classification label vector for the final output probability distribution, and adds a learnable position encoding matrix for additional positional information. The Transformer Encoder module uses a combination of linear transformation and activation function to obtain a fine-grained classification of the input image by transforming the classification label vector.

[0006] This invention addresses fine-grained detection of different targets in various images. To improve the detection accuracy of fine-grained image classification, it fully leverages the potential of CNNs (CNNs) to extract local features and ViT (ViT) to extract global features. A bilinear interaction network of CNN and ViT is proposed, where CNN and ViT branches are run in parallel to maximize the preservation of both local and global representations. Furthermore, a downsampling (Down) module and an upsampling (UP) module are used to interconnect the two networks, fusing the local features from the CNN branch with the global representation from the ViT branch. To avoid the influence of background noise and the inability of low-level features to learn low-level image features, feature enhancement modules (FEM and LFE) are proposed. The algorithm demonstrates effectiveness and good generalization ability.

[0007] The method of this invention can be flexibly applied to fine-grained detection of different types of image targets, and has the characteristics of strong versatility and high detection accuracy.

[0008] Furthermore, the CNN module includes a 1×1 downward projection convolution module, a 3×3 spatial convolution module, a feature enhancement module (FEM), a 1×1 upward projection convolution module, and a residual connection between the input and output, wherein the convolution module consists of a convolutional layer (Conv2d), layer normalization, and a ReLU activation function.

[0009] Furthermore, the LFE module includes five convolutional layers. The first four convolutional layers use a kernel of 3, a stride of 2, and padding of 2, and incorporate ReLU and batch normalization operations. The fifth convolutional layer is a kernel of 1, a stride of 1, and padding of 2.

[0010] Furthermore, step 2 also includes: A Feature Enhancement Module (FEM) is added to the CNN module to enhance the features of the input feature map.

[0011] Furthermore, the addition of a Feature Enhancement Module (FEM) to the CNN module to enhance the input feature map includes: A Feature Enhancement Module (FEM) is added to the CNN module. The FEM enhances the input feature map. The first output feature map is obtained by performing convolution operations using convolution kernels of size 3×3 and 5×5 respectively. Second output feature map After each convolutional operation, batch normalization and SiLU activation are performed sequentially; the first output feature map is then processed. Second output feature map Perform element-wise addition to obtain the third output feature map. : (1) (2) (3) In the formula, Indicates batch normalization. Indicates the SiLU activation function; The third output feature map is processed using max pooling and average pooling operations. After downsampling, a concatenation operation is performed to obtain the fourth output feature map. Then, apply a 3×3 convolution kernel to the fourth output feature map. After integration, the fifth output feature map is obtained. Fifth output feature map A new feature vector is obtained after being input into the fully connected layer of the Feature Enhancement Module (FEM). : (4) (5) (6) In the formula, This indicates a max pooling operation. This indicates the average pooling operation. Indicates a fully connected operation; New eigenvectors The softmax function preset by the Feature Enhancement Module (FEM) is used to calculate the kernel weights of different convolutional layers and then compared with the first output feature map. Second output feature map Perform weighted fusion to obtain the first weighted feature map. Second weighted feature map Then, for the first weighted feature map... Second weighted feature map The final feature map is obtained by adding elements one by one. : (7) In the formula, γ represents the convolution kernel weight.

[0012] Furthermore, step 3, obtaining the global image information of the input image, includes: The ViT module divides the input image into fixed-size image patches; it then converts each image patch into a classification label vector using a linear mapping; and to incorporate positional information, it uses a positional encoding matrix to embed the position of each image patch into the classification label vector. The patch embedding is concatenated with the position encoding matrix to form the input embedding matrix, which is then passed as input to the Transformer Encoder module. The Transformer Encoder module captures the relationship between global image information and image patches. The Transformer Encoder module obtains global image information through position-aware pooling operations.

[0013] Furthermore, the Transformer Encoder module described in step 3 consists of Multi-Head Attention and MLP Block. Each sub-layer uses Residual Connection, and each sub-layer ends with Layer Normalization. The operation of Multi-Head Attention is denoted as MSA, and the operation of MLP Block is denoted as Multilayer Perceptron (MLP). Multi-Head Attention refers to mapping parameters to different subspaces, performing attention calculations separately, and finally concatenating the results.

[0014] Furthermore, step 3 feeds the global image information of the input image from the ViT branch network into the CNN branch network; simultaneously, it feeds the local features of the input image from the CNN branch network back to the patch embedding module in the ViT module, including: When the input image is fed into the ViT branch network, the feature map in the downsampling down module is aligned with the number of channels in the patch embedding module in the ViT module through a 1×1 convolution, and the spatial dimension alignment is completed by using downsampling down. Feature map mappings are added to the patch embedding module in the ViT module to obtain a feature map sequence, which is then input into the Transformer Encoder module to obtain global image information. When the global image information obtained from the ViT branch network is fed back to the CNN branch network, the global image information is upsampled (UP) to align the spatial scale. Upsampling (UP) aligns the channel dimension with the dimension of the feature map in the CNN branch network through average pooling, reshaping, 1×1 convolution, and layer normalization. The global information obtained from the ViT branch network is fed back to the cascaded CNN module, which, together with the feature enhancement module FEM, realizes the information interaction between local and global information.

[0015] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention aims to improve the detection accuracy of fine-grained image classification, addressing the issues of CNN networks' strength in extracting local image features but their inability to fully capture global image information, and ViT modules' strength in capturing global image information but their lack of ability to model local information. It proposes a fine-grained image classification method based on information interaction, combining CNN and ViT modules, which possesses good applicability and versatility.

[0016] This invention integrates CNN and ViT in parallel to maximize the preservation of local features and global representations of the input image information. It proposes Feature Enhancement Module (FEM) and Feature Enhancement Module (LFE) to address the issues of loss of detail due to background noise when directly inputting images into the parallel network, and the inability of ViT to learn low-level image features. Furthermore, it utilizes a two-branch information interaction mechanism (Downsampling and Upsampling modules) to significantly enhance the global perception capability of local features and the local details of the global representation through a fusion process. Attached Figure Description

[0017] Figure 1 This is a flowchart of the CB-ViT model of the present invention; Figure 2 This is a network structure diagram of the ViT of this invention; Figure 3 This is a network structure diagram of the LFE of this invention; Figure 4 This is the CNN branch network structure of the present invention; Figure 5This is a structural diagram of the feature enhancement module FEM of the present invention; Figure 6 This is the CB-ViT model network structure of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0019] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0020] Example This embodiment proposes a fine-grained image classification method based on information interaction. Please refer to [link / reference]. Figure 1 and Figure 6 This includes the following steps: Step 1: Construct the CB-ViT model, which includes a CNN branch network and a ViT branch network; wherein, the CNN branch network includes a ResNet module and a cascaded CNN module connected in sequence, and the ViT branch network includes an LFE module and a cascaded ViT module connected in sequence; the cascaded CNN module is composed of n CNN modules connected in sequence, and the cascaded ViT module is composed of n ViT modules connected in sequence. Step 2: Input the input image into the ResNet module of the CNN branch network and the LFE module of the CB-ViT model respectively. The ResNet module extracts features from the input image to obtain feature maps, and inputs the feature maps into the cascaded CNN module to obtain local features of the input image, enhance the ability of the CNN module to extract global image information, and reduce the loss of feature information. Step 3: Obtain low-level features of the input image through the LFE module, and input the low-level features of the input image into the ViT branch network for global feature extraction to obtain global image information of the input image; In the CB-ViT model, the ViT module includes a patch embedding module and a Transformer Encoder module connected in sequence. Global image information of the input image in the ViT branch network is continuously fed into the CNN branch network to enhance the global perception capability of the CNN. At the same time, local features of the input image in the CNN branch network are gradually fed back into the patch embedding module in the ViT module to enrich the local details of the ViT branch network. This process forms information interaction. The feature dimensions of the CNN module and the ViT module are different during the interaction. The feature mapping dimension of the CNN module is C × H × W, where C, H, and W are the channel, height, and width, respectively. The shape of the patch embedding in the ViT module is (K + 1) × E, where K, 1, and E represent the number of patch embeddings in the input image, the class label, and the embedding dimension, respectively. Step 4: The patch embedding module transforms the input image data to be classified into a sequence of feature maps, projects the sequence of feature maps into a high-dimensional space through linear mapping, adds a classification label vector for the final output probability distribution, and adds a learnable position encoding matrix for additional positional information. The Transformer Encoder module uses a combination of linear transformation and activation function to obtain a fine-grained classification of the input image by transforming the classification label vector.

[0021] In this invention, the CB-ViT model (CNN and ViT bilinear interactive network) is a bilinear information interaction combining CNN branch networks and ViT branch networks. Its purpose is to fully utilize the advantages of CNN and ViT to improve the detection accuracy of fine-grained image classification; for example... Figure 1 The diagram shown is the overall network flowchart. The ResNet module can preferably be ResNet50. The classification label vector is a one-dimensional vector. The activation function can be: .

[0022] To improve the detection accuracy of fine-grained image classification, this invention fully leverages the potential of CNNs (Convolutional Neural Networks) to extract local features and ViTs (Vision Transformers) to extract global features. A bilinear interactive network of CNN and ViT is proposed, which maximizes the preservation of both local features and global representations through parallel operation of CNN and ViT branches. Downsampling and upsampling are used to interactively connect the CNN and ViT branches, fusing the local features from the CNN branches with the global representations from the ViT branches. To avoid the influence of background noise and the inability of low-level features to learn low-level image features, a feature enhancement module (FEM) and a feature enhancement module (LFE) are proposed. The proposed method exhibits good effectiveness and generalization.

[0023] As a preferred technical solution, in this embodiment, the CNN module includes: Based on the definition in the ResNet module, the CNN module is divided into five stages, each consisting of multiple convolutional modules, including a 1×1 downward projection convolutional module, a 3×3 spatial convolutional module, a feature enhancement module (FEM), a 1×1 upward projection convolutional module, and a residual connection between the input and output. The convolutional modules consist of Conv2d convolutional layers, Layer Normalization, and ReLU activation functions. In the cascaded CNN module, the 3×3 convolutional kernels slide on overlapping feature maps, providing the ability to preserve fine local features of the input image.

[0024] As a preferred technical solution, in this embodiment, in the ViT module, the input image is convolved with a convolutional kernel of size p*p for patch labeling; where the stride is p, usually set to 16, which is the size of each patch; however, directly convolving with the convolutional kernel will result in the inability to learn the low-level features of the image. To address this issue, the LFE module (Low-level feature extraction) is proposed, and the detailed network structure is as follows: Figure 3 As shown, Figure 3 This is the result after LFE processing. The LFE module includes five convolutional layers. The first four convolutional layers use a kernel of 3, a stride of 2, and padding of 2, and incorporate ReLU and batch normalization operations. The fifth convolutional layer is a kernel of 1, a stride of 1, and padding of 2. This operation improves the performance and convergence speed of the ViT module.

[0025] As a preferred technical solution, in this embodiment, step 2 further includes: A Feature Enhancement Module (FEM) is added to the CNN module to enhance the features of the input feature map.

[0026] When the target to be classified occupies only a portion of the input image, its features may be affected by complex background information or noise. Traditional convolutional modules struggle to accurately capture the key features of objects, thus impacting the accuracy of classification results. To address these issues, CNN branch networks utilize ResNet modules for feature extraction and employ a Feature Enhancement Module (FEM) to enhance information interaction and representation capabilities, thereby acquiring more detailed features. The FEM network is shown below. Figure 5 As shown.

[0027] As a preferred technical solution, in this embodiment, the step of adding a Feature Enhancement Module (FEM) to the CNN module to enhance the features of the input feature map includes: A Feature Enhancement Module (FEM) is added to the CNN module. The FEM enhances the input feature map. The first output feature map is obtained by performing convolution operations using convolution kernels of size 3×3 and 5×5 respectively. Second output feature map After each convolutional operation, batch normalization and SiLU activation are performed sequentially; the first output feature map is then processed. Second output feature map Perform element-wise addition to obtain the third output feature map. : (1) (2) (3) In the formula, Indicates batch normalization. Indicates the SiLU activation function; The third output feature map is processed using max pooling and average pooling operations. After downsampling, a concatenation operation is performed to obtain the fourth output feature map. The max pooling operation is used to extract strong feature information from the feature map, enhancing the network's invariance and robustness. The average pooling operation, while acquiring features with global image information, reduces computation and parameter count, and helps prevent overfitting. A 3×3 convolution kernel is then used to convolve the fourth output feature map. After integration, the fifth output feature map is obtained. Fifth output feature map A new feature vector is obtained after being input into the fully connected layer of the Feature Enhancement Module (FEM). : (4) (5) (6) In the formula, This indicates a max pooling operation. This indicates the average pooling operation. Indicates a fully connected operation; New eigenvectors The softmax function preset by the Feature Enhancement Module (FEM) is used to calculate the kernel weights of different convolutional layers and then compared with the first output feature map. Second output feature map Perform weighted fusion to obtain the first weighted feature map. Second weighted feature map Then, for the first weighted feature map... Second weighted feature map The final feature map is obtained by adding elements one by one. : (7) In the formula, γ represents the convolution kernel weight.

[0028] The aforementioned Feature Enhancement Module (FEM) adaptively adjusts the receptive field size of the convolutional kernel, allowing selective focus on target regions in the image, effectively reducing feature information loss and optimizing the interaction of feature information within the context. The FEM module enhances the features of both the original and adversarial samples, converts the feature maps into vector form, and calculates the cosine similarity between the enhanced feature vectors as the loss function.

[0029] As a preferred technical solution, in this embodiment, step 3, obtaining the global image information of the input image, includes: The overall framework of the ViT module is as follows: Figure 2 As shown, the ViT module divides the input image into fixed-size image patches; each image patch is converted into a classification label vector through a linear mapping, usually achieved by flattening pixel values; in order to incorporate positional information, a positional encoding matrix is ​​used to embed the position of each image patch into the classification label vector; The patch embedding is concatenated with the positional encoding matrix to form the input embedding matrix, which is then passed as input to the Transformer Encoder module. The Transformer Encoder module captures the relationship between global image information and image patches. The Transformer Encoder module obtains global image information through position-aware pooling operations, which are typically performed on the output of an encoder layer.

[0030] As a preferred technical solution, in this embodiment, the Transformer Encoder module described in step 3 consists of Multi-Head Attention and an MLP Block. Each sub-layer uses ResidualConnection, and each sub-layer ends with Layer Normalization. The operation of Multi-Head Attention is denoted as MSA, and the operation of MLP Block is denoted as Multilayer Perceptron (MLP). Multi-Head Attention refers to mapping parameters to different subspaces, performing attention calculations separately, and finally concatenating the results.

[0031] MSA refers to the Multi-Head Attention operation, and MLP refers to the MLP Block operation. Since the depth of the Encoder is S, the downsampling process requires the input image to first undergo Layer Normalization, then MSA, then Layer Normalization, and finally MLP. This process is repeated S times. Finally, the classification label vector is extracted and Layer Normalization is performed to obtain the final result.

[0032] As a preferred technical solution, in this embodiment, step 3 feeds the global image information of the input image in the ViT branch network into the CNN branch network to enhance the global perception capability of the CNN; simultaneously, the local features of the input image in the CNN branch network are gradually fed back into the patch embedding module in the ViT module, including: When the input image is fed into the ViT branch network, the feature map in the downsampling down module is aligned with the number of channels in the patch embedding module in the ViT module through a 1×1 convolution, and the spatial dimension alignment is completed by using downsampling down. Feature map mappings are added to the patch embedding module in the ViT module to obtain a feature map sequence, which is then input into the Transformer Encoder module to obtain global image information. When the global image information obtained from the ViT branch network is fed back to the CNN branch network, the global image information is upsampled (UP) to align the spatial scale. Upsampling (UP) aligns the channel dimension with the dimension of the feature map in the CNN branch network through average pooling, reshaping, 1×1 convolution, and layer normalization. The global information obtained from the ViT branch network is fed back to the cascaded CNN module, which, together with the feature enhancement module FEM, realizes the information interaction between local and global information.

[0033] Implementing the embodiments of the present invention has the following beneficial effects: To utilize local features and global representations, this invention designs the CB-ViT model (a bilinear interaction network of CNN and ViT). Considering complementarity, in the CB-ViT model, the global context of the ViT branch is continuously fed into the feature map to enhance the global perception capability of the CNN branch. Similarly, the local features of the CNN branch network are gradually fed back into the patch embedding module in the ViT module to enrich the local details of the ViT branch network. This process constitutes the interaction. The CNN branch network and the ViT branch network are each composed of N (e.g., 12) repeated convolutions and ViT blocks. Downsampling and upsampling modules are used as connection modules. For the CNN branch, the input image undergoes initial feature extraction through the ResNet module, followed by convolution operations with a 1×1 downprojective convolution module and a 3×3 spatial convolution module. The feature enhancement module (FEM) strengthens information interaction, preserves fine-grained local features, and inputs them into the downsampling module (Down module). The number of channels is aligned through 1×1 convolution, and spatial dimension alignment is completed through average pooling reconstruction. The feature map is added to the patch embedding in the ViT branch network through the LayerNorm module. For the ViT branch network, the input image passes through the LFE module to obtain low-level features. Primary feature extraction is then performed on these low-level features to obtain image patches. These patches are fed into the Linear Projection of Flattened Patches layer to flatten the input image, i.e., converting the two-dimensional features of the input image into a one-dimensional ViT patch embedding through linear transformation. Since the patches have a specific order, a positional encoding matrix is ​​added to obtain the feature map sequence of the input image. This feature map sequence is then input into the Transformer Encoder module, where it is combined with local information fed into the CNN branch network through the downsampling module. The Transformer Encoder module's Multilayer Perceptron (MLP) classifies the data and outputs the final classification result. Simultaneously, the classification result is converted into two dimensions through the upsampling module and then aligned using a 1×1 convolution. Layer Normalization regularizes the dimensionally aligned features after the 1×1 convolution and inputs them into the CNN branch network. The global image information obtained by the ViT branch network is fed back into the CNN branch network, combining with the Feature Enhancement (FEM) module to achieve information interaction between local and global image information, ultimately achieving fine-grained classification of the input image.

[0034] This invention proposes using a Feature Enhancement Module (FEM) and a Feature Enhancement Module (LFE), and employs a two-branch information interaction mechanism through a Downsampling module and an Upsampling module. This significantly enhances the global perception capability of local features and the local details of the global representation. The improved method achieves an accuracy of 94.2%, exhibits good generalization on different datasets, and demonstrates stronger robustness to the optimizer.

[0035] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A fine-grained image classification method based on information interaction, characterized in that, Includes the following steps: Step 1: Construct the CB-ViT model, which includes a CNN branch network and a ViT branch network; wherein, the CNN branch network includes a ResNet module and a cascaded CNN module connected in sequence, and the ViT branch network includes an LFE module and a cascaded ViT module connected in sequence; the cascaded CNN module is composed of n CNN modules connected in sequence, and the cascaded ViT module is composed of n ViT modules connected in sequence. Step 2: Input the input image into the ResNet module of the CNN branch network and the LFE module of the CB-ViT model respectively. The ResNet module extracts features from the input image to obtain feature maps, and then inputs the feature maps into the cascaded CNN module to obtain the local features of the input image. Step 3: Obtain low-level features of the input image through the LFE module, and input the low-level features of the input image into the ViT branch network for global feature extraction to obtain global image information of the input image; In the CB-ViT model, the ViT module includes a patch embedding module and a TransformerEncoder module connected in sequence; it continuously feeds the global image information of the input image in the ViT branch network into the CNN branch network; at the same time, it gradually feeds the local features of the input image in the CNN branch network into the patch embedding module in the ViT module. Step 4: The patch embedding module transforms the input image data to be classified into a sequence of feature maps, projects the sequence of feature maps into a high-dimensional space through linear mapping, adds a classification label vector for the final output probability distribution, and adds a learnable position encoding matrix for additional positional information. The Transformer Encoder module uses a combination of linear transformation and activation function to obtain a fine-grained classification of the input image by transforming the classification label vector.

2. The method according to claim 1, characterized in that, The CNN module includes a 1×1 downward projection convolution module, a 3×3 spatial convolution module, a feature enhancement module (FEM), a 1×1 upward projection convolution module, and a residual connection between the input and output, which are connected in sequence. The convolution module consists of a convolutional layer (Conv2d), layer normalization, and a ReLU activation function.

3. The method according to claim 1, characterized in that, The LFE module includes five convolutional layers. The first four convolutional layers use a kernel of 3, a stride of 2, and padding of 2, and incorporate ReLU and batch normalization operations. The fifth convolutional layer is a kernel of 1, a stride of 1, and padding of 2.

4. The method according to claim 1, characterized in that, Step 2 also includes: A Feature Enhancement Module (FEM) is added to the CNN module to enhance the features of the input feature map.

5. The method according to claim 4, characterized in that, The addition of a Feature Enhancement Module (FEM) to the CNN module to enhance the input feature map includes: A Feature Enhancement Module (FEM) is added to the CNN module. The FEM performs convolution operations on the input feature map R using 3×3 and 5×5 kernels to obtain the first output feature map R1 and the second output feature map R2. After each convolution operation, batch normalization and SiLU activation are performed sequentially. Finally, the first output feature map R1 and the second output feature map R2 are added element-wise to obtain the third output feature map R. m : R1=σ(BN(Conv 3×3 (R))) (1) R2=σ(BN(Conv 5×5 (R))) (2) In the formula, BN(·) represents batch normalization, and σ(·) represents the SiLU activation function; The third output feature map R is processed using max pooling and average pooling operations. m After downsampling, a concatenation operation is performed to obtain the fourth output feature map R. n Then, apply a 3×3 convolution kernel to the fourth output feature map R. n After integration, the fifth output feature map R is obtained. p The fifth output feature map R p After being input into the pre-defined fully connected layer of the Feature Enhancement Module (FEM), a new feature vector M is obtained: R p =Conv 3×3 (R n ) (5) M=FC(R P ) (6) In the formula, This indicates a max pooling operation. FC(·) represents the average pooling operation, and FC(·) represents the fully connected operation. The new feature vector M calculates the kernel weights of different convolutional layers using the softmax function preset by the feature enhancement module FEM, and then performs weighted fusion with the first output feature map R1 and the second output feature map R2 respectively to obtain the first weighted feature map. Second weighted feature map Then, on the first weighted feature map Second weighted feature map The final feature map R is obtained by adding elements one by one. x : In the formula, γ represents the convolution kernel weight.

6. The method according to claim 1, characterized in that, Step 3, obtaining the global image information of the input image, includes: The ViT module divides the input image into fixed-size image patches; it then converts each image patch into a classification label vector using a linear mapping; and to incorporate positional information, it uses a positional encoding matrix to embed the position of each image patch into the classification label vector. The patch embedding is concatenated with the position encoding matrix to form the input embedding matrix, which is then passed as input to the TransformerEncoder module. The Transformer Encoder module captures the relationship between global image information and image patches. The Transformer Encoder module obtains global image information through position-aware pooling operations.

7. The method according to claim 1, characterized in that, The Transformer Encoder module described in step 3 consists of Multi-Head Attention and MLP Block. Each sub-layer uses Residual Connection, and each sub-layer ends with Layer Normalization. The operation of Multi-Head Attention is denoted as MSA, and the operation of MLP Block is denoted as Multilayer Perceptron (MLP). Multi-Head Attention refers to mapping parameters to different subspaces, performing attention calculations separately, and finally concatenating the results.

8. The method according to claim 2, characterized in that, Step 3 feeds the global image information of the input image from the ViT branch network into the CNN branch network; simultaneously, it feeds the local features of the input image from the CNN branch network back to the patch embedding module in the ViT module, including: When the input image is fed into the ViT branch network, the feature map in the downsampling down module is aligned with the number of channels in the patch embedding module in the ViT module through a 1×1 convolution, and the spatial dimension alignment is completed by using downsampling down. Feature map mappings are added to the patch embedding module in the ViT module to obtain a feature map sequence, which is then input into the Transformer Encoder module to obtain global image information. When the global image information obtained from the ViT branch network is fed back to the CNN branch network, the global image information is upsampled (UP) to align the spatial scale. Upsampling (UP) aligns the channel dimension with the dimension of the feature map in the CNN branch network through average pooling, reshaping, 1×1 convolution, and layer normalization. The global information obtained from the ViT branch network is fed back to the cascaded CNN module, which, together with the feature enhancement module FEM, realizes the information interaction between local and global information.

Citation Information

Patent Citations

  • A fine-grained clothing retrieval method based on CNN-Transformer dual-flow network

    CN115410067B

  • Crop disease identification method based on local selection and feature interaction

    CN116824366A

  • Communication signal classification method based on local feature window fusion ViT network

    CN116861163A