Transformer-based image classification model and iterative method thereof

Abstract

The application provides a transformer-based image classification model and an iteration method thereof, and comprises an image input module, an image classification module, a result output module, wherein the image input module is used for acquiring images; the image classification module comprises a ConvMAE network, a multilayer perceptron task head and a support vector machine task head; the image classification module is used for analyzing and processing the input images to obtain classification results; and the result output module is used for outputting the classification results. By introducing the ConvMAE network into the image classification module, important features in the images can be automatically learned, and effective feature representation and expression can be performed; the ConvMAE network is quickly fine-tuned by using the multilayer perceptron task head and the support vector machine task head, so that the complexity of the input data is reduced, the fine-tuning speed is improved, the classification accuracy is improved, and the problems of slow iteration speed and poor training effect of the existing transformer-based model are solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image classification model based on transformer and its iterative method. Background Technology

[0002] With the development of computer technology, transformers are now commonly used to extract key information from images and encode it into fixed-dimensional vectors, which are then passed to a classification task for image classification. To ensure the accuracy of the classification results, data fine-tuning is necessary.

[0003] However, once the fine-tuning data reaches a certain scale, although the main structural parameters of the transformer are frozen, gradient backpropagation is still required, which lengthens the transformer fine-tuning cycle. Furthermore, fine-tuning classification networks can lead to deteriorating results, as the network forgets historical features, resulting in poor performance metrics.

[0004] Furthermore, with the advent of the era of large models, large models require more computing power and high-quality data for training, thus necessitating longer training cycles. In addition, the diversity of data (such as images, text, audio, etc.) leads to differences and correlations between data, and their feature representations have high dimensionality and complexity, posing challenges to model training and inference. Summary of the Invention

[0005] The purpose of this invention is to provide a transformer-based image classification model and its iterative method to solve the problems of slow iteration speed and poor training effect of existing transformer-based models.

[0006] To address the aforementioned technical problems, this invention provides an image classification model based on transformers, comprising:

[0007] The image input module is used to acquire images;

[0008] The image classification module includes a ConvMAE network, a multilayer perceptron task head, and a support vector machine task head; the image classification module is used to analyze and process the input image to obtain the classification result.

[0009] The results output module is used to output the classification results.

[0010] Optionally, in the transformer-based image classification model, the ConvMAE network includes a first convolutional neural network, a second convolutional neural network, an encoder, and a decoder arranged sequentially; the first convolutional neural network is used to perform vector normalization on the input image and extract features; the second convolutional neural network is used to divide the input image into image blocks and perform vector normalization and feature extraction on each image block; the encoder is used to encode the extracted features; and the decoder is used to decode the encoding output by the encoder to reconstruct the input image.

[0011] Optionally, in the transformer-based image classification model, both the first convolutional neural network and the second convolutional neural network include a block embedding unit and a mask convolutional unit arranged sequentially; the embedding unit is used to perform vector normalization on the input image using the patchify function; the mask convolutional unit is used to randomly mask a feature map of a preset ratio and perform feature extraction.

[0012] Optionally, in the transformer-based image classification model, the encoder includes a block embedding unit and 11 hierarchically connected transformer units arranged sequentially; the embedding unit is used to perform vector normalization on the input image using the patchify function; the transformer unit includes a multi-head self-attention mechanism and a feedforward neural network for extracting high-level image information.

[0013] Optionally, in the transformer-based image classification model, the decoder includes a linear layer and a transformer unit arranged sequentially; the linear layer is used to acquire all features; and the transformer unit is used to reconstruct the image using the acquired features.

[0014] Optionally, in the transformer-based image classification model, the ConvMAE network further includes a feature fusion unit and a mask transmission unit; the feature fusion unit is used to fuse the features output by the first convolutional neural network, the second convolutional neural network, and the encoder, and input the fused features to the decoder; the mask transmission unit is used to transmit a mask vector of size [N, 56, 56] to the first convolutional neural network, a mask vector of size [N, 28, 28] to the second convolutional neural network, and a mask vector of size [1, 1, decoder_dim] to the encoder and the decoder.

[0015] To address the aforementioned technical problems, this invention also provides an iterative method for a transformer-based image classification model, used to iterate over the transformer-based image classification model as described in any of the preceding claims, the iterative method comprising:

[0016] Obtain the labeled image;

[0017] The labeled image is input into the ConvMAE network to encode and reconstruct the input image, and the output vector is obtained.

[0018] The output vectors are fine-tuned using the multilayer perceptron task head and the support vector machine task head respectively to obtain their corresponding prediction vectors;

[0019] The predicted vectors from the multilayer perceptron task head and the support vector machine task head are weighted and averaged, and then classified to obtain the final classification result.

[0020] Optionally, in the iterative method of the transformer-based image classification model, the method for obtaining labeled image data includes:

[0021] Obtain the original image;

[0022] Preprocess the original image;

[0023] According to the classification requirements, the feature information in the preprocessed original image is labeled;

[0024] Pre-classify the labeled images.

[0025] Optionally, in the iterative method of the transformer-based image classification model, the method of inputting the labeled image into the ConvMAE network to encode and reconstruct the input image to obtain the output vector includes:

[0026] The first convolutional neural network is used to perform vector normalization on the image and extract features;

[0027] The image is divided into image blocks using a second convolutional neural network, and vector normalization and feature extraction are performed on each image block.

[0028] The extracted features are encoded using an encoder;

[0029] The encoder output is decoded using a decoder to reconstruct the input image.

[0030] Optionally, in the iterative method of the transformer-based image classification model, the method of fine-tuning the output vector using the multilayer perceptron task head and the support vector machine task head to obtain their respective corresponding prediction vectors includes:

[0031] The transformer network is initialized based on the pre-trained weights from ImageNet.

[0032] Connect the multilayer perceptron task head to the tail of the initialized transformer network;

[0033] The transformer network is fine-tuned using a multilayer perceptron task head to obtain the prediction vector;

[0034] Freeze the backbone network parameters of the transformer network and replace the multilayer perceptron task head with the support vector machine task head;

[0035] The transformer network is fine-tuned using the support vector machine task head to obtain the prediction vector.

[0036] This invention provides a transformer-based image classification model and its iterative method, comprising: an image input module for acquiring images; an image classification module including a ConvMAE network, a multilayer perceptron task head, and a support vector machine task head; the image classification module is used to analyze and process the input image to obtain classification results; and a result output module is used to output the classification results. By introducing a ConvMAE network into the image classification module, important features in the image can be automatically learned and effectively represented and expressed. Furthermore, the ConvMAE network is rapidly fine-tuned using the multilayer perceptron task head and the support vector machine task head, which improves classification accuracy while reducing the complexity of input data and increasing fine-tuning speed. This solves the problems of slow iteration speed and poor training effect of existing transformer-based models. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the structure of the transformer-based image classification model provided in this embodiment;

[0038] Figure 2 This is a schematic diagram of the ConvMAE network structure provided in this embodiment;

[0039] Figure 3 This is a schematic diagram of the block embedding unit provided in this embodiment;

[0040] Figure 4 This is a schematic diagram of the structure of the mask convolution unit provided in this embodiment;

[0041] Figure 5 This is a schematic diagram of the structure of the transformer unit provided in this embodiment;

[0042] Figure 6 A flowchart illustrating the iterative method of the transformer-based image classification model provided in this embodiment;

[0043] Figure 7 This is a flowchart illustrating the process of fine-tuning using MLP and SVM provided in this embodiment;

[0044] Figure 8 This is a flowchart illustrating the iterative method for the transformer-based image classification model provided in this embodiment. Detailed Implementation

[0045] The image classification model based on transformers and its iterative method proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and use imprecise scales, and are only used to facilitate and clarify the illustration of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different figures may emphasize different aspects and sometimes use different scales.

[0046] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects in order to describe embodiments of the invention, and are not used to describe a specific order or sequence. It should be understood that such uses of terminology are interchangeable where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] This embodiment provides an image classification model based on transformers, such as Figure 1 As shown, it includes: an image input module for acquiring images; an image classification module, including a ConvMAE network, a multilayer perceptron task head, and a support vector machine task head; the image classification module is used to analyze and process the input image to obtain classification results; and a result output module is used to output the classification results.

[0048] The transformer-based image classification model provided in this embodiment introduces a ConvMAE network into the image classification module, which can automatically learn important features in the image and perform effective feature representation and expression. It also uses a multilayer perceptron task head and a support vector machine task head to quickly fine-tune the ConvMAE network, which can improve classification accuracy while reducing the complexity of input data and improving the fine-tuning speed. This solves the problems of slow iteration speed and poor training effect of existing transformer-based models.

[0049] Specifically, in this embodiment, such as Figure 2 As shown, the ConvMAE network includes a first convolutional neural network (CNN Block Stage 1), a second convolutional neural network (CNN Block Stage 2), an encoder (Encoder Stage 3), and a decoder (Decoder Stage 4) arranged sequentially. The first convolutional neural network is used to perform vector normalization on the input image and extract features. The second convolutional neural network is used to divide the input image into image blocks and perform vector normalization and feature extraction on each image block. The encoder is used to encode the extracted features. The decoder is used to decode the encoding output by the encoder to reconstruct the input image.

[0050] Furthermore, in this embodiment, both the first convolutional neural network and the second convolutional neural network include a patch embedding unit and a masked convolution block arranged sequentially; the patch embedding unit is used to perform vector normalization on the input image using the patchify function; the masked convolution block is used to randomly mask feature maps of a preset ratio and perform feature extraction.

[0051] Furthermore, in this embodiment, the encoder includes a patch embedding unit and 11 layered transformer units arranged sequentially; the patch embedding unit is used to perform vector normalization on the input image using the patchify function; the transformer unit includes a multi-head self-attention mechanism and a feedforward neural network for extracting high-level image information.

[0052] Furthermore, in this embodiment, the decoder includes a linear layer and a transformer block arranged sequentially; the linear layer is used to acquire all features; and the transformer block is used to reconstruct the image using the acquired features.

[0053] In practical applications, the ConvMAE network is a hybrid CNN+Transformer network model. In the entire network structure, the image passes through the patchify function to obtain the input required by the block embedding unit. This input then flows into the block embedding unit of the first convolutional neural network (CNN) to perform vector normalization on the image / image patch. Features are then extracted by the mask convolution unit of the first CNN. Specifically, the mask convolution unit randomly masks a portion of the feature map, for example, 75%. Next, the dimensions of the current data features H (image height) and W (image width) are reduced to H / 4 and W / 4 of the original image size, respectively, and fed into the block embedding unit of the second CNN, where features are extracted by the mask convolution unit. Subsequently, the dimensions of the current data features H and W are reduced to H / 8 and W / 8 of the original image size, respectively, and fed into the block embedding unit of the encoder, where features are extracted by 11 cascaded transformer units. Finally, the global information is extracted by the linear layer of the decoder and one transformer unit.

[0054] In one specific embodiment, the first convolutional neural network consists of one block embedding unit and two mask convolutional units. The block embedding unit divides the input image into multiple small blocks and passes them to the mask convolutional units. The mask convolutional units employ a convolutional neural network architecture with residual connections to extract local features. The second convolutional neural network consists of one block embedding unit and two mask convolutional units. Similarly, the block embedding unit passes the output of the patchify function to the mask convolutional units to extract higher-level features. Each transformer unit in the encoder includes a multi-head self-attention mechanism and a feedforward neural network for extracting high-dimensional image information.

[0055] Furthermore, in this embodiment, the ConvMAE network further includes a feature fusion unit (multi-scale fusion) and a block-wise masking unit; the feature fusion unit is used to fuse the features output by the first convolutional neural network, the second convolutional neural network, and the encoder, and input the fused features to the decoder; the block-wise masking unit is used to pass a mask vector of size [N, 56, 56] to the first convolutional neural network, a mask vector of size [N, 28, 28] to the second convolutional neural network, and a mask vector of size [1, 1, decoder_dim] to the encoder and the decoder.

[0056] In practical applications, in the first convolutional neural network, the feature map passes through a convolutional layer with a large stride and padding (stride 4, padding 4), reducing the size of the feature map to 1 / 4 of its original size. Then, the feature map is multiplied by a mask to allow the model to selectively focus on specific regions. Finally, a flattening operation transforms the processed feature map into a one-dimensional vector vector1. Similarly, in the second convolutional neural network, the feature map passes through a convolutional layer with a large stride and padding (stride 2, padding 2), reducing the size of the feature map to 1 / 8 of its original size. Then, the feature map is multiplied by a mask and flattened to obtain a one-dimensional vector vector2. The encoder outputs a K-dimensional vector vector3, representing the information extracted by the Transformer. Next, the feature fusion unit adds vector1, vector2, and vector3 to obtain a one-dimensional vector containing all the information from the first, second, and third scales; this vector is then fed into the decoder for further processing and generation of the final output. In this way, by integrating the information extracted from different stages, the transformer model is provided with richer input, which helps the model better understand the context and key information of the input data, thereby improving its modeling and expressive capabilities.

[0057] Furthermore, in practical applications, ConvMAE pre-generates a 1 / 4 downsampled mask 1 and a 1 / 8 downsampled mask 2. Masks 1 and 2 are obtained from the vector of mask 4 in the decoder through a reconstruction operation, which can reconstruct the vector into a 1 / 16 downsampled mask 3. First, N images of the input [N, 3, 224, 224] are processed by patchify to obtain [N, L, D], where L is the number of image patches and D is the encoding dimension of each image patch. Then, an N×L dimension of random noise between [0, 1] is generated, and the values ​​in the noise are sorted in ascending order to obtain id_restore. Next, an [N, L] dimension mask matrix of all 1s is generated, and the length to be retained is calculated using the mask rate. Finally, id_restore is used to restore the mask, ensuring that the mask order remains consistent with the original image. In this way, ConvMAE reduces resource consumption during training while preserving the original image information. The mask transmission unit reduces the original large amount of mask information into a few smaller downsampled masks, and achieves random masking of image patches of the input image through random noise and sorting operations. This not only reduces the computation and storage requirements for model training, but also improves the training effect and generalization ability of the model.

[0058] By fusing features from multiple outputs through the feature fusion unit, it is possible to capture multi-layered information of the image at different scales, thereby improving model performance. Furthermore, by providing downsampled graph masks through the mask transmission unit, it is possible to specify data regions corresponding to masking mechanisms at different stages of the model, which helps improve the model's image reconstruction capabilities while reducing computational complexity during model training.

[0059] In this embodiment, as Figure 3 As shown, the block embedding unit comprises a convolutional layer, a layer normalization layer, and a GELU function layer arranged sequentially. The block embedding unit encodes local information in the image into vector representations by dividing the image into fixed-size blocks and generating an embedding vector for each block. This allows for better capture of the image's features and structure. The embedding vector of each block can be considered a feature representation of that region and can be used for subsequent processing tasks. Using low-dimensional vector representations reduces computational and memory overhead and enables better handling of large images. Compared to processing the entire image, processing image blocks significantly reduces the number of parameters, thereby improving computational efficiency. Furthermore, since each image block has its own vector representation, it can better handle images of different sizes and shapes.

[0060] And, in this embodiment, as Figure 4 As shown, the masked convolutional unit includes a feature embedding layer, a normalization layer (Norm1), a convolutional layer (Convolution1), an attention layer, a convolutional layer (Convolution2), a normalization layer (Norm2), and a feedforward neural network layer (FFN) arranged sequentially. A mask is introduced between the convolutional layer (Convolution1) and the attention layer. Between the convolutional layer (Convolution2) and the normalization layer (Norm2), the outputs of the feature embedding layer and the convolutional layer (Convolution2) are concatenated. After the feedforward neural network layer (FFN), the result of the previous concatenation is concatenated with the output of the feedforward neural network layer (FFN) to obtain the final output.

[0061] In practical applications, normalization layers can be multiple layers to normalize features, ensuring that feature distributions of different dimensions have similar magnitudes, facilitating network learning. Convolutional layers can have 1×1 kernels to transform features, mapping features with *dim* input channels to features with *dim* output channels. New feature maps are obtained through masking, preserving the original features while controlling the model's learning of unmasked regions, allowing the model to better focus on regions of interest and improve its understanding of mask information. Attention layers can also be convolutional layers with a kernel size of 5, padding of 2, and grouped convolutions of *dim*. They perform local convolutions on the feature map to capture interaction information between different regions, helping the model better understand the relationships and contextual information in the image. Concatenation operations can be linear addition, introducing non-linear transformations and fusing information from multiple feature layers to improve the model's expressive and generalization abilities.

[0062] And, in this embodiment, as Figure 5 As shown, the transformer unit comprises a feature embedding layer, a normalization layer (Norm1), a transformer attention layer, a normalization layer (Norm2), and a multilayer perceptron (MLP), arranged sequentially. After the transformer attention layer, the outputs of the feature embedding layer and the transformer attention layer are concatenated; after the multilayer perceptron, the result of the previous concatenation is concatenated with the output of the multilayer perceptron (MLP) to obtain the final output.

[0063] In practical applications, the feature embedding layer maps input features to a higher-dimensional space, thereby increasing the expressive power of the features and making them more suitable for subsequent attention mechanisms and multilayer perceptrons. Normalization layers can be layer-by-layer normalization layers, ensuring that features of different dimensions have similar scales, thus helping to improve the model's convergence speed and generalization ability. The transformer attention layer calculates the similarity between input feature maps and uses attention weights based on this similarity to focus on important information at different locations, enabling the model to automatically learn the inherent relationships and contextual information of the input features. A multilayer perceptron typically consists of multiple fully connected layers, with each fully connected layer having the same number of neurons as the previous layer. The number of neurons in the last fully connected layer is usually equal to the number of categories in the task. By training the multilayer perceptron, it can learn how to extract useful information from the fused features and map it to the correct category labels. The concatenation operation is usually performed by adding or concatenating elements along the channel dimension.

[0064] This embodiment also provides an iterative method for a transformer-based image classification model, used to iterate over the transformer-based image classification model described above, such as... Figure 6 As shown, the iterative method includes:

[0065] S1, Obtain the labeled image;

[0066] S2, the labeled image is input into the ConvMAE network to encode and reconstruct the input image, and the output vector is obtained;

[0067] S3, the output vector is fine-tuned using the multilayer perceptron task head and the support vector machine task head respectively to obtain their respective prediction vectors;

[0068] S4. The predicted vectors from the multilayer perceptron task head and the support vector machine task head are weighted and averaged, and then classified to obtain the final classification result.

[0069] The iterative method for image classification models based on transformers provided in this embodiment utilizes ConvMAE networks to encode and reconstruct input images. It can automatically learn important features in the image and perform effective feature representation and expression. Furthermore, it uses multilayer perceptron task heads and support vector machine task heads for rapid fine-tuning. This method can improve classification accuracy while reducing the complexity of input data and increasing fine-tuning speed, thus solving the problems of slow iteration speed and poor training effect of existing transformer-based models.

[0070] Specifically, in this embodiment, step S1, the method for obtaining labeled image data, includes:

[0071] S11, Obtain the original image.

[0072] In one specific embodiment, the acquired raw images are images of pleural and peritoneal fluid samples. These images include different types of cells such as ADE-M (adenocarcinoma clusters), ADE-S (single adenocarcinoma), MC-M (mesothelial clusters), and MC-S (single mesothelial cells), as well as T1 (blank and blurred), T2 (filamentous material), T3 (impurities), and T4 (inflammatory cells).

[0073] S12, preprocess the original image.

[0074] Taking the above-mentioned pleural and peritoneal fluid sample images as an example, to facilitate subsequent automatic annotation, the images need to be processed using HSV+LUV to segment the outer contours of the cells, making it easier to extract the cell contours and morphological features. This method has high accuracy and robustness, and can effectively reduce the difficulty and time cost of annotation.

[0075] S13, According to the classification requirements, the feature information in the preprocessed original image is labeled.

[0076] The preprocessed images can then be labeled based on the morphology, structure, and staining of each cell. While labeling can be done manually, it is time-consuming, labor-intensive, and prone to errors. Therefore, appropriate processing models can be used. Of course, after automatic labeling by the processing model, manual sampling can be performed to confirm the accuracy of the automatic labeling.

[0077] S14, pre-classify the labeled images.

[0078] In this embodiment, a classic convolutional neural network (CNN) model, ResNet50, is used for pre-classification. The ResNet50 model has strong feature extraction and classification capabilities, which can meet the processing requirements of image pre-classification.

[0079] Furthermore, in this embodiment, step S2, inputting the labeled image into the ConvMAE network to encode and reconstruct the input image to obtain the output vector, includes the following methods:

[0080] S21, The first convolutional neural network is used to perform vector normalization on the image and extract features;

[0081] S22, the second convolutional neural network is used to divide the image into image blocks, and vector normalization and feature extraction are performed on each image block;

[0082] S23, The extracted features are encoded using an encoder;

[0083] S24 uses a decoder to decode the encoder output to reconstruct the input image.

[0084] The specific implementation of this step can be found above and will not be repeated here. In this step, a ConvMAE network is used to encode and reconstruct the input image: the input image I is encoded into a hidden vector, and then a decoder is used to reconstruct the hidden vector into the predicted image J. The mean squared error loss (MSELoss) between the predicted image J and the input image I is calculated, and this loss is used to optimize the model parameters. Through this process, the ConvMAE network learns the ability to represent and reconstruct images.

[0085] Furthermore, in this embodiment, as Figure 7 As shown, step S3, which involves fine-tuning the output vector using the multilayer perceptron task head and the support vector machine task head to obtain their respective corresponding prediction vectors, includes the following methods:

[0086] S31, initialize the transformer network based on ImageNet pre-trained weights.

[0087] S32, connect the Multilayer Perceptron Task Header (MLP) to the tail of the initialized transformer network.

[0088] S33 uses a multilayer perceptron task head to fine-tune the transformer network and obtain the prediction vector.

[0089] In this step, all parameters are frozen, retaining only the output vector of the encoder in the transformer network. This output vector is then fed into a multilayer perceptron (MLP) task head for fine-tuning. The MLP task head is a fully connected neural network that receives the ConvMAE output vector as input and adjusts it according to the specific task. By fine-tuning the MLP task head, the pre-trained feature extraction capabilities are utilized to adapt the model to the specific task, improving its performance on that task.

[0090] When fine-tuning using a multilayer perceptron task head, several hyperparameters require adjustment, including the number of network layers, dropout rate, learning rate, weight decay, and optimizer. Improper fine-tuning can lead to performance degradation of the Transformer, such as catastrophic forgetting of historical knowledge. Therefore, more measures must be taken to prevent overfitting during fine-tuning. Thus, this embodiment adds SVM fine-tuning to the MLP fine-tuning:

[0091] S34, freeze the backbone network parameters of the transformer network and replace the multilayer perceptron task head with the support vector machine task head (SVM).

[0092] S35 uses the support vector machine task head to fine-tune the transformer network and obtain the prediction vector.

[0093] In this step, all parameters are frozen, retaining only the encoder's output vector, which is then fed into the Support Vector Machine (SVM) task head for fine-tuning. SVM is a classic machine learning algorithm used for classification problems. By fine-tuning the SVM task head, the pre-trained feature extraction capabilities are utilized for image classification, further enhancing the diversity of features.

[0094] Because SVM is robust to overfitting and highly effective for high-dimensional data, and because transformers have learned to extract meaningful representations in the form of long embedding vectors through MLP fine-tuning, CUML-SVM can be chosen for fine-tuning in practical applications. CUML-SVM is simple to tune parameters, has a fast processing speed, and supports diverse prediction statistics, making it the best candidate for task head classifiers.

[0095] In this embodiment, step S4 involves taking a weighted average of the prediction vectors from the multilayer perceptron task head and the support vector machine task head, and then classifying them to obtain the final classification result.

[0096] In this step, the predicted vectors extracted from the fine-tuned MLP and SVM task heads are weighted and averaged. The purpose of weighted averaging is to comprehensively utilize the prediction information from both task heads to obtain a more comprehensive and accurate feature representation. Then, the softmax function is used to classify the weighted average vector to obtain the final classification result.

[0097] Thus, by averaging the prediction results from the MLP head and the SVM head, the advantages of both methods can be combined to obtain more accurate prediction results. This SVM-based fine-tuning method can fully utilize the semantic information learned by the Transformer and the classification capabilities of the SVM, achieving better performance in classification or regression tasks. Meanwhile, CUML-SVM provides GPU acceleration, making the training process faster and more efficient.

[0098] The iterative method for image classification based on the transformer provided in this embodiment, in practical applications, proceeds as follows: Figure 8As shown: The image is input into the ConvMAE network, where it is encoded and reconstructed to obtain the reconstructed image (output vector). The analysis and processing of the ConvMAE network is a normal image classification process. During model iteration, the transformer in the encoder of the ConvMAE network is fine-tuned using MLP and SVM respectively to obtain the predicted vector; then, the predicted vector is weighted and averaged before being classified and output, completing the iteration.

[0099] Thus, by using the feature vectors generated by the transformer as input, SVM can find the most representative feature subset, improving the efficiency of model training and generalization ability. By reducing feature dimensions, feature interactions, and fast fine-tuning steps, the fine-tuning cycle can be significantly shortened, improving the model's fine-tuning efficiency. This allows the model to converge in fewer iterations and avoids the problem of excessively long iteration cycles in traditional fine-tuning methods.

[0100] In practical applications, Swin-Transformer + SVM can be used to replace standalone SVM for fine-tuning. Swin-Transformer captures global and local features in images through self-attention and local window interaction, enabling image representation learning and classification. It divides the image into several small patches and hierarchically encodes these patches to perceive the entire image. Swin-Transformer is an end-to-end deep learning model that extracts meaningful image features through large-scale training data and adaptive parameter learning. SVM classifies data by mapping samples to a high-dimensional space and constructing decision boundaries based on support vectors (sample points on the hyperplane). During training, SVM requires manual selection of appropriate kernel functions and adjustment of hyperparameters to achieve good performance on specific datasets. In image classification tasks, after extracting image features using Swin-Transformer, these features are used as input to the SVM model, which then makes the final classification decision. By combining the representation learning capabilities of Swin-Transformer and the classification capabilities of SVM, a more powerful image classification model can be obtained.

[0101] Of course, Mask Autoencoder (MAE) + SVM can also be used instead of a standalone SVM for fine-tuning. Mask-Autoencoder is a neural network model used for unsupervised learning and feature learning. Its main goal is to learn a sparse representation of data through an autoencoder. MAE consists of an encoder and a decoder, where the encoder compresses the input data into a low-dimensional representation, and the decoder attempts to reconstruct the original data from this low-dimensional representation. During training, MAE encourages the encoder to learn meaningful features by introducing a sparsity penalty term. Once training is complete, the encoder part of MAE can be used to extract feature representations of the input data. These features can be considered as removing noise and redundancy while retaining the main information. Using these features as input, SVM can be further used for classification tasks.

[0102] However, while the two alternatives mentioned above can achieve better image classification models compared to fine-tuning a standalone SVM, they consume more training resources and time. Therefore, in practical applications, a reasonable choice needs to be made based on actual requirements.

[0103] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. In addition, the different parts between embodiments can also be combined with each other, and this invention does not limit this.

[0104] This embodiment provides a transformer-based image classification model and its iterative method, comprising: an image input module for acquiring images; an image classification module including a ConvMAE network, a multilayer perceptron task head, and a support vector machine task head; the image classification module is used to analyze and process the input image to obtain classification results; and a result output module is used to output the classification results. By introducing a ConvMAE network into the image classification module, important features in the image can be automatically learned and effectively represented and expressed. The ConvMAE network is then rapidly fine-tuned using the multilayer perceptron task head and the support vector machine task head. This reduces the complexity of the input data, improves the fine-tuning speed, and enhances classification accuracy, thus solving the problems of slow iteration speed and poor training performance of existing transformer-based models.

[0105] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. An iterative method for a transformer-based image classification model, used to iterate over the transformer-based image classification model, wherein the image classification model includes: The image input module is used to acquire images; The image classification module includes a ConvMAE network, a multilayer perceptron task head, and a support vector machine task head; The image classification module is used to analyze and process the input image to obtain the classification result; The results output module is used to output the classification results; The iterative method is characterized by comprising: Obtain the labeled image; The labeled image is input into the ConvMAE network to encode and reconstruct the input image, and the output vector is obtained. The output vectors are fine-tuned using the multilayer perceptron task head and the support vector machine task head respectively to obtain their corresponding prediction vectors; The predicted vectors from the multilayer perceptron task head and the support vector machine task head are weighted and averaged, and then classified to obtain the final classification result.

2. The iterative method for the image classification model based on transformer according to claim 1, characterized in that, The ConvMAE network includes a first convolutional neural network, a second convolutional neural network, an encoder, and a decoder arranged sequentially. The first convolutional neural network is used to perform vector normalization on the input image and extract features. The second convolutional neural network is used to divide the input image into image blocks and perform vector normalization and feature extraction on each image block. The encoder is used to encode the extracted features. The decoder is used to decode the encoding output by the encoder in order to reconstruct the input image.

3. The iterative method for the image classification model based on transformer according to claim 2, characterized in that, Both the first convolutional neural network and the second convolutional neural network include a block embedding unit and a mask convolution unit arranged sequentially; the embedding unit is used to perform vector normalization on the input image using the patchify function; the mask convolution unit is used to randomly mask feature maps of a preset ratio and perform feature extraction.

4. The iterative method for the image classification model based on transformer according to claim 2, characterized in that, The encoder includes a block embedding unit arranged sequentially and 11 hierarchically connected transformer units; the embedding unit is used to perform vector normalization on the input image using the patchify function; the transformer unit includes a multi-head self-attention mechanism and a feedforward neural network for extracting high-dimensional image information.

5. The iterative method for the image classification model based on transformer according to claim 2, characterized in that, The decoder includes a linear layer and a transformer unit arranged sequentially; the linear layer is used to acquire all features; the transformer unit is used to reconstruct the image using the acquired features.

6. The iterative method for the image classification model based on transformer according to claim 2, characterized in that, The ConvMAE network further includes a feature fusion unit and a mask transmission unit; the feature fusion unit is used to fuse the features output by the first convolutional neural network, the second convolutional neural network, and the encoder, and input the fused features to the decoder; the mask transmission unit is used to transmit a mask vector of size [N, 56, 56] to the first convolutional neural network, a mask vector of size [N, 28, 28] to the second convolutional neural network, and a mask vector of size [1, 1, decoder_dim] to the encoder and the decoder.

7. The iterative method for the image classification model based on transformer according to claim 1, characterized in that, The method for obtaining labeled image data includes: Obtain the original image; Preprocess the original image; According to the classification requirements, the feature information in the preprocessed original image is labeled; Pre-classify the labeled images.

8. The iterative method for the image classification model based on transformer according to claim 1, characterized in that, The method of inputting the labeled image into the ConvMAE network to encode and reconstruct the input image to obtain the output vector includes: The first convolutional neural network is used to perform vector normalization on the image and extract features; The image is divided into image blocks using a second convolutional neural network, and vector normalization and feature extraction are performed on each image block. The extracted features are encoded using an encoder; The encoder output is decoded using a decoder to reconstruct the input image.

9. The iterative method for the image classification model based on transformer according to claim 1, characterized in that, The method of fine-tuning the output vector using a multilayer perceptron task head and a support vector machine task head to obtain their respective corresponding prediction vectors includes: The transformer network is initialized based on the pre-trained weights from ImageNet. Connect the multilayer perceptron task head to the tail of the initialized transformer network; The transformer network is fine-tuned using a multilayer perceptron task head to obtain the prediction vector; Freeze the backbone network parameters of the transformer network and replace the multilayer perceptron task head with the support vector machine task head; The transformer network is fine-tuned using the support vector machine task head to obtain the prediction vector.

Images