Image block division preprocessing method of visual transformer

Abstract

The application discloses a kind of image block division pre-processing methods of visual Transformer, the method is for the locality of visual Transformer, translation invariance lacking to image, PVT image classification algorithm is improved, on the basis thereof, image block has been treated with multi-scale feature fusion, multi-scale feature fusion image block embedding layer (MSFF_PE) is proposed, through the lightweight multi-scale feature fusion module designed, the multi-scale feature information in image block is enriched, and the feature representation capability of image block is strengthened.For the modeling of lacking local information in the structure of encoder, a multi-scale local enhancement feedforward neural network layer (MSLE_FF) is proposed, the one-dimensional sequence output by the attention mechanism is restored to a two-dimensional image for local information interaction, and the weighted information of local features is obtained by introducing distance importance. Finally, the effectiveness of the method of the application is verified through experiments.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image block segmentation preprocessing method using a visual Transformer. Background Technology

[0002] In recent years, visual Transformer-based models have rapidly developed in the field of computer vision, achieving superior performance in various computer vision tasks such as image classification, object detection, and semantic segmentation. The success of visual Transformer models mainly relies on their self-attention mechanism, which allows them to acquire a larger receptive field than CNNs, thus enabling them to model global dependencies and representing a major research direction in classification algorithms. While visual Transformers have achieved success on large-scale datasets, their performance lags behind convolutional neural networks with similar parameter sizes when trained on smaller datasets. Training on large-scale datasets requires more demanding graphics cards and equipment, characterized by long training cycles and high trial-and-error costs. Therefore, compared to large-scale datasets, small to medium-sized datasets have wider applications in machine learning and computer vision. Applying visual Transformer models to small to medium-sized datasets can lower the research and development threshold and is of great significance for iterating specific scenarios and models.

[0003] Because Transformers are used to process one-dimensional sequence inputs, image inputs need to be divided into image patches and then stretched into one-dimensional vectors. This prevents the attention mechanism from utilizing the inherent properties of two-dimensional images, necessitating the use of large-scale data to learn high-quality feature representations. Convolutional neural networks excel at handling visual tasks because the convolution operation traverses the image in a sliding window manner, exhibiting translation invariance. Furthermore, the shared weights and local receptive fields of the convolutional kernels better reflect the highly correlated spatial relationships between adjacent pixels in an image. Therefore, one direction for improvement is to introduce convolution into the image patch embedding layer, endowing image patches with richer feature representations and enhancing image attributes such as multi-scale characteristics and locality within the patches.

[0004] The visual Transformer divides the input 2D image into blocks and stretches them into 1D vectors at the image patch embedding layer, which then enters the main encoder part of the model. Attention layer, global information exchange between image blocks occurs. However, in a 2D image, the most relevant region to a given pixel is often a local area centered on that pixel; generally, the greater the distance, the lower the relevance. This global information exchange mechanism of self-attention ignores modeling image locality, making it difficult to learn the inherent properties of the image, resulting in poor performance on small to medium-sized datasets. Summary of the Invention

[0005] Based on the analysis of existing technologies for visual Transformers, and addressing the lack of translation invariance, locality, and other inherent image properties in the visual Transformer model, this paper improves the image patch embedding layer and feedforward neural network in PVT. This improves the modeling of global dependencies while acquiring the locality, multi-scale characteristics, and translation invariance of images, thereby enhancing the performance of the visual Transformer model on small to medium-sized datasets.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an image block segmentation preprocessing method for visual Transformer, comprising the following steps:

[0007] S1: Obtain several publicly labeled images as a training set, with one image serving as a training sample.

[0008] S2: Construct and train an improved Transformer classification model. The improved Transformer classification model is based on the existing Transformer classification model with the following two improvements: first, the image patch embedding layer is changed to a multi-scale feature fusion image patch embedding layer MSFF_PE; second, the feedforward neural network is improved to a multi-scale local enhancement feedforward neural network layer MSLE_FF.

[0009] The training samples are input into the improved Transformer classification model, passing through an improved image patch embedding layer (MSFF_PE), a spatial reduction attention module, and an improved feedforward neural network (MSLE_FF) to continuously update the feature maps. Finally, a linear layer outputs the final predicted classification result. The loss between the true and predicted labels of the training samples is calculated, and backpropagation is used to calculate the gradient of the parameters of each layer of the model. The model parameters are adjusted based on the calculated gradients to minimize the loss function. This process is iterated until the loss function is minimized, at which point the trained Transformer classification model is obtained.

[0010] S3: Input an image to be classified into the trained Transformer classification model. The trained Transformer classification model calculates the probability value of the image to be classified into each class and outputs the class corresponding to the highest probability value as the predicted class of the image to be classified.

[0011] Furthermore, in S2, the process of inputting training samples into MSFF_PE and calculating the feature map corresponding to each training sample is as follows:

[0012] y1 = conv7(input)

[0013] y2 = MSFF(y1)

[0014] y3 = conv2(y2)

[0015] patch1, ..., patch n =flatten(y3)×E (3)

[0016] in, This represents the input image MSFF_PE, conv k This indicates a convolution with kernel size k. MSFF represents the multi-scale feature fusion module imp_Inception(a). The stride of both conv7 and conv2 convolutions is set to 2. The MSFF module does not change the size of the input feature map, so the output after the image patch embedding layer... "Flatten" means stretching the spatial dimensions of the feature, i.e., length H and width W, resulting in n = H / 4 × W / 4 image patches. It is a fully connected layer, therefore the output is... This serves as the input to the attention layer. H, W, and C represent the height, width, and channel dimensions of the feature map of the input image, respectively, while D represents the dimension set for the hidden layer.

[0017] Furthermore, the processing procedure of imp_Inception(a) is as follows:

[0018]

[0019] Where input' represents the feature map input to imp_Inception(a), output' represents the feature map output by imp_Inception(a), and conv k This indicates performing a convolution with a kernel size of k, where the channel dimension of the convolution is 1 / 4 of the input channels. `maxpool3` indicates max pooling with a kernel size of 3. i ' represents the output of the i-th branch.

[0020] Furthermore, in step S2, the feature map is processed by MSLE_FF to obtain an updated feature map, the process of which is as follows:

[0021]

[0022] Where "input" represents the input feature map of MSLE_FF, "output" represents the output of MSLE_FF, θ1 and θ2 represent the dropout parameters, FC represents a fully connected layer, and y i " represents the output of the i-th layer, DWDilated m conv nDWconv3(.) represents dilated convolution with an expansion rate of d = m and a kernel size of k = n. DWconv3(.) represents depthwise separable convolution with a kernel size of 3. GELU(.) represents the activation function.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects:

[0024] This invention addresses the lack of locality and translation invariance in the Visual Transformer (PVT) image classification algorithm by improving upon it. Building upon this, multi-scale feature fusion is performed on image patches, resulting in a proposed multi-scale feature fusion image patch embedding layer (MSFF_PE). This lightweight multi-scale feature fusion module enriches the multi-scale feature information within image patches, enhancing their feature representation capabilities. To address the lack of local information modeling in the encoder structure, a multi-scale local enhancement feedforward neural network layer (MSLE_FF) is proposed. This layer reconstructs the one-dimensional sequence output by the attention mechanism into a two-dimensional image for local information interaction and introduces distance importance to obtain weighted information about local features. Finally, these improvements are incorporated into a baseline model named MS-PVT. Training and testing on the Mini-ImageNet and CIFAR-10 datasets validate the effectiveness of the proposed method, achieving Top-1 accuracy improvements of 5.60% and 2.92% on the two datasets, respectively, with only a 0.4M increase in parameters. Attached Figure Description

[0025] Figure 1 Here is a diagram of the imp_Inception(a) module structure.

[0026] Figure 2 Here is the structure diagram of the imp_Inception(b) module.

[0027] Figure 3 Image patch embedding layer with multi-scale feature fusion.

[0028] Figure 4 Comparison of ViT, Shunted Transformer, PVT, and feedforward neural network layers with multi-scale local enhancement.

[0029] Figure 5 A schematic diagram of dilated convolution. Detailed Implementation

[0030] Multi-scale feature fusion modules typically divide the network into multiple branches in one layer and stack convolutions of different depths on each branch to merge the features of branches with different receptive field sizes, thereby obtaining features containing multi-scale information.

[0031] Based on the improvement ideas of the Res2net module, this invention proposes an improved lightweight Inception module, which can achieve the same receptive field width and depth as Inception while reducing the parameters of the multi-scale feature fusion module. This module will be referred to as imp_Inception(a) below, and its specific structure is as follows: Figure 1 As shown, imp_Inception(a) consists of four parallel branches, each with one-quarter of the number of input feature channels. The first branch performs a 1×1 convolution, and the output is split into two branches: one outputs directly, and the other is superimposed on the input of the 3×3 convolution in the second branch. Similarly, the output of the second branch is superimposed on the input before the 3×3 convolution in the third branch. The fourth branch remains unchanged. By superimposing the output of the second branch with the input of the third branch, the third branch can also obtain the receptive field of two 3×3 convolutions, i.e., the size of a 5×5 convolution. In addition, residual connections are also added.

[0032] The improved method described above can achieve the same multi-scale features while reducing the number of parameters in the Inception module. The calculation process of the module is shown in formula (1):

[0033]

[0034] Where input' represents the feature map input to imp_Inception(a), output' represents the feature map output by imp_Inception(a), and conv k This indicates performing a convolution with a kernel size of k, where the channel dimension of the convolution is 1 / 4 of the input channels. `maxpool3` indicates max pooling with a kernel size of 3. i ' represents the output of the i-th branch.

[0035] Based on the improved ideas of the Ghost module, this invention also proposes an improved lightweight Inception module, which will be referred to as imp_Inception(b) in the following text. The specific structure is as follows: Figure 2As shown, imp_Inception(b) also consists of four parallel branches. The first branch performs a 1×1 convolution, with the output channel dimension being 1 / 4 of the input. The second branch performs a 1×1 convolution, with the output channel dimension being 1 / 8 of the input channel dimension. Then, similar to the Ghost module, the feature map output after the 3×3 convolution is fed into a depthwise separable convolution to obtain similar feature maps. The feature maps from the two paths are then concatenated. Similarly, the third branch performs a 1×1 convolution followed by two separate convolutions, and then concatenates the features along the channel dimension. The fourth branch remains unchanged. By compressing the channels of the second and third branches, the same multi-scale features can be obtained while reducing the number of parameters. In addition, residual connections are added, just like in the imp_Inception(a) module.

[0036] The calculation process of the imp_Inception(b) module is shown in formula (2):

[0037] y1 = conv1(input)

[0038] y 2_1 =conv3(conv1(input))

[0039] y 2_2 =DWconv3(y 2_1 )

[0040] y2 = concat(y 2_1 ,y 2_2 )

[0041] y 3_1 =conv3(conv3(conv1(input)))

[0042] y 3_2 =DWconv3(y 3_1 )

[0043] y3 = concat(y 3_1 ,y 3_2 )

[0044] y4 = conv1(maxpool3(input))

[0045] output=concat(y1,y2,y3,y4)+input (2)

[0046] Where input represents the input features, output represents the output features, and conv kThis indicates performing a convolution with a kernel size of k, where the number of channels in the first and last branches is 1 / 4 of the number of input channels, and the number of channels in the other two branches is 1 / 8. `DWconv3` represents a depthwise separable convolution with a kernel size of 3, and `maxpool3` represents a max pooling convolution with a kernel size of 3. i This represents the output of the i-th branch.

[0047] In PVT, the image patch embedding layer consists of convolutions with a kernel size of 4 and a stride of 4. Considering the lack of two-dimensional attributes in the visual Transformer model, to improve its performance in recognizing multi-sized targets on small to medium-sized datasets, we improved the image patch embedding layer, naming it the Multi-scale Feature Fusion Patch Embedding (MSFF_PE). First, features are processed using convolutions with a kernel size of 7 and a stride of 2. Then, imp_Inception(a) is used to learn multi-scale feature representations. Finally, convolutions with a kernel size of 2 and a stride of 2 are used for feature dimensionality reduction, thus reducing the size of the image feature map to as low as H / 4 × W / 4. By adding the multi-scale feature fusion module, the feature information within the image patch is enriched, and different receptive fields are provided, which is more conducive to the recognition of targets of different sizes. The structure diagram is shown below. Figure 3 As shown, a 7×7 convolution and a multi-scale feature fusion module are added to the image patch embedding layer of the baseline model to fuse features at different depths and then divide the image patch, increasing the receptive field of different scales within the image patch, and representing the image features more accurately and comprehensively.

[0048] The expression for MSFF_PE is shown in formula (3):

[0049] y1 = conv7(input)

[0050] y2 = MSFF(y1)

[0051] y3 = conv2(y2)

[0052] patch1, ..., patch n =flatten(y3)×E (3)

[0053] in, This represents the input image MSFF_PE, conv kThis indicates a convolution with kernel size k. MSFF represents the multi-scale feature fusion module imp_Inception(a). The stride of both conv7 and conv2 convolutions is set to 2. The MSFF module does not change the size of the input feature map, so the output after the image patch embedding layer... "Flatten" means stretching the spatial dimensions of the feature, i.e., length H and width W, resulting in n = H / 4 × W / 4 image patches. It is a fully connected layer, therefore the output is... As input to the attention layer. y3 represents the output of MSFF_PE, and patch indicates that the feature map output by MSFF_PE is divided into image patches, and each image patch is stretched into a one-dimensional vector in the length (H) and width (W) dimensions. H, W, and C represent the height, width, and channel size of the feature map of the input image, respectively. D represents the dimension set by the hidden layer, which is generally set to 64.

[0054] In multi-head attention layers, the global dependencies between image patches are modeled. However, in a 2D image, for a point (i, j), the most relevant region is often a square region centered at (i, j) with a side length of k. The interaction of global information in the self-attention mechanism ignores the locality of the image, resulting in a lack of modeling of local features. Subsequent work addressed this issue by improving feedforward neural network layers. For example, in PVT, a depthwise separable convolution is placed between two fully connected layers to restore the input features to image form, enhancing the locality of the features; in ShuntedTransformer, the depthwise separable convolution is set as a residual connection, superimposed with the original features from the fully connected layers, further enhancing the locality of the features.

[0055] Based on the aforementioned ideas of enhancing the locality of image patches and multi-scale feature fusion, a Multi-scale Local-enhanced Feed Forward (MSLE_FF) layer is proposed. In the baseline model PVT, the depthwise separable convolution kernel size is set to 3 to obtain local information. Considering the inclusion of multi-scale features, a possible strategy is to add new branches and set the depthwise separable convolution kernel to 5 or 7 to obtain information from different receptive fields. This approach undoubtedly increases the computational cost of the model, and also presents the following problem: In a two-dimensional image, the set of pixels with the highest correlation to position (i,j) is the set of pixels at a distance of one pixel {(i′,j′)|i′=i+p,j′=j+q}, where -1≤p,q≤1. The closer the distance, the greater the correlation between pixels; the farther the distance, the smaller the correlation between pixels. Therefore, even with the addition of receptive fields of different sizes, it is still necessary to consider the issue that the dependence between neighboring regions is greater than the dependence between distant regions. Taking into account both computational complexity and distance importance, this invention combines dilated convolution and depthwise separable convolution on top of the baseline model, adding a branch of depthwise separable dilated convolution (DW-Dilated Convolution). Simultaneously, the dilation rate of the dilated convolution is used as an evaluation factor for distance importance. Figure 4 As shown. Because the DWConv branch has a kernel size of 3, and all pixels are one pixel away from the center point, its importance is 1. The DW-Dilated convolution has a dilation rate of d, meaning it is d pixels away from the center, and its importance is set to 1 / d. Simultaneously, due to the dilatation of dilated convolution, the receptive field of features can be increased without increasing the number of parameters, obtaining multi-scale information and sparse local features further away, consistent with the two-dimensional properties of the image. Both branches are added to the main branch via residual connections. By using MSLE_FF, the locality of the image can be modeled, enhancing the interaction of local information between image patches.

[0056] A schematic diagram of dilated convolution is shown below. Figure 5 As shown, the receptive field of a convolutional kernel can be expanded by introducing "dip" (zero padding) into the elements of the standard convolutional kernel without increasing the number of parameters. The figure shows that the same 3×3 convolution, by setting the dilation rate d = 2, can achieve the effect of a 5×5 convolution. The equivalent receptive field size of the dilated convolution is k′ = k + (k-1) × (d-1), while d is defaulted to 1 in ordinary convolution. Therefore, setting different dilation rates can yield features with different receptive field sizes.

[0057] The experiment sets the kernel size k=3 and the dilation rate d=3 for dilated convolution, which effectively obtains the receptive field of a 7×7 convolution. Simultaneously, mimicking depthwise separable convolution, channel-by-channel dilation is performed, significantly reducing the number of parameters and computational cost. Considering the locality of 2D images, the importance of features at different distances varies; therefore, the dilation rate d is used as an importance evaluation factor to weight the acquired features. The expression for the MSLE_FF module is shown below:

[0058]

[0059] Where "input" represents the input feature map of MSLE_FF, "output" represents the output of MSLE_FF, θ1 and θ2 represent the dropout parameters, FC represents a fully connected layer, and y i " represents the output of the i-th layer, DWDilated m conv n DWconv3(.) represents dilated convolution with an expansion rate of d = m and a kernel size of k = n. DWconv3(.) represents depthwise separable convolution with a kernel size of 3. GELU(.) represents the activation function.

[0060] Experiments and Analysis

[0061] 1. Dataset

[0062] Currently, most visual Transformer algorithms typically use ImageNet as their dataset, requiring more demanding graphics cards and computer equipment, resulting in long training cycles and high trial-and-error costs. This invention aims to validate the model's performance using the Mini-ImageNet dataset. Mini-ImageNet is a dataset created by the Google DeepMind team, extracting images from 100 categories of the ImageNet dataset, with 600 images per category, totaling 60,000 images. Since this dataset is designed for small sample sizes, its training set contains 64 categories, the validation set contains 16 categories, and the test set contains 20 categories; therefore, preprocessing of the dataset is necessary. By collecting images according to categories and dividing them into training and test sets in an 8:2 ratio, a dataset suitable for image classification was obtained. Mini-ImageNet has 100 categories, covering a diverse range of categories from various animals, portraits, food, and landscapes, with 48,000 images in the training set and 12,000 images in the test set.

[0063] In addition, to ensure the validity of the experiment, the CIFAR-10 dataset, commonly used in the field of image classification, was also used. The CIFAR-10 dataset contains 60,000 images across 10 categories, with 6,000 images per category. All images have a resolution of 32×32, with 50,000 images in the training set and 10,000 images in the test set.

[0064] 2. Evaluation Indicators

[0065] To evaluate the performance of PVT and the method of this invention, Top-1 accuracy, Top-5 accuracy, precision, recall, and F1 score were selected to assess the model performance.

[0066] Accuracy equals the proportion of correctly classified samples out of the total number of samples. Top-k accuracy means that if the model correctly classifies any of the top k categories predicted, the prediction is considered correct; otherwise, it is considered incorrect. Precision equals the proportion of samples predicted as positive that are actually positive. Recall equals the ratio of correctly predicted positive samples to all actually positive samples. Higher values ​​for these two metrics indicate better detection performance. Ideally, this value should be 1. However, these two metrics actually influence each other; if one is high, the other will be low. Typically, depending on the actual situation, one metric should be high while the other is as high as possible. Therefore, the overall performance of the model can be evaluated using the F1 score. The formulas for accuracy, recall, and F1 are as follows:

[0067]

[0068] Where TP represents the number of positive samples that were correctly detected, FP represents the number of negative samples that were incorrectly detected, and FN represents the number of positive samples that were incorrectly detected.

[0069] 3. Results and Analysis

[0070] This invention presents experiments on the Mini-ImageNet and CIFAR-10 datasets using the PVT algorithm, along with improved image patch embedding layers and feedforward neural network layers. An A100 GPU was used, and the deep learning framework employed was PyTorch 1.7.1, with Python version 3.7. Common training strategies for visual Transformers were employed, including the same data augmentation and regularization techniques used in DeiT to prevent overfitting, such as Mixup, RandAugment, and CutMix. The model was trained for 300 epochs using the AdamW optimizer with a weight decay of 0.05 and a learning rate of 5×10⁻⁶. -4 The Mini-ImageNet dataset has a batch size of 128, while the CIFAR-10 dataset has a batch size of 64.

[0071] 4. Experimental Results and Analysis

[0072] We conducted experimental verification in two groups, both of which used PVT as the baseline model.

[0073] The first set of experiments focused on the image patch embedding layer MSFF_PE, which incorporates multi-scale feature fusion. Different improvements were made to the image patch embedding layer based on different multi-scale feature fusion modules. The multi-scale feature fusion modules corresponding to MSFF_PE(a), MSFF_PE(b), MSFF_PE(c), and MSFF_PE(d) are Inception, Res2net, imp_Inception(a), and imp_Inception(b), respectively. Five experiments were conducted. The Top-1 accuracy, Top-5 accuracy, precision, recall, F1 score, and number of model parameters for each model are shown in Tables 1 and 2.

[0074] Table 1 Performance metrics of MSFF_PE on the Mini-ImageNet dataset

[0075]

[0076] Table 2 Performance metrics of MSFF_PE on the CIFAR-10 dataset

[0077]

[0078] As shown in Tables 1 and 2, the improved image patch embedding layer MSFF_PE using multi-scale feature fusion performs better on both the Mini-ImageNet and CIFAR-10 datasets, demonstrating the effectiveness of the proposed method for enhancing the multi-scale information contained within image patches. The introduction of convolution also strengthens the locality and translation invariance of the image. Among the four models, the model corresponding to MSFF_PE(c) has the best performance, namely the image patch embedding layer improved by imp_Inception(a). Furthermore, compared to the original Inception module, the improved lightweight Inception module has fewer parameters. Compared to the baseline model, MSFF_PE(c) increases the Top-1 accuracy by 1.11% and 0.54% on Mini-ImageNet and CIFAR-10, respectively. Therefore, this invention selects imp_Inception(a) as the multi-scale feature fusion module for MSFF_PE.

[0079] The second set of experiments targeted the multi-scale local augmentation feedforward neural network MSLE_FF, and was also evaluated on two datasets. The experimental results compared with the baseline model are shown in Tables 3 and 4.

[0080] As shown in Tables 3 and 4, the improved MSLE_FF model achieved enhanced performance on both the Mini-ImageNet and CIFAR-10 datasets, demonstrating that adding depthwise separable dilated convolutions strengthens information interaction between image patches and introduces multi-scale information. Compared to the baseline model, the Top-1 accuracy on Mini-ImageNet and CIFAR-10 increased by 4.36% and 2.61%, respectively. The feedforward neural network based on local enhancement achieves better performance than the image patch embedding layer based on multi-scale feature fusion, mainly because the image patch embedding layer only operates before the image enters the attention mechanism, while the improved feedforward neural network layer operates after each attention mechanism layer, thus achieving better results.

[0081] Table 3 Performance metrics of MSLE_FF on the Mini-ImageNet dataset

[0082]

[0083] Table 4 Performance metrics of MSLE_FF on the CIFAR-10 dataset

[0084]

[0085] 5. Experimental Results

[0086] We used PVT as the baseline model for our experiments, and added MSFF_PE and MSLE_FF to it. Since both improvements are based on multi-scale feature fusion and information interaction of image patches, we incorporated both improvements into the baseline model simultaneously, naming it MS-PVT. MSFF_PE used the best-performing imp_Inception(a), and the effectiveness of several multi-scale feature fusion modules was verified in subsequent ablation experiments. The Top-1 accuracy, Top-5 accuracy, precision, recall, F1 score, and number of model parameters for MS-PVT, the baseline model, MSFF_PE, and MSLE_FF are shown in Tables 5 and 6.

[0087] Table 5 Performance metrics of MS-PVT on the Mini-ImageNet dataset

[0088]

[0089] Table 6 Performance metrics of MS-PVT on the CIFAR-10 dataset

[0090]

[0091] Experimental results show that the proposed method outperforms the baseline model on most metrics. Compared to the baseline model, the number of model parameters only increases by 0.4M, while the Top-1 accuracy increases by 5.60% and 2.92% on the Mini-ImageNet and CIFAR datasets, respectively. By comparing the performance metrics of the baseline model and the improved MS-PVT model across various metrics, the effectiveness of the proposed method can be demonstrated.

Claims

1. A method for image block segmentation preprocessing using a visual Transformer, characterized in that, Includes the following steps: S1: Obtain several publicly labeled images as a training set, with one image serving as a training sample; S2: Construct and train an improved Transformer classification model. The improved Transformer classification model is an improvement on the existing Transformer classification model in the following two ways: First, the image patch embedding layer is changed to a multi-scale feature fusion image patch embedding layer MSFF_PE; second, the feedforward neural network is improved to a multi-scale local enhancement feedforward neural network layer MSLE_FF. In step S2, the process of inputting training samples into MSFF_PE and calculating the feature map corresponding to each training sample is as follows: （3） in, This indicates that the input image is MSFF_PE. This indicates a convolution with kernel size k, and MSFF represents the multi-scale feature fusion module imp_Inception(a). , The stride of each convolution is set to 2. The MSFF module does not change the size of the input feature map, therefore the output after the image patch embedding layer... "flatten" means to stretch the spatial dimensions of the feature, namely length H and width W, then we have Image blocks, It is a fully connected layer, therefore the output is... , which serves as the input to the attention layer; H, W, and C represent the height, width, and channel dimensions of the feature map of the input image, respectively, and D represents the dimension set by the hidden layer; The processing procedure of imp_Inception(a) is as follows: （1） Where input' represents the feature map input to imp_Inception(a), and output' represents the feature map output by imp_Inception(a). This indicates that a convolution with a kernel size of k is performed, and the number of channels in the convolution is 1 / 4 of the number of input channels. This indicates max pooling with a kernel size of 3. Indicates the output of the i-th branch; The training samples are input into the improved Transformer classification model, which passes through the improved image patch embedding layer MSFF_PE, the spatial reduction attention module, and the improved feedforward neural network MSLE_FF to continuously update the feature map. Finally, the final predicted classification result is output through the linear layer. The loss between the true label and the predicted label of the training sample is calculated, and the gradient of the parameters of each layer of the model is calculated using backpropagation. The model parameters are adjusted according to the calculated gradient to minimize the loss function. The above steps are iterated until the loss function is minimized, at which point the trained Transformer classification model is obtained. In step S2, the feature map is processed by MSLE_FF to obtain the updated feature map. The process is as follows: （4） in, This represents the feature map of the input MSLE_FF. This indicates the output of MSLE_FF. and This indicates the parameters for setting dropout; FC stands for fully connected layer. This represents the output of the i-th layer. This represents dilated convolution, with its dilation rate... convolution kernel DWconv3(.) represents a depthwise separable convolution with a kernel size of 3, and GELU(.) represents the activation function. S3: Input an image to be classified into the trained Transformer classification model. The trained Transformer classification model calculates the probability value of the image to be classified into each class and outputs the class corresponding to the highest probability value as the predicted class of the image to be classified.

Images