A vision transformer simplification method based on parameter sharing

Abstract

The application discloses a Vision Transformer simplification method based on parameter sharing, step one: simplifying a visual backbone model; step two: pre-training an image classification model; and step three: training a target detection and semantic segmentation model; the application designs a new parameter sharing MLP structure to reduce the parameters of a ViT model, half of the parameters of an MLP block are reduced by performing parameter sharing on a mirror structure of a linear layer, the mirror structure of the linear layer is combined with a lightweight attention block to construct an efficient PSM-Former model.

Description

Technical Field

[0001] This invention relates to improvements to the Vision Transformer (ViT) model, belonging to the fields of computer vision and deep learning, and particularly to reducing the number of parameters in the ViT model through parameter sharing and lightweight attention mechanisms. Background Technology

[0002] In recent years, the Vision Transformer (ViT) has achieved great success in various computer vision tasks. The excellent performance of the Transformer is mainly attributed to its Multi-Head Self-Attention (MHSA) module. Therefore, many researchers have improved the performance of the Vision Transformer (ViT) by modifying the token mixer of the MHSA module. Recently, some studies have found that even when the original attention module is completely replaced by simple operations such as Fourier Transform or Spatial Multi-Layer Perceptron (MLP), variants of the ViT model still maintain competitive performance. Building on this, some researchers have proposed a general architecture for the Vision Transformer, MetaFormer, which replaces the MHSA module with a parameterless token mixer and demonstrates good performance. Furthermore, ShiftViT uses partial shift operations instead of the MHSA module and achieves good results, further confirming the competitive performance of the MetaFormer architecture.

[0003] Despite significant efforts to improve the simplicity and efficiency of ViT models, and the development of lightweight ViT models by reducing model parameters, the computational complexity and parameter count of current models remain high, and the key issue of balancing accuracy and efficiency remains unresolved. Analysis of the composition of basic units in various ViT models reveals the presence of MLP modules in their structures, which constitute the majority of the ViT model's parameters. For example... Figure 1 As shown, in the standard ViT model and the Pyramid Vision Transformer (PVT), the MLP module accounts for two-thirds of the total parameters. In the hybrid model of the Efficient Model (EMO), the MLP module even accounts for 81% of the parameters. The number of parameters in the MLP module dominates the main parameters of most ViT models, but most current research focuses on the token mixer part, with less attention paid to the MLP module. Therefore, this paper focuses on another major component of the ViT model—the MLP module—and attempts to find possible ways to simplify the MLP block, thereby reducing the overall parameters of the ViT model without sacrificing performance. This has led to the exploration of the structure of the MLP module and the proposal of a novel MLP module based on parameter sharing. SUMMARY

[0004] The MLP module is composed of two linear layers with symmetric structure. The two linear layers have similar parameter shapes and can transpose each other, i.e. two 1x1 convolutional blocks as shown in Figure 2 According to this observation, a parameter sharing multi-layer perceptron (PS-MLP) structure is proposed by sharing the parameters of the two linear layers. This design will reduce the parameters of the MLP module by half, thereby significantly reducing the parameters of the ViT model.

[0005] In addition, inspired by the bottleneck structure, a lightweight self-attention mechanism for spatial dimension compression and expansion is proposed, called scaled multi-head self-attention module (S-MHSA), to further refine the features and combine with the parameter sharing MLP to form the final PSM-Former structure. PSM-Former is a visual backbone network. A large number of experiments in multiple computer vision tasks show that the proposed PSM-Former achieves competitive performance compared with related state-of-the-art methods.

[0006] The technical scheme of the present application is a Vision Transformer simplification method based on parameter sharing. The implementation steps of the method are as follows:

[0007] Step one: simplify the visual backbone model;

[0008] S11: Simplify the attention module of the ViT model: For an input image, first normalize the image, and then send the image to the self-attention layer for global feature extraction. Unlike previous methods, the image will be downsampled before entering the self-attention layer, reducing the size of the image and greatly reducing the number of Tokens processed by the self-attention layer, thereby reducing the parameter quantity and computational complexity required for attention operation. After the image passes through the self-attention layer, it is restored to its original size by the transpose convolution layer, which can further extract local features of the image and enhance the visual representation of the model.

[0009] S12: Simplify the MLP module of the ViT model: After the image passes through the attention module, it will pass through the MLP module. The attention module can realize feature extraction and interaction of the image in the spatial dimension, while the MLP module can realize interaction of the image in the channel dimension. Unlike the conventional MLP module, the proposed MLP module based on parameter sharing can reduce the parameter quantity by half with the help of parameter sharing technology. Specifically, a parameter W is constructed, and itself and its transpose W T are applied to the dimension increasing layer and the dimension decreasing layer of the MLP module respectively, using the same parameter in the two convolution operations, thereby greatly reducing the parameter quantity.

[0010] S13 combines the attention module and the MLP module after simplification to build a visual backbone model PSM-Former.

[0011] Step two: pre-training an image classification model

[0012] S21 input: training data in the ImageNet-1K dataset.

[0013] S22 image classification model training: pre-training the PSM-Former on the ImageNet-1K dataset so that the model can accurately extract image features.

[0014] S23 model loss: training the model using the cross-entropy loss function and reporting the model's accuracy on the ImageNet-1K validation set.

[0015] S24 visualization: using the Grad-CAM tool to visualize the attention area of the visual backbone model PSM-Former in an image.

[0016] S25 output: pre-trained image classification model.

[0017] Step three: training target detection and semantic segmentation models

[0018] S31 uses the visual backbone model PSM-Former in step two as the backbone of the detector RetinaNet and Mask R-CNN to build a new target detection model, and uses the ImageNet-1K pre-trained weights to initialize PSM-Former.

[0019] S32 uses the visual backbone model PSM-Former in step two as the backbone of the semantic segmentation head FPN to build a new semantic segmentation model, and uses the ImageNet-1K pre-trained weights to initialize PSM-Former.

[0020] S33 trains the target detection and semantic segmentation models, and reports the target detection results using RetinaNet, the target detection and instance segmentation results using Mask R-CNN, and the semantic segmentation results using FPN on the COCO2017 dataset.

[0021] Compared with the prior art, the application proposes a new parameter-shared MLP module to reduce the parameters of the ViT model, which reduces half of the parameters of the MLP block by parameter sharing of the mirror structure of the linear layer. Combined with the proposed scale multi-head self-attention module (S-MHSA), an efficient PSM-Former model is constructed. The image first passes through the S-MHSA module to extract the spatial features of the image, then passes through the parameter-shared MLP module for feature interaction in the channel dimension, and finally passes through a 1000-channel classification layer with Softmax to realize the image classification task.

[0022] In order to verify the effectiveness of the application, semantic segmentation experiments are carried out on the ADE20K dataset, and target detection and instance segmentation experiments are carried out on the COCO2017 dataset. The visual backbone model PSM-Former is connected with the detector RetinaNet, Mask R-CNN and the semantic segmentation head FPN respectively to construct the target detection and semantic segmentation model. The image first passes through the PSM-Former to extract the features, and then passes through the RetinaNet (MaskR-CNN) or FPN module to realize the target detection and semantic segmentation task.

[0023] Scale multi-head self-attention module (S-MHSA);

[0024] In the traditional multi-head self-attention module, the parameter quantity of the module increases with the input image resolution at a quadratic rate, which limits the application of the ViT model in high-resolution visual tasks such as target detection and semantic segmentation tasks. On the other hand, the self-attention module usually only focuses on the long-range dependency between pixels, extracts global features and ignores local features. Therefore, the S-MHSA module is proposed, which can reduce the parameter quantity of the model on the one hand, and further extract local features of the image on the other hand to enhance the feature representation of the vision.

[0025] Inspired by the bottleneck structure of the "compression-expansion" attention network, an efficient scale multi-head self-attention module (S-MHSA) is constructed, as shown in Figure 4 For the input feature map, first use the down-sampling module to reduce the resolution to , where r is the down-sampling factor. Then, through the MHSA operation, the global feature representation is constructed. Finally, the size of the feature map is restored to the original size using the transposed convolution. Among them, the convolution kernel size of the transposed convolution is set to be the same as the down-sampling factor r, which makes the restored feature map have the same size as the original input feature map. Specifically, for the input feature map X ∈ R n×c , first down-sample it to get Then linearly transform X1 into query Key Value where n = H x W is the number of pixels, H, W represent the height and width of the image respectively, c, c k , c v are the dimensions of input feature map, key (query) and value respectively. Based on these, the self-attention module is implemented as follows:

[0026]

[0027] where d is the dimension of key (query), Sofmtax(·) represents the activation function. Then, the resolution of the feature map is restored to the original size using two-dimensional transpose convolution as follows:

[0028] Y = TPConv(C2)

[0029] where Y ∈ R n×c is the output of the S-MHSA module, whose size is the same as the input feature X. TPConv(·) represents the transpose convolution operation.

[0030] In the above S-MHSA module, by downsampling the size of the input feature, the parameter quantity of the attention operation is reduced and the computational complexity of the model is reduced. At the same time, the transpose convolution operation can further extract the local features of the image, while the multi-head self-attention can capture the global features, which helps to make up for the limitation of multi-head self-attention only focusing on long-range dependencies.

[0031] Parameter-shared MLP module;

[0032] With the development of computer vision tasks, the required image resolution is getting higher and higher, such as target detection and semantic segmentation tasks usually require larger image resolution than image classification. However, the parameter quantity of the ViT model will grow at a quadratic rate with the input image resolution, so compared with the convolutional neural network, the ViT model has more parameters. Although there have been many studies to preliminarily reduce the parameter quantity of the ViT model, most of the current researches focus on the simplification of the token mixing part, and less attention is paid to the MLP module. According to the above analysis, the parameters of the MLP module account for the main parameters of the ViT model. Therefore, based on the structure of MLP and the parameter sharing technology, a new parameter-shared MLP module, namely PS-MLP, is proposed.

[0033] Generally, the MLP module in ViT model is composed of linear layers, which are fully connected layers. However, the role of these linear layers is similar to 1x1 convolution, so 1x1 convolution can be used instead of linear layer, which is also proved by some researches. From the perspective of convolution, it is found that the parameters of the two convolution layers used for channel expansion and compression have a transposed relationship and form a mirror structure, that is, when the parameters of one convolution layer are transposed, the same shape as the parameters of the other convolution layer will be obtained, as shown in Figure 2 Based on this observation, a PS-MLP module based on parameter sharing technology is proposed.

[0034] The traditional MLP block is usually composed of two fully connected nonlinear activation layers, which can be represented as:

[0035] MLP(X)=GERU(XW1+b1)W2+b2

[0036] Where X∈R H×W×C is the input feature map, (where H, W, C represent the height, width and channel number of the image respectively), W1∈R C×rC , W2∈R rC×C represent the weights of the two linear layers respectively. b1 and b2 are the bias terms of the two linear layers, and r is the channel expansion factor. GERU(·) is an activation function.

[0037] The proposed PS-MLP is similar to the structure of the inverse residual module to some extent, which is composed of an expansion layer, a deep convolution layer and a projection layer. Unlike this, the mirror structure is adopted for the expansion layer and the projection layer, that is, the two convolution layers of expansion and projection in PS-MLP use a shared parameter matrix W∈R C×rC , (where C represents the number of image channels respectively) as shown in Figure 2 Accordingly, a layer of deep convolution (DWConv) is also added between the two convolution layers. The whole running process of PS-MLP can be represented as:

[0038] PS-MLP(X)=DWConv(XW)W T

[0039] Where W∈R C×rC , W T ∈Rr C×C represent the parameters shared by the two 1x1 convolution layers, and r is the channel expansion factor. DWConv(·) represents a depth separable convolution. In order to simplify the formula, the activation function layer and the bias term in the original MPL are omitted. Since it involves the operation of convolution kernel, the 1x1 convolution layer is implemented using the underlying function torch.nn.function.conv2d() in the PyTorch library.

[0040] Variants of PSM-Former

[0041] Combining the two modules described above, PSM-Former is constructed. As shown in Figure 4 , given an input image of size HxW, H, W represent the height and width of the image respectively, the resolutions of the four stages of PSM-Former are and At each stage, multiple parameter sharing modules PS-Block are sequentially stacked for feature extraction while maintaining the same resolution as the input feature. To produce a hierarchical representation, a patch embedding layer consisting of a 3x3 convolution is applied before each stage to reduce the size of the intermediate feature (resolution down-sampling by 2) and project it to a larger dimension (channel number expanded by 2). The embedding dimensions of the four stages are 64, 128, 320, and 512, respectively. In the four stages, the expansion rate of the PS-MLP block is set to 4, and the down-sampling rate of the S-MHSA module is set to 4, 2, 2, and 1, respectively. The PSM-Former model ends with a global average pooling layer and a 1000-channel classification layer with Softmax. The common configuration of the four stages of PSM-Former is shown in Figure 5 .

[0042] To meet different applications, different numbers of PS-Block are selected for the four stages of PSM-Former, and different variants of PSM-Former with parameter quantities from 9.7M to 31.6M are constructed, namely the micro, small, medium, and large versions of PSM-Former, PSM-Former-T, PSM-Former-S, PSM-Former-M, and PSM-Former-L, respectively.

[0043] Training target detection and semantic segmentation models

[0044] Evaluate PSM-Former on the detection task on the COCO dataset. Specifically, PSM-Former is used as the backbone of two commonly used detectors, RetinaNet and MaskR-CNN, and the pre-trained weights of ImageNet1K dataset are used to initialize PSM-Former. During training, the AdamW optimizer is used, the initial learning rate lr=5x10 -5 , and the batch size is 8. During training, all images are adjusted to have a short side of 800 pixels and a long side of no more than 1333 pixels.

[0045] ADE20K is chosen to evaluate the proposed semantic segmentation model. PSM-Former is used as the backbone of the semantic segmentation head FPN, and the model performance is measured according to mIoU. ImageNet-1K pre-trained weights are used to initialize the backbone, and the model is trained for 80K epochs with an AdamW optimizer, an initial learning rate of lr = 1 × 10 -4 During training, the image is randomly resized and cropped to 512 × 512. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The proportion of the parameter amount of each component of the existing various versions of the ViT model is counted.

[0047] Figure 2 The MLP structure with shared and non-shared parameters is shown. The left side is the MLP structure with non-shared parameters, which has independent parameters M1 and W2; the right side is the MLP structure with shared parameters, which has the same parameters W and its transpose W T C is the number of feature channels, and r is the channel expansion coefficient.

[0048] Figure 3 The comparison chart of PSM-Former and other models in terms of accuracy, FLOPs and parameter amount on ImageNet-1K dataset is shown. The size of each bubble is proportional to the number of model parameters.

[0049] Figure 4 The overall structure of the PSM-Former model is shown.

[0050] Figure 5 The commonly used parameter configuration chart of the PSM-Former model is shown.

[0051] Figure 6 The image classification results of different models on ImageNet-1K dataset are shown. All models are trained on ImageNet-1K training set, and the accuracy on the validation set is reported.

[0052] Figure 7 The target detection results of different backbone networks using RetinaNet and the target detection and instance segmentation results of Mask R-CNN on COCOval2017 dataset are shown.

[0053] Figure 8 The results of semantic segmentation using FPN on different backbone networks on ADE20K dataset are shown.

[0054] Figure 9 The visualization chart of the attention area of different backbone models is shown. DETAILED DESCRIPTION

[0055] The effectiveness of the proposed visual backbone network PSM-Former on three visual tasks is verified through experiments: image classification on the ImageNet1K dataset, semantic segmentation on the ADE20K dataset, and target detection and instance segmentation on the COCO2017 dataset, and compared with a group of most advanced related methods.

[0056] The classification results of PSM-Former and the comparison results on ImageNet-1K are as shown in Figure 6 It can be seen that due to the use of parameter sharing in the model, the parameter amount of PSM-Former is greatly reduced compared with CNN or other MetaFormer-based models, but still achieves higher performance and competitiveness. For example, PSM-Former-M obtains a top-1 accuracy of 81.1% with only 22M parameters and 4.3G floating point operations (FLOPs), which is 1.3% higher than DeiT-S, a representative ViT model, under similar parameter amount. In order to achieve similar accuracy, PVT-M needs to increase 100% of the parameters, i.e. 44M, and increase 56% of the computational complexity (6.7G), while the accuracy only increases by 0.1%. Even compared with PoolFormer, the model of the present application still shows better performance. Among them, PoolFormer-S24 has an accuracy of 80.3% with 21M parameters and 3.4G FLOPs, while PSM-Former-M has an accuracy of 81.1% with the same number of parameters, which is nearly 1% higher than PoolFormer-S24.

[0057] The target detection and instance segmentation task results of PSM-Former on the COCO2017 dataset are as shown in Figure 7The results of semantic segmentation on ADE20K using FPN are shown in Table 2.

[0058] The results of semantic segmentation on ADE20K using FPN are shown in Table 2. Figure 8 As can be seen, the PSM-Former-based models are always superior to the CNN-based ResNet models, as well as the Transformer-based PVT and PoolFormer models. For example, PSM-Former-S achieves 41.5% mIoU, which is 3.8, 1.7 and 1.2 higher than ResNet-50 (36.7%), PVT-small (39.8%) and PoolFormer-S24 (40.3%), respectively. On the other hand, PSM-Former-S uses only half the parameters (15.7M vs 32M) while performing on par with SwinT. These results show that PSM-Former backbone networks can achieve competitive performance in semantic segmentation tasks.

[0059] To better illustrate the effectiveness of the method of the present application, the Grad-CAM tool is used to visualize the relevant areas of attention of different models. As shown in Table 3, Figure 9 As can be seen, the PSM-Former-based models are always superior to the CNN-based ResNet models, as well as the Transformer-based PVT and PoolFormer models. For example, PSM-Former-S achieves 41.5% mIoU, which is 3.8, 1.7 and 1.2 higher than ResNet-50 (36.7%), PVT-small (39.8%) and PoolFormer-S24 (40.3%), respectively. On the other hand, PSM-Former-S uses only half the parameters (15.7M vs 32M) while performing on par with SwinT. These results show that PSM-Former backbone networks can achieve competitive performance in semantic segmentation tasks.

[0059] To better illustrate the effectiveness of the method of the present application, the Grad-CAM tool is used to visualize the relevant areas of attention of different models. As shown in Table 3, Figure 9 As can be seen, the PSM-Former-based models are always superior to the CNN-based ResNet models, as well as the Transformer-based PVT and PoolFormer models. For example, PSM-Former-S achieves 41.5% mIoU, which is 3.8, 1.7 and 1.2 higher than ResNet-50 (36.7%), PVT-small (39.8%) and PoolFormer-S24 (40.3%), respectively. On the other hand, PSM-Former-S uses only half the parameters (15.7M vs 32M) while performing on par with SwinT. These results show that PSM-Former backbone networks can achieve competitive performance in semantic segmentation tasks.

Claims

1. A parameter sharing based Vision Transformer simplification method, characterized in that, The implementation steps of the method are as follows: Step one: simplify the visual backbone model; S11: Simplify the attention module of the ViT model: for an input image, first normalize the image, and then input the image into the self-attention layer for global feature extraction; Before the image enters the self-attention layer, it will be down-sampled to reduce the number of tokens processed by the self-attention layer; after the image passes through the self-attention layer, it is restored to the original size through the transpose convolution layer, while extracting the local features of the image and enhancing the visual representation; S12: Simplifying the MLP module of the ViT model: After passing through the attention module, the image will pass through the MLP module; the attention module realizes the feature extraction and interaction of the image in the spatial dimension, and the MLP module realizes the interaction of the image in the channel dimension; a parameter and its transpose are applied to the dimension increasing layer and the dimension decreasing layer of the MLP module respectively, and the same parameter is used in the two convolution operations, thereby reducing the parameter quantity. S13: Combine the simplified attention module and the MLP module to construct the visual backbone model PSM-Former; Step two: pre-training the image classification model; S21: input: training data in the ImageNet-1K dataset; S22: image classification model training: pre-training the PSM-Former on the ImageNet-1K dataset to accurately extract image features; S23: model loss: use the cross-entropy loss function to train the model and report the accuracy of the model on the ImageNet-1K validation set; S24: visualization: use the Grad-CAM tool to visualize the attention area of the visual backbone model PSM-Former in an image; S25: output: pre-trained image classification model; Step three: train the target detection and semantic segmentation model; S31: use the visual backbone model PSM-Former in step two as the backbone of the detector RetinaNet and Mask R-CNN to construct a new target detection model, and use the ImageNet-1K pre-trained weight to initialize PSM-Former; S32: use the visual backbone model PSM-Former in step two as the backbone of the semantic segmentation head FPN to construct a new semantic segmentation model, and use the ImageNet-1K pre-trained weight to initialize PSM-Former; S33: train the target detection and semantic segmentation model, and report the target detection results of RetinaNet, the target detection and instance segmentation results of Mask R-CNN, and the semantic segmentation results of FPN on the COCO2017 dataset.

2. The method of claim 1, wherein, The PSM-Former model is constructed. An image first passes through an S-MHSA module to extract spatial features of the image, then passes through a parameter-shared MLP module for channel-dimension feature interaction, and finally passes through a 1000-channel classification layer with to implement an image classification task.

3. The method of claim 1, wherein, Perform semantic segmentation experiments on the ADE20K dataset, target detection and instance segmentation experiments on the COCO2017 dataset; connect the detector RetinaNet, Mask R-CNN and semantic segmentation head FPN behind the visual backbone model PSM-Former to construct a target detection and semantic segmentation model; the image first passes through PSM-Former to extract features, and then realizes the target detection and semantic segmentation task through RetinaNet or FPN module.

4. The method of claim 1, wherein, Scale multi-head self-attention module S-MHSA; For the input feature map, first use the down-sampling module to reduce the resolution to the original , where is the down-sampling factor; then, by MHSA operation, construct the global feature representation; finally, use the transpose convolution to restore the size of the feature map to the original size; wherein the convolution kernel size of the transpose convolution is set to be the same as the down-sampling factor , so that the restored feature map has the same size as the original input feature map; for the input feature map , first down-sample it to get , then linearly transform to query , key , value , wherein is the number of pixels, respectively represent the height and width of the image, , , respectively represent the dimensions of the input feature map, key and value; based on these, the self-attention module is implemented as follows: ， wherein is the dimensionality of the bond, denotes the activation function; the resolution of the feature map is restored to the original size using a two-dimensional transpose convolution, as follows: ， wherein is the output of the S-MHSA module, which has the same size as the input features ; denotes a transposed convolution operation.

5. The method of claim 4, wherein, Parameter sharing MLP module; The MLP module in the ViT model consists of linear layers, replaced by convolutions The MLP module consists of two fully connected, nonlinear activation layers, denoted as: ， wherein, is the input feature map, wherein respectively represent the height, width and number of channels of the image, , respectively represent the weights of the two linear layers; and are the bias terms of the two linear layers, * is a channel expansion factor, is an activation function; The PS-MLP is composed of an expansion layer, a deep convolution layer and a projection layer; mirror structures are adopted for the expansion layer and the projection layer, that is, shared parameter matrices are used for the two layers of convolution in the expansion and projection of the PS-MLP , represents the number of image channels; a deep convolution DWConv layer is added between the two convolution layers; and the whole running process of the PS-MLP is represented as: ， wherein , denotes parameters shared by two 1x1 convolutional layers, * is a channel expansion factor; denotes a depthwise separable convolution; the activation function layer and the bias term in the original MLP are omitted; due to the operation involving the convolution kernel, the underlying function torch.nn.function.conv2d() in the PyTorch library is used to implement the convolutional layer.

6. The method of claim 5, wherein, Various variants of PSM-Former; Combining the above two modules of scale multi-head self-attention module S-MHSA and parameter sharing MLP module, a PSM-Former is constructed; given an input image of size , respectively represent the height and width of the image, the resolutions of the four stages of the PSM-Former are , , and respectively; in each stage, multiple parameter sharing modules PS-Block are sequentially stacked for feature extraction while maintaining the same resolution as the input features; in order to produce hierarchical representation, a patch embedding layer composed of a 3 convolution is applied before each stage; the embedding dimensions of the four stages are 64, 128, 320 and 512 respectively; in the four stages, the expansion rate of the PS-MLP block is set to 4, and the down-sampling rate of the S-MHSA module is set to 4, 2, 2 and 1 respectively; the PSM-Former model ends with a global average pooling layer and a 1000 channel classification layer with Softmax. Different numbers of PS-Blocks are selected for the four stages of the PSM-Former, and different variants of the PSM-Former with parameter quantities from 9.7M to 31.6M are constructed, i.e. the micro, small, medium and large versions of the PSM-Former, namely PSM-Former-T, PSM-Former-S, PSM-Former-M and PSM-Former-L.

7. The method of claim 1, wherein the Vision Transformer is simplified based on parameter sharing. training a target detection and semantic segmentation model; Evaluate the PSM-Former on the detection task on the COCO dataset; the PSM-Former as the backbone of two commonly used detectors, RetinaNet and Mask R-CNN, the weights pre-trained on the ImageNetlK dataset are used to initialize the PSM-Former; during the training process, the AdamW optimizer is used, the initial learning rate is 0.01, and the batch size is 8; during the training process, all images are adjusted to have a short side of 800 pixels and a long side of no more than 1333 pixels; ADE20K is selected to evaluate the proposed semantic segmentation model; PSM-Former is used as the backbone of the semantic segmentation head FPN, and the model performance is measured according to mIoU; the pre-trained weight of ImageNet-1K is used to initialize the backbone, and the model is trained for 80K epochs with a batch size of 16, using the AdamW optimizer with an initial learning rate of ; during the training process, the image is randomly resized and cropped to 512 × 512.

Images