Vision Transform simplification method based on parameter sharing

Abstract

The invention discloses a Vision Transform simplification method based on parameter sharing. The Vision Transform simplification method comprises the following steps: step 1, simplifying a visual backbone model; 2, pre-training an image classification model; 3, training a target detection and semantic segmentation model; according to the method, a new parameter sharing MLP structure is designed to reduce parameters of a ViT model, half of parameters of an MLP block are reduced by performing parameter sharing on a mirror image structure of a linear layer, and the MLP block is combined with a proposed lightweight attention block to construct an efficient PSM-Former model.

Description

Technical Field

[0001] The present invention relates to the improvement of a Vision Transformer (ViT) model, and belongs to the field of computer vision and deep learning, and in particular to reducing the number of parameters of the ViT model through parameter sharing and a lightweight attention mechanism. Background Art

[0002] In recent years, the Vision Transformer (ViT) has achieved great success in various computer vision tasks. The excellent performance of the Transformer is mainly attributed to its multi-head self-attention (MHSA) module. Therefore, many people have improved the performance of the Vision Transformer (ViT) by improving the token mixer of the multi-head self-attention. Recently, some studies have found that even if the original attention module is completely replaced by simple operations such as Fourier transform or spatial multi-layer perceptron (MLP), variants of the ViT model still have competitive performance. On this basis, some scholars proposed a general architecture of the Vision Transformer, namely MetaFormer, which replaces the MHSA module with a parameter-free token mixer and shows good performance. In addition, ShiftViT uses partial shift operations to replace the MHSA module and achieves good results, which once again confirms the competitive performance of the MetaFormer architecture.

[0003] Although people have made a lot of efforts to improve the simplicity and efficiency of the ViT model and proposed a series of lightweight ViT models by reducing model parameters, the computational complexity and parameter count of the current model are still very high, and the key issue of balancing accuracy and efficiency has not yet been resolved. By analyzing the composition of the basic units in various ViT models, it is found that there are MLP modules in the structures of various ViT models, and the MLP modules occupy the main parameters of the ViT model. Figure 1 As shown in the figure, in the standard ViT model and the Pyramid Vision Transformer (PVT), the MLP module accounts for two-thirds of its total parameters. In the hybrid model of the high efficiency model (EMO), the MLP module even accounts for 81% of the parameters. The amount of parameters of the MLP module dominates the main parameters of most ViT models, but most of the current research focuses on the token mixer part, and less attention is paid to the MLP module. To this end, we focus on another major component of the ViT model, the MLP module, and try to find possible ways to simplify the MLP block, thereby reducing the parameters of the overall ViT model without sacrificing performance. This motivates the exploration of the structure of the MLP module and proposes a new MLP module based on parameter sharing. Summary of the invention

[0004] The MLP module consists of two linear layers with symmetrical structures. These two linear layers have similar parameter shapes and can be transposed to each other, such as Figure 2 The two 1×1 convolution blocks shown. Based on this observation, a parameter-sharing multilayer perceptron (PS-MLP) structure is proposed by sharing the parameters of the two linear layers. This design will reduce the parameters of the MLP module by half, thereby significantly reducing the parameters of the ViT model.

[0005] In addition, inspired by the bottleneck structure, a lightweight self-attention mechanism for spatial dimension compression and expansion, called scaled multi-head self-attention module (S-MHSA), is proposed to further refine the features and combine with parameter-sharing MLP to form the final PSM-Former structure. PSM-Former is a visual backbone network. Extensive experiments on multiple computer vision tasks show that the proposed PSM-Former achieves competitive performance compared with related state-of-the-art methods.

[0006] The technical solution of the present invention is a Vision Transformer simplification method based on parameter sharing, and the implementation steps of the method are as follows:

[0007] Step 1: Streamline the visual backbone model;

[0008] S11 streamlines the attention module of the ViT model: For an input image, the image is first normalized and then sent to the self-attention layer for global feature extraction. Unlike previous methods, the image is downsampled before entering the self-attention layer to reduce the size of the image, greatly reducing the number of tokens processed by the self-attention layer, thereby reducing the number of parameters and calculations required for the attention operation. After passing through the self-attention layer, the image is restored to its original size through the transposed convolution layer. At the same time, this operation can further extract local features of the image and enhance the visual representation of the model.

[0009] S12 Streamlined MLP module of ViT model: After the image passes through the attention module, it will pass through the MLP module. The attention module can realize the feature extraction and interaction of the image at the spatial level, while the MLP module can realize the interaction of the image in the channel dimension. Different from the conventional MLP module, the proposed MLP module based on parameter sharing can reduce the number of parameters by half with the help of parameter sharing technology. Specifically, a parameter W is constructed and itself and its transpose W are transposed. T They are respectively applied to the dimension increase layer and dimension reduction layer of the MLP module, and the same parameters are used in these two convolution operations, thereby greatly reducing the number of parameters.

[0010] S13 combines the streamlined attention module with the MLP module to construct the visual backbone model PSM-Former.

[0011] Step 2: Pre-train image classification model;

[0012] S21 Input: Training data from the ImageNet-1K dataset.

[0013] S22 image classification model training: PSM-Former is pre-trained on the ImageNet-1K dataset so that the model can accurately extract image features.

[0014] S23 Model Loss: Use the cross entropy loss function to train the model and report the accuracy of the model on the ImageNet-1K validation set.

[0015] S24 Visualization: Use the Grad-CAM tool to visualize the focus area of ​​the visual backbone model PSM-Former in an image.

[0016] S25 output: pre-trained image classification model.

[0017] Step 3: Train object detection and semantic segmentation models;

[0018] S31 uses the visual backbone model PSM-Former in step 2 as the backbone of the detector RetinaNet and Mask R-CNN, builds a new target detection model, and uses the ImageNet-1K pre-trained weights to initialize PSM-Former.

[0019] S32 uses the visual backbone model PSM-Former in step 2 as the backbone of the semantic segmentation head FPN, builds a new semantic segmentation model, and uses the ImageNet-1K pre-trained weights to initialize PSM-Former.

[0020] S33 trains the target detection and semantic segmentation models, and reports the target detection results of the model using RetinaNet on the COCO2017 dataset, the target detection and instance segmentation results using Mask R-CNN, and the semantic segmentation results using FPN on the ADE20K dataset.

[0021] Compared with the previous technology, this paper proposes a new parameter-sharing MLP module to reduce the parameters of the ViT model. This structure reduces half of the parameters of the MLP block by sharing the parameters of the mirror structure of the linear layer. It is combined with the proposed scaled multi-head self-attention module (S-MHSA) to construct an efficient PSM-Former model. The image first passes through the S-MHSA module to extract the spatial features of the image, then passes through the parameter-sharing MLP module for feature interaction in the channel dimension, and finally passes through a 1000-channel classification layer with Softmax to achieve the image classification task.

[0022] In order to verify the effectiveness of the present invention, semantic segmentation experiments were conducted on the ADE20K dataset, and object detection and instance segmentation experiments were conducted on the COCO2017 dataset. The detector RetinaNet, Mask R-CNN and semantic segmentation head FPN were connected to the visual backbone model PSM-Former to build the object detection and semantic segmentation models. The image is first extracted by PSM-Former, and then the object detection and semantic segmentation tasks are realized by RetinaNet (MaskR-CNN) or FPN module.

[0023] Scaled Multi-Head Self-Attention Module (S-MHSA);

[0024] In the traditional multi-head self-attention module, the number of module parameters will grow quadratically with the input image resolution, which limits the application of the ViT model in high-resolution visual tasks, such as object detection and semantic segmentation tasks. On the other hand, the self-attention module usually only focuses on the long-range dependencies between pixels, extracts global features and ignores local features. Therefore, the present invention proposes an S-MHSA module, which is used to reduce the number of model parameters on the one hand, and can further extract local features of the image on the other hand to enhance the visual feature representation.

[0025] Inspired by the bottleneck structure of the “compression-expansion” attention network, an efficient scaled multi-head self-attention module (S-MHSA) is constructed, such as Figure 4 As shown. For the input feature map, the downsampling module is first used to reduce the resolution to the original Where r is the downsampling factor. Then, the global feature representation is constructed through MHSA operation. Finally, the size of the feature map is restored to its original size using transposed convolution. The convolution kernel size of the transposed convolution is set to the same as the downsampling factor r, which makes the restored feature map have the same size as the original input feature map. Specifically, for the input feature map X∈R n×c , first downsample it to get Then linearly transform X1 into query key value Where n = H × W is the number of pixels, H and W represent the height and width of the image respectively, and c and c k 、c v are the dimensions of the input feature map, key (query), and value, respectively. Based on these, the self-attention module is implemented as follows:

[0026]

[0027] Where d is the dimension of the key (query) and Sofmtax(·) represents the activation function. Then, a 2D transposed convolution is used to restore the resolution of the feature map to its original size as follows:

[0028] Y=TPConv(C2)

[0029] where Y∈R n×c is the output of the S-MHSA module, which has the same size as the input feature X. TPConv(·) represents the transposed convolution operation.

[0030] In the above S-MHSA module, by downsampling the size of the input features, the number of parameters of the attention operation is reduced, which reduces the computational complexity of the model. At the same time, the transposed convolution operation can further extract the local features of the image, while the multi-head self-attention can capture the global features, which helps to make up for the limitation that the multi-head self-attention only focuses on long-range dependencies.

[0031] Parameter-sharing MLP module;

[0032] With the development of computer vision tasks, the required image resolution is getting higher and higher. For example, target detection and semantic segmentation tasks usually require larger image resolution than image classification. However, the number of parameters of the ViT model will increase quadratically with the input image resolution, so compared with the convolutional neural network, the ViT model has more parameters. Although many studies have initially reduced the number of parameters of the ViT model, most of the current research focuses on simplifying the token mixing part, and less attention is paid to the MLP module. According to the above analysis, the parameters of the MLP module occupy the main parameters of the ViT model. Therefore, using the structure of MLP, a new parameter sharing MLP module, namely PS-MLP, is proposed based on parameter sharing technology.

[0033] Typically, the MLP modules in the ViT model consist of linear layers, which are fully connected layers. However, these linear layers act like 1×1 convolutions, so 1×1 convolutions can be used to replace linear layers, which has also been proven by some studies. From the perspective of convolution, it is found that the parameters of the two convolutional layers used for channel expansion and compression have a transposed relationship and form a mirror structure, that is, when the parameters of one convolutional layer are transposed, the same shape as the parameters of the other convolutional layer will be obtained, such as Figure 2 Based on this observation, a PS-MLP module based on parameter sharing technique is proposed.

[0034] The traditional MLP block usually consists of two fully connected nonlinear activation layers and can be expressed as:

[0035] MLP(X)=GERU(XW1+b1)W2+b2

[0036] Where X∈R H×W×C is the input feature map, (where H, W, C represent the height, width and number of channels of the image respectively), W1∈R C×rC , W2∈R rC×C They represent the weights of the two linear layers, b1 and b2 are the bias terms of the two linear layers, r is the channel expansion factor, and GERU(·) is the activation function.

[0037] The proposed PS-MLP is similar to the structure of the reverse residual module to a certain extent, consisting of an expansion layer, a deep convolution layer, and a projection layer. The difference is that the expansion layer and the projection layer adopt a mirror structure, that is, the expansion and projection layers in the PS-MLP use a shared parameter matrix W∈R C×rC , (where C represents the number of image channels) Figure 2 As shown. Accordingly, a layer of depthwise convolution (DWConv) is added between the two convolutional layers. The entire operation process of PS-MLP can be expressed as:

[0038] PS-MLP(X)=DWConv(XW)W T

[0039] Where W∈R C×rC , W T ∈Rr C×C Denotes the parameters shared by two 1x1 convolutional layers, and r is the channel expansion factor. DWConv(·) represents depthwise separable convolution. To simplify the formula, the activation function layer and bias term in the original MPL are omitted. Since it involves operations on the convolution kernel, the underlying function torch.nn.function.conv2d() in the PyTorch library is used to implement the 1×1 convolutional layer.

[0040] Various variants of PSM-Former

[0041] Combining the above two modules, PSM-Former is constructed. Figure 4 As shown, given an input image of size H×W, H and W represent the height and width of the image respectively. The resolutions of the four stages of PSM-Former are and In each stage, multiple parameter sharing modules PS-Block are stacked sequentially for feature extraction while maintaining the same resolution as the input features. In order to produce a hierarchical representation, a patch embedding layer consisting of 3×3 convolutions is applied before each stage to reduce the size of the intermediate features (resolution downsampled by 2 times) and project them to a larger dimension (the number of channels is expanded by 2 times). The embedding dimensions of the four stages are 64, 128, 320, and 512, respectively. In these four stages, the expansion rate of the PS-MLP block is set to 4, and the downsampling rate of the S-MHSA module is set to 4, 2, 2, and 1, respectively. The PSM-Former model ends with a global average pooling layer and a 1000-channel classification layer with Softmax. The common configurations of the four stages of PSM-Former are as follows Figure 5 shown.

[0042] In order to meet different applications, different numbers of PS-Blocks are selected for the four stages of PSM-Former, and different variants of PSM-Former with parameter amounts ranging from 9.7M to 31.6M are constructed, namely, micro, small, medium and large versions of PSM-Former, namely PSM-Former-T, PSM-Former-S, PSM-Former-M and PSM-Former-L, respectively.

[0043] Training object detection and semantic segmentation models

[0044] PSM-Former is evaluated on the detection task of COCO dataset. Specifically, PSM-Former is used as the backbone of two commonly used detectors, RetinaNet and MaskR-CNN, and the weights pre-trained on ImageNet1K dataset are used to initialize PSM-Former. During the training process, AdamW optimizer is used, and the initial learning rate lr=5×10 -5 , with a batch size of 8. During training, all images were resized to have a short side of 800 pixels and a long side no larger than 1333 pixels.

[0045] ADE20K is selected to evaluate the proposed semantic segmentation model. PSM-Former is used as the backbone of the semantic segmentation head FPN, and the model performance is measured based on mIoU. The ImageNet-1K pre-trained weights are used to initialize the backbone, and the model is trained for 80K epochs with a batch size of 16, using the AdamW optimizer with an initial learning rate of lr=1×10 -4 During training, images are randomly resized and cropped to 512×512. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The figure shows the proportion of parameters of each component of various existing versions of the ViT model.

[0047] Figure 2 Schematic diagram of MLP structure with shared and non-shared parameters. The left side is the MLP structure with non-shared parameters, which has independent parameters M1 and W2; the right side is the MLP structure with shared parameters, which has the same parameters W and its transpose W T C is the number of characteristic channels, and r is the channel expansion coefficient.

[0048] Figure 3 Comparison chart of PSM-Former with other models in terms of accuracy, FLOPs, and number of parameters on the ImageNet-1K dataset. The size of each bubble is proportional to the number of model parameters.

[0049] Figure 4 This is an overview of the overall structure of the PSM-Former model.

[0050] Figure 5 This is a common parameter configuration diagram for the PSM-Former model.

[0051] Figure 6 The image classification results of different models on the ImageNet-1K dataset. All models were trained on the ImageNet-1K training set and the accuracy on the validation set was reported.

[0052] Figure 7 The target detection results using RetinaNet with different backbone networks on the COCOval2017 dataset and the target detection and instance segmentation results using Mask R-CNN are shown.

[0053] Figure 8 Results of semantic segmentation using FPN with different backbone networks on the ADE20K dataset.

[0054] Fig. 9 Visualization of the focus areas of different backbone models. DETAILED DESCRIPTION

[0055] Experiments verify the effectiveness of the proposed visual backbone network PSM-Former on three visual tasks: image classification on the ImageNet1K dataset, semantic segmentation on the ADE20K dataset, object detection and instance segmentation on the COCO2017 dataset, and compare them with a group of state-of-the-art related methods.

[0056] The classification results of PSM-Former and the comparison results on ImageNet-1K are shown in the figure. Figure 6 As shown. It can be seen that due to the use of parameter sharing in the model, the number of parameters of PSM-Former is greatly reduced compared with CNN or other MetaFormer-based models, but it still achieves high performance and competitiveness. For example, PSM-Former-M obtains 81.1% top-1 accuracy with only 22M parameters and 4.3G floating point operations (FLOPs), which is 1.3% higher than the representative ViT model DeiT-S at a similar number of parameters. In order to achieve similar accuracy, PVT-M needs to increase 100% of the parameters, i.e. 44M, while increasing the computational complexity by 56% (6.7G), while the accuracy is only improved by 0.1%. Even compared with PoolFormer, the model of the present invention still shows better performance. Among them, PoolFormer-S24 has an accuracy of 80.3% with 21M parameters and 3.4G FLOPs, while PSM-Former-M has an accuracy of 81.1% with the same number of parameters, which is nearly 1% higher than PoolFormer-S24.

[0057] The results of PSM-Former on the target detection and instance segmentation tasks on the COCO2017 dataset are as follows: Figure 7As shown. The study shows that when equipped with RetinaNet, PSM-Former-based models consistently outperform ResNet-based models as well as MetaFormer models such as PoolFormer and PVT. For example, PSM-Former-T achieves an AP of 36.6, significantly outperforming ResNet-18 (31.8AP). PSM-Former-T has a slight advantage over PoolFormer (36.2), but has fewer parameters (19.2M vs. 21.3M). At the same parameter count, PSM-Former-S is 2 points higher than PoolFormerS12 (38.4 vs. 36.2). Specifically, PSM-Former-S outperforms PoolFormer-S12 and ResNet-18 (bounding box AP is 37.9, 37.3, and 34.0, respectively). In summary, for COCO object detection and instance segmentation tasks, PSM-Former is consistently competitive, especially outperforming all ResNet-based counterparts.

[0058] The semantic segmentation results of different backbones using FPN on ADE20K are as follows Figure 8 As shown. It can be seen that the PSM-Former based model consistently outperforms the CNN based ResNet model, as well as the Transformer based PVT and PoolFormer models. For example, PSM-Former-S achieves 41.5% mIoU compared to ResNet-50 (36.7%), PVT-small (39.8%), and PoolFormer-S24 (40.3%), an improvement of 3.8, 1.7, and 1.2, respectively. On the other hand, PSM-Former-S uses only half the parameters (15.7M vs 32M) while achieving comparable performance to SwinT. These results show that the PSM-Former backbone network can achieve competitive performance in semantic segmentation tasks.

[0059] In order to better illustrate the effectiveness of the method of the present invention, the Grad-CAM tool is used to visualize the relevant areas of interest for different models. Fig. 9 As shown in Figure 2, CNN-based ResNet tends to focus on local regions, while Transformer-based models such as DeiT and PoolFormer prioritize global features. In contrast, PSM-Former shows the ability to perceive both local and global features, which confirms the effectiveness of the proposed PS-MLP and S-MHSA blocks.

Claims

1. A Vision Transformer simplification method based on parameter sharing, characterized in that: The implementation steps of this method are as follows: Step 1: Streamline the visual backbone model; S11 streamlines the attention module of the ViT model: for an input image, the image is first normalized and then sent to the self-attention layer for global feature extraction; The image will be downsampled before entering the self-attention layer to reduce the number of tokens processed by the self-attention layer. After passing through the self-attention layer, the image will be restored to its original size through the transposed convolution layer, while the local features of the image will be extracted to enhance the visual representation. S12 streamlines the MLP module of the ViT model: After passing through the attention module, the image will pass through the MLP module; the attention module realizes the feature extraction and interaction of the image at the spatial level, and the MLP module realizes the interaction of the image in the channel dimension; construct a parameter W and transpose it with itself W T They are applied to the dimension increase layer and dimension reduction layer of the MLP module respectively, and the same parameters are used in the two convolution operations to reduce the number of parameters; S13 combines the streamlined attention module with the MLP module to construct the visual backbone model PSM-Former; Step 2: Pre-train image classification model; S21 input: training data in the ImageNet-1K dataset; S22 Image classification model training: PSM-Former is pre-trained on the ImageNet-1K dataset so that the model can accurately extract image features; S23 Model Loss: Use the cross entropy loss function to train the model and report the accuracy of the model on the ImageNet-1K validation set; S24 Visualization: Use the Grad-CAM tool to visualize the focus area of ​​the visual backbone model PSM-Former in an image; S25 output: pre-trained image classification model; Step 3: Train object detection and semantic segmentation models; S31 uses the visual backbone model PSM-Former in step 2 as the backbone of the detector RetinaNet and Mask R-CNN, builds a new object detection model, and uses the ImageNet-1K pre-trained weights to initialize PSM-Former; S32 uses the visual backbone model PSM-Former in step 2 as the backbone of the semantic segmentation head FPN, builds a new semantic segmentation model, and uses the ImageNet-1K pre-trained weights to initialize PSM-Former; S33 trains the target detection and semantic segmentation models, and reports the target detection results of the target detection and semantic segmentation models on the COCO2017 dataset using RetinaNet, the target detection and instance segmentation results using MaskR-CNN, and the semantic segmentation results using FPN on the ADE20K dataset.

2. According to the parameter sharing-based Vision Transformer simplification method of claim 1, it is characterized in that: To construct the PSM-Former model, the image first passes through the S-MHSA module to extract the spatial features of the image, then passes through the parameter-sharing MLP module for feature interaction in the channel dimension, and finally passes through a 1000-channel classification layer with Softmax to achieve the image classification task.

3. The Vision Transformer simplification method based on parameter sharing according to claim 1, characterized in that: Semantic segmentation experiments are conducted on the ADE20K dataset, and object detection and instance segmentation experiments are conducted on the COCO2017 dataset. The detectors RetinaNet, Mask R-CNN, and semantic segmentation head FPN are connected to the visual backbone model PSM-Former to build object detection and semantic segmentation models. The image is first passed through PSM-Former for feature extraction, and then the object detection and semantic segmentation tasks are achieved through the RetinaNet or FPN module.

4. The Vision Transformer simplification method based on parameter sharing according to claim 1, characterized in that: Scaled multi-head self-attention module S-MHSA; For the input feature map, the downsampling module is first used to reduce the resolution to the original Where r is the downsampling factor; then, the global feature representation is constructed through MHSA operation; finally, the size of the feature map is restored to its original size using transposed convolution; wherein the convolution kernel size of the transposed convolution is set to be the same as the downsampling factor r, so that the restored feature map has the same size as the original input feature map; for the input feature map X∈R n×c , first downsample it to get Then linearly transform X1 into query key value Where n = H × W is the number of pixels, H and W represent the height and width of the image respectively, and c and c k 、c v are the dimensions of the input feature map, key, and value, respectively; based on these, the self-attention module is implemented as follows: Where d is the dimension of the key, Sofmtax(·) represents the activation function; the resolution of the feature map is restored to its original size using a two-dimensional transposed convolution, as shown below: Y=TPCov(X2) where Y∈R n×c is the output of the S-MHSA module, which has the same size as the input feature X; TConv(·) represents the transposed convolution operation.

5. The Vision Transformer simplification method based on parameter sharing according to claim 4, characterized in that: Parameter-sharing MLP module; The MLP module in the ViT model consists of linear layers replaced by 1×1 convolutions; the MLP module consists of two fully connected nonlinear activation layers, expressed as: MLP(X)=GERU(XW1+b1)W2+b2 Where X∈R H×W×C is the input feature map, where H, W, C represent the height, width and number of channels of the image respectively, W1∈R C×rC , W2∈R rC×C They represent the weights of the two linear layers respectively; b1 and b2 are the bias terms of the two linear layers, r is the channel expansion factor, and GERU(·) is the activation function; PS-MLP consists of an expansion layer, a deep convolution layer, and a projection layer. The expansion layer and the projection layer use a mirror structure, that is, the expansion and projection layers in PS-MLP use a shared parameter matrix W∈R C×rC , where C represents the number of image channels; a layer of deep convolution DWConv is added between the two convolutional layers; the entire operation process of PS-MLP is expressed as: PS-MLP(X)=DWConv(XW)W T Where W∈R C×rC , W T ∈R rC×C represents the parameters shared by two 1x1 convolutional layers, r is the channel expansion factor; DWConv(·) represents depthwise separable convolution; the activation function layer and bias term in the original MPL are omitted; since it involves the operation of the convolution kernel, the underlying function torch.nn.function.conv2d() in the PyTorch library is used to implement the 1×1 convolutional layer.

6. The Vision Transformer simplification method based on parameter sharing according to claim 5, characterized in that: Various variants of PSM-Former; PSM-Former is constructed by combining the above-mentioned scaled multi-head self-attention module S-MHSA and parameter-sharing MLP module. Given an input image of size H×W, where H and W represent the height and width of the image respectively, the resolutions of the four stages of PSM-Former are and In each stage, multiple parameter-sharing modules PS-Block are stacked sequentially for feature extraction while maintaining the same resolution as the input features; to produce a hierarchical representation, a patch embedding layer consisting of 3×3 convolutions is applied before each stage; the embedding dimensions of the four stages are 64, 128, 320, and 512, respectively; in these four stages, the expansion rate of the PS-MLP block is set to 4, and the downsampling rate of the S-MHSA module is set to 4, 2, 2, and 1, respectively; the PSM-Former model ends with a global average pooling layer and a 1000-channel classification layer with Softmax; We select different numbers of PS-Blocks for the four stages of PSM-Former and construct different variants of PSM-Former with parameter sizes ranging from 9.7M to 31.6M, namely, micro, small, medium, and large versions of PSM-Former, namely PSM-Former-T, PSM-Former-S, PSM-Former-M, and PSM-Former-L, respectively.

7. The Vision Transformer simplification method based on parameter sharing according to claim 1, characterized in that: Train object detection and semantic segmentation models; We evaluate PSM-Former on the detection task of the COCO dataset. Specifically, PSM-Former is used as the backbone of two commonly used detectors, RetinaNet and MaskR-CNN. The weights pre-trained on the ImageNet1K dataset are used to initialize PSM-Former. During the training process, the AdamW optimizer is used with an initial learning rate of lr = 5×10 -5 , the batch size is 8; during training, all images are resized to have a short side of 800 pixels and a long side no larger than 1333 pixels; ADE20K is selected to evaluate the proposed semantic segmentation model. PSM-Former is used as the backbone of the semantic segmentation head FPN, and the model performance is measured based on mIoU. The ImageNet-1K pre-trained weights are used to initialize the backbone, and the model is trained for 80K epochs with a batch size of 16, using the AdamW optimizer with an initial learning rate of lr=1×10 -4 ; During training, images are randomly resized and cropped to 512×512.

Images