Hyperspectral image classification method based on metaformer and deep dilated feature fusion network

Abstract

The application provides a hyperspectral image classification method based on a MetaFormer and a deep expansion feature fusion network, and mainly comprises the following steps: firstly, a deep expansion separation convolution (DDSC) is used to amplify the spectral dimension and fine difference features; then, a multi-scale pooling feature fusion module (MPFormer) is used for multi-scale space-spectrum information aggregation and extraction; in the extraction process, a front-back feature adaptive weighted fusion module (AWFM) is used to perform feature adaptive weighted fusion on low-level features obtained by a shallow MPFormer and high-level features obtained by a deep MPFormer; and finally, a softmax is used for classification to obtain a result. The method has sufficient competitiveness in classification accuracy, and has great advantages in parameter quantity compared with other same type models.

Description

Technical Field

[0001] This invention mainly relates to the field of image classification technology, specifically a hyperspectral image classification method based on MetaFormer and deep extended feature fusion network. Background Technology

[0002] Hyperspectral images (HSI) possess rich spectral information, offering unique advantages in accurate ground feature classification compared to ordinary images. Captured by a hyperspectral imager, hyperspectral images collect hundreds of spectral bands during imaging, unlike ordinary RGB images. These more spectral bands contain more information, aiding in accurate ground feature identification. Therefore, hyperspectral images have wide applications in agriculture, mineral exploration, environmental monitoring, and military reconnaissance.

[0003] Traditional hyperspectral classification methods often require manual extraction of features needed by the classifier. However, the extraction criteria for these features are subject to subjective factors, which is not conducive to the objectivity of the results.

[0004] Compared to traditional methods, deep learning-based methods have significant advantages in extracting deep image features. For example, a five-layer weighted 1D-CNN performs deep feature extraction and classification along the spectral dimension. However, hyperspectral images often exhibit phenomena such as "different objects with the same spectrum" and "different spectra with the same object," which are difficult to distinguish using only spectral channel information. Another approach is 2D-CNN, which uses a semi-supervised method to automatically extract hyperspectral image features. While 2D-CNN-based methods have certain advantages, hyperspectral images are often presented as 3D cubes, and using features from only a single dimension is insufficient to achieve satisfactory results.

[0005] The Transformer, which has recently made significant strides in NLP, has also been applied to image processing. Existing techniques include image segmentation followed by flattening and then using image-specific positional encoding as input for Visual Information Transformation (VIT). Other techniques involve grouping adjacent spectral bands as input and applying VIT to hyperspectral images; however, this mechanical, fixed grouping does not effectively represent the subtle differences between spectral channels. This latest technique introduces convolution into VIT for the first time, combining local and global information to achieve better classification results. For example, a spectral-spatial feature tokenization converter extracts low-order features through convolution and high-order features through VIT combined with Gaussian weights, thus achieving superior classification performance.

[0006] However, in existing technologies, VIT (vision transformer) achieves excellent accuracy by focusing on global information. But current mainstream methods often ignore subtle differences between spectra, leading to insufficient utilization of spectral information. These models still have some unresolved issues:

[0007] 1) Subtle spectral differences between near-continuous spectral channels often cannot be effectively extracted and utilized, making it difficult to effectively classify mixed pixels with high inter-class similarity.

[0008] 2) The VIT framework serialization results in significant differences between low-level and high-level features, which are not effectively combined, leading to insufficient feature utilization.

[0009] 3) The self-attention mechanism in VIT introduces high quadratic complexity, especially serialized VIT, which leads to too many model parameters. Summary of the Invention

[0010] To address the shortcomings of current technologies, this invention combines existing technologies and, based on practical applications, provides a hyperspectral image classification method based on MetaFormer and deep extended feature fusion networks. This method not only boasts sufficient competitiveness in classification accuracy but also offers a significant advantage in parameter quantity compared to other similar models.

[0011] The technical solution of this invention is as follows: a hyperspectral image classification method based on MetaFormer and deep extended feature fusion network, the method mainly includes the following steps:

[0012] First, the spectral dimension is amplified to enhance subtle differences through deep dilatation and separation convolution (DDSC). Then, multi-scale spatial spectral information is aggregated and extracted using the multi-scale pooling feature fusion module (MPFormer). During the extraction process, the low-level features obtained from the shallow MPFormer and the high-level features obtained from the deep MPFormer are adaptively weighted and fused using the front-to-back feature adaptive fusion module (AWFM). Finally, softmax is used for classification to obtain the results.

[0013] Furthermore, for Depth Dilated Separate Convolution (DDSC), firstly, a channel-wise convolution with a size of 3x3 and an output channel that is k times the number of input channels is used to amplify the channel dimension details, and then a point convolution with a size of 1x1 is used for channel fusion.

[0014] Furthermore, for Deep Dilated Separate Convolutional (DDSC), the input hyperspectral image is first processed... Select N patches from the given list, where X = [x1, x2, ..., x...]. N In this context, C, H, and W refer to the number of spectral channels, height, and width of the hyperspectral image, respectively, and N is the number of selected patches. The patches after depthwise dilation and separation convolution can be represented as:

[0015]

[0016] Where w is the trainable weight matrix of the convolution kernel, The input is the patches in HSI X, where i and j represent the coordinates of the pixel values ​​in x, k is the dilation factor, d is the channel dimension, and m and v are the two dimensions of the convolution kernel.

[0017] Then, pointwise convolution is used to fuse the features from each layer, x point To output patches, the formula can be expressed as:

[0018]

[0019] Where w is the trainable weight matrix of the convolution kernel, d is the channel dimension, and i and j represent the coordinates of the pixel values ​​in x.

[0020] Furthermore, the vector output by the depthwise dilatation separating convolution DDSC is... The process using MPFormer is as follows:

[0021] First, patch embedding is performed, followed by flattening and linear transformation. Where S = H × W, l is the dimension after linear transformation, and ClassToken X cls With X L X is obtained by splicing. LC The formula is as follows:

[0022] X LC =concat([X L1 ,X L2 ,...,X LN ],X cls (3)

[0023] The concat operation, which joins vectors or matrices, involves concatenating vectors or matrices, such as vector X. L1 ,X L2 ,...,X LN Let X be a vector L The amount

[0024] Then, features are extracted using the MPM and MLP modules, as shown in the following formula:

[0025]

[0026]

[0027] in X represents element-wise addition. MPF X′ represents the output of the intermediate process. MPFThis represents the final output of the MPFormer module. Norm indicates normalization processing, MLP stands for Multilayer Perceptron, and MPM indicates multi-scale feature fusion via MPMixer. The MPMixer module adapts to dense predictions in hyperspectral images by utilizing multi-scale AvgPooling and feature fusion, performing feature mixing without requiring any trainable parameters. Its formula is expressed as:

[0028]

[0029]

[0030] Where X MPM The multi-scale spatial spectral feature matrix s obtained after the MPMixer module represents the scale label of the multi-scale pooling. It is a mixture of feature matrices obtained by pooling operations at different scales, Avg s It is the average pooling scale, and p and q are the step size calculation variables that need to be manipulated in the pixel coordinates during the pooling operation.

[0031] Furthermore, the feature map obtained after passing through the multi-scale pooling feature fusion module MPFormer The shallow and deep feature maps are concatenated spatially using the adaptive weighted fusion module AWFM, and then adaptively extracted using 1×2 Conv, as shown in the following formula:

[0032] X AWF =Conv(Concat(X′) MPF1 ,X′ MPF2 (8)

[0033] Where X AWF X′ represents the output after adaptive feature extraction. MPF1 and X′ MPF2 This represents the feature matrix obtained after passing through different MPFormers. Conv represents the convolution operation, and Concat represents the concatenation operation of vectors or matrices.

[0034] Then X′ MPF1 With X′ MPF2 After adding the positions, multiply by the weight, and finally sum to X. AWF The final output X′ is obtained by fusion. AWF The formula is as follows:

[0035]

[0036] Where ω represents the normalized weight coefficient.

[0037] The beneficial effects of this invention are:

[0038] This invention employs DDSC adaptive amplification to refine spectral channel details, followed by MPFormer for feature extraction and processing, AWFM for feature fusion, and finally softmax for classification. A series of ablation comparison experiments demonstrate that the proposed hyperspectral image classification method not only possesses sufficient competitiveness in classification accuracy but also exhibits a significant advantage in parameter quantity compared to other similar models. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the overall framework of DDMN proposed in this invention.

[0040] Figure 2 This is a schematic diagram of an MPFormer and a multi-scale feature mixer.

[0041] Figure 3 This is a diagram of the adaptive weighted feature fusion module.

[0042] Figure 4 In the figure, (a) overall accuracy plots for different patch sizes in the IP dataset, (b) PU dataset, and (c) HS dataset.

[0043] Figure 5 The convergence speed of different models in terms of training accuracy over the first 50 epochs on the IP, PU, ​​and HS datasets is shown. Detailed Implementation

[0044] The present invention will be further described in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0045] This embodiment provides a hyperspectral image classification method based on MetaFormer and Deep Dilated Metaformer Network. The classification model used in the method is called DDMN (Deep Dilated Metaformer Network), which cleverly combines convolution and VIT (vision transformer).

[0046] refer to Figure 1 As shown, the overall framework of DDMN in this embodiment adopts MetaFormer, which is a general architecture abstracted from Transformer. In this embodiment, Transformer framework, MPMixer, DDSC and AWFM are the main modules that constitute DDMN, and the model is an improvement on VIT.

[0047] In this embodiment, the hyperspectral image (HIS) classification method first amplifies subtle differences in the spectral dimension through Depthwise Dilated Separable Convolution (DDSC). Then, it uses a serial multi-scale pooling feature fusion module (MPFormer) to aggregate and extract multi-scale spatial spectral information. During extraction, an adaptive weighted fusion module (AWFM) adaptively weights and fuses the low-level features obtained from the shallow MPFormer layer and the high-level features obtained from the deep MPFormer layer. Finally, softmax is used for classification to obtain the result. Notably, in this embodiment, the self-attention mechanism, which introduces secondary complexity in VIT, is replaced with a multi-scale feature aggregation module to reduce the number of trainable parameters in the model.

[0048] The DDMN proposed in this embodiment first expands the spectral channels of the convolutional layer through depth dilation, thereby amplifying subtle differences in spectral features. Then, feature extraction is performed using MKFormer, which requires very few parameters and effectively extracts fused features. Finally, a residual-like approach is used to adaptively fuse the low-level features extracted by the shallow MKFormer layer and the high-level features extracted by the deep layer for classification.

[0049] Unlike ordinary convolution, Depthwise Separable Convolution (DDSC) in this embodiment requires two steps: first, channel-wise convolution is performed on the target, and then pointwise convolution is performed on all channels to restore the dimensionality. Depthwise separable convolution can effectively reduce parameters, but often at the expense of some accuracy. For hyperspectral images, extremely similar ground features exhibit very subtle differences in nearly continuous spectral channels. DDSC's ability to extract these subtle spectral differences by expanding the channel dimension is crucial for distinguishing similar ground features. Channel-wise convolution can act on each spectral channel individually, reducing interference between channels and allowing each convolution kernel to focus more precisely on the corresponding channel features.

[0050] Specifically, this embodiment employs a 3x3 convolution with output channels that are k times the number of input channels to amplify the channel-dimensional details, and then uses a 1x1 pointwise convolution for channel fusion. From the input HIS... Select N patches from the given list, where X = [x1, x2, ..., x...]. NWhere C, H, and W refer to the number of spectral channels, height, and width of the HIS, respectively, and N is the number of selected patches. The patches after depthwise dilation and separation convolution can be represented as:

[0051]

[0052] Where w is the trainable weight matrix of the convolution kernel, It is the patches in the input HSI X (where i and j are the coordinates of pixels in x), i and j represent the coordinates of pixel values ​​in x, k is the dilation factor, d is the channel dimension, and m and v are the two dimensions of the convolution kernel;

[0053] In the following steps, pointwise convolutions are used to fuse features from each layer. A 1×1 pointwise convolution does not change the width and height of the input feature map and allows control over the number of channels in the output feature map. point To output patches, the formula can be expressed as:

[0054]

[0055] Where w is the trainable weight matrix of the convolution kernel, d is the channel dimension, and i and j represent the coordinates of the pixel values ​​in x.

[0056] By expanding the channel dimension by a factor of k through depthwise dilation convolution, subtle differences between channels can be amplified. Unlike ordinary convolution, which is subject to interference from adjacent channels, this is more conducive to feature expression. Then, point convolution is used to fuse the features between channels and restore the dimension of the feature map.

[0057] For the multi-scale pooling feature fusion module MPFormer in this embodiment, the vector output by the depth dilation separable convolution DDSC is... The process via MPFormer is as follows: Figure 2 As shown, patch embedding is performed first, followed by flattening and linear transformation. Where S = H × W, l is the dimension after linear transformation, and ClassToken X cls With X L X is obtained by splicing. LC The formula is as follows:

[0058] X LC =concat([X L1 ,X L2 ,...,X LN ],X cls (3)

[0059] In this embodiment, the positional parameters are discarded because ordinary images, when processed by VIT, are segmented into adjacent patches as input, and there are obvious spatial dependencies between adjacent patches. However, for dense prediction tasks like HIS classification, each input consists of multiple random patches in an image, with very few spatial dependencies.

[0060] Then, features are extracted using the MPM and MLP modules, as shown in the following formula:

[0061]

[0062]

[0063] in X represents element-wise addition. MPF X′ represents the output of the intermediate process. MPF The output of the MPFormer module is represented by "Norm", "MLP" indicates normalization, "MPM" indicates multi-layer perceptron, and "MPM" indicates multi-scale feature fusion via MPMixer. The MPMixer module replaces the original self-attention to adapt to dense predictions in hyperspectral images. This embodiment proposes the MPMixer module to adapt to dense predictions in HSI, such as... Figure 2 As shown, it utilizes multi-scale AvgPooling and performs feature fusion, achieving feature mixing without requiring any trainable parameters. Its formulas are expressed as equations (6) and (7):

[0064]

[0065]

[0066] Where X MPM The multi-scale spatial spectral feature matrix s obtained after the MPMixer module represents the scale label of the multi-scale pooling. It is a mixture of feature matrices obtained by pooling operations at different scales, Avg s It is the average pooling scale, and p and q are the step size calculation variables that need to be manipulated in the pixel coordinates during the pooling operation.

[0067] The adaptive weighted fusion module AWFM proposed in this embodiment is as follows: Figure 3 As shown, the feature map obtained after passing through the multi-scale pooling feature fusion module MPFormer The shallow and deep feature maps are concatenated spatially using the adaptive weighted fusion module AWFM, and then adaptively extracted using 1×2 Conv, as shown in the following formula:

[0068]

[0069] Where X AWF X′ represents the output after adaptive feature extraction. MPF1 and X′ MPF2 This represents the feature matrix obtained after passing through different MPFormers. Conv represents the convolution operation, and Concat represents the concatenation operation of vectors or matrices.

[0070] Then X′ MPF1 With X′ MPF2 After adding the positions, multiply by the weight, and finally sum to X. AWF The final output X′ is obtained by fusion. AWF The formula is as follows:

[0071]

[0072] Where ω represents the normalized weight coefficient.

[0073] To verify the advancement of the proposed method in this embodiment, experiments were conducted using the HSI datasets from Indian Pines (IP), Pavia University (PU), and Houston (HS). Details of the three datasets are shown in Table 1. To ensure fairness in the comparative experiments, this embodiment used challenging quantitative samples from SpectralFormer as training samples for model training and labeled samples for model testing.

[0074] Table 1. Standard training and test sets for each category of the Indian Pine Dataset, the University of Pavia Dataset, and the Houston 2013 Dataset.

[0075]

[0076] In addition, experiments were conducted using different numbers of training samples to verify the effectiveness of the proposed method. 2D-CNN, 3D-CNN, ViT, and SpectralFormer were used for comparison. In the relevant experiments, all parameters adopted specified metrics, and the epochs for the three DDMN datasets were set to 200 (IP), 320 (PU), and 300 (HS). The learning rate was 0.005, the patch size was 9×9, and the Adam optimizer was used to optimize the network structure. This embodiment selects three common evaluation metrics: overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Kappa) to evaluate network performance. The proposed CTM in this embodiment was implemented on the Ubuntu 18.04 platform, using the PyTorch 1.6.0 framework, a Platinum 8255C CPU, an RTX 3080 GPU, and 40GB of RAM.

[0077] 1. Ablation test

[0078] The DDMN method proposed in this embodiment comprises three main modules: DDSC, MPMixer, and AWFM. Since the model is based on an improved Transformer, VIT is chosen as the baseline method, and all hyperparameters are set identically to the original model. First, DDSC is used to amplify subtle differences between adjacent spectral channels, addressing the problem of insufficient spectral information extraction and utilization. Then, due to the significant differences in features between the shallow and deep layers of the proposed sequential network, AWFM is designed to adaptively fuse shallow and deep features through weighted fusion, enhancing information interaction between layers and reducing information loss. Finally, replacing self-attention with MPMixer in this embodiment significantly reduces the number of parameters without sacrificing classification accuracy. To verify the effectiveness of each module in the proposed DDMN method, a series of ablation experiments were conducted using the detailed network configurations described in Table 2 and HSI classification results. The effectiveness is demonstrated through ablation experiments using four different networks, as shown below.

[0079] NET1: VIT (attention)

[0080] NET2: DDSC + ViT (attention)

[0081] NET3:DDSC+AWFM+ViT(attention)

[0082] NET4:DDSC+AWFM+MPFormer(MPMixer)

[0083] The classification results of the above networks are shown in Table 2. Each module contributes to DDMN to varying degrees on different datasets. Taking the IP dataset as an example, NET1 has the lowest accuracy, indicating that ViT's classification performance is poor on challenging quantitative data. Comparing the results of NET2 and NET3, OA increased by 17.49%, AA by 11.89%, and the Kappa coefficient by 19.59%. Such a large increase is not only due to the addition of local spatial features after convolution, but also to the improved utilization of spectral information through depth expansion separating convolution. Comparing the accuracy of NET3 and NET2 shows that the full fusion of low-level and high-level features enhances the information interaction ability between shallow and deep layers. The Kappa coefficient shows that the impact of class imbalance is reduced and classification consistency is improved. The difference between NET4 and NET3 is that NET4 replaces the self-attention mechanism in ViT with MPMixer. From the perspective of classification accuracy, there is only a small improvement. However, it is worth noting that the number of parameters in NET4 is reduced by 36.36% year-on-year. Different modules have also made positive contributions on different datasets, thus proving the effectiveness of each module.

[0084] Table 2: Contribution of different components to CMT

[0085]

[0086]

[0087] 2. The impact of patch size on model accuracy

[0088] A comparative analysis was conducted on the proposed DDMN and three other Transformer-based algorithms, ViT, SpectralFormer, and SSFTT, with different input patch sizes. The results are as follows: Figure 4 As shown in Table 3, DDMN consistently demonstrates commendable classification accuracy across all three datasets while maintaining excellent stability. This achievement can be attributed to MPMixer's ability to effectively fuse multi-scale neighborhood-shared features. Furthermore, this embodiment compares and analyzes the training parameters and runtime (including both training and testing phases) for various datasets with different patch sizes, with results shown in Table 3. DDMN stands out due to its advantageous combination of parameter efficiency and runtime efficiency. Although it may not be the fastest in terms of runtime, its performance remains highly competitive, highlighting the robustness of MPMixer.

[0089] Table 3. Comparison of different methods with different patch sizes on the IP, PU, ​​and HS datasets in terms of time and PARAMS(m).

[0090]

[0091] 3. Comparative Experimental Analysis

[0092] The comparison results of various networks are shown in Table 4. Taking the PU dataset as an example, DDMN achieves an 11.62% improvement in OA compared to ViT. The main reason is that DDMN contains more spatial information and channel details, as well as a full fusion of shallow and deep information. In addition, compared with the improved version of ViT (i.e., SpectralFormer), the OA of this embodiment is 2.95% higher. The reason is that although SpectralFormer injects more subtle spectral differences through overlapping grouping, this mechanical enhancement of spectral channels does not produce a significant improvement. In contrast, this embodiment expands the output channels of the depth convolution to amplify subtle spectral differences between channels and adaptively extract relevant features, thereby achieving better classification performance. This method also shows a significant improvement compared with 3D-CNN and 2D-CNN, as the latter two methods rely entirely on convolution-based feature extraction. Even compared with more advanced CNNs (i.e., RSSAN with attention mechanism), the model of this embodiment maintains an OA advantage of 2.79%. In addition, the OA of the first 50 rounds of training for each model was compared, such as... Figure 5 As shown in the diagram, DDMN (shown by the red line) clearly demonstrates fast convergence across all datasets, with stable parameter fitting throughout the training process. Furthermore, DDMN shows varying degrees of improvement on two additional datasets. Notably, DDMN exhibits significant stability across different datasets, indicating its ability to effectively capture richer spectral details and variations, thus enabling it to effectively address the challenges posed by the characteristics of diverse datasets.

[0093] Table 4. Classification accuracy (%) of different methods on the IP and PUandHS datasets

[0094]

[0095] In summary, this embodiment proposes a model DDMN for hyperspectral image classification. It uses DDSC to adaptively amplify and refine spectral channel details, then MPFormer for feature extraction and processing, with AWFM used for feature fusion, and finally softmax for classification. Experiments demonstrate that the proposed DDMN exhibits excellent performance in terms of both accuracy and parameter parameters.

Claims

1. A hyperspectral image classification method based on MetaFormer and deep extended feature fusion network, characterized in that, The method mainly includes the following steps: First, the spectral dimension is amplified to enhance subtle differences through deep dilatation and separation convolution (DDSC). Then, the multi-scale pooling feature fusion module (MPFormer) is used to aggregate and extract multi-scale spatial spectral information. During the extraction process, the adaptive weighted fusion module (AWFM) is used to adaptively weight and fuse the low-level features obtained from the shallow MPFormer and the high-level features obtained from the deep MPFormer. Finally, softmax is used for classification to obtain the results. The vector output by the depthwise dilated separable convolution (DDSC) The process using MPFormer is as follows: First, patch embedding is performed, followed by flattening and linear transformation. ,in , It is the dimension after linear transformation, which will be ClassToken. and By splicing The formula is as follows: （3） in This represents the concatenation operation of vectors or matrices. For vectors The amount, Then through and The module extracts features using the following formula: （4） （5） in This indicates an element-wise addition operation. This represents the output of the intermediate process. This represents the final output of the MPFormer module. This indicates that normalization is being performed. This represents a multilayer perceptron. This indicates multi-scale feature fusion using MPMixer. The MPMixer module adapts to dense predictions in hyperspectral images, utilizing multi-scale AvgPooling for feature fusion. It performs feature mixing without requiring any trainable parameters, and its formula is expressed as: （6） （7） in It is the multi-scale spatial spectral feature matrix obtained after the MPMixer module. The scale label indicates the multi-scale pooling. It is a mixed feature matrix obtained by pooling operations at different scales. It is the average pooling scale. and It is the step size calculation variable that needs to be manipulated in the pixel coordinates during the pooling operation; The feature map obtained after passing through the multi-scale pooling feature fusion module MPFormer The shallow and deep feature maps are spatially concatenated using the adaptive weighted fusion module AWFM, which integrates the features before and after the initial phase. Adaptive feature extraction is performed using the following formula: （8） in This represents the output after adaptive feature extraction. and This represents the feature matrix obtained after passing through different MPFormers. This represents the convolution operation. Represents the concatenation operation of vectors or matrices; After that and Add the positions, multiply by the weights, and then sum. The final output is obtained by fusion. The formula is as follows: （9） in, This represents the normalized weighting coefficient.

2. The hyperspectral image classification method based on MetaFormer and deep extended feature fusion network according to claim 1, characterized in that, For Depth Dilated Separate Convolution (DDSC), firstly, a channel-wise convolution with a size of 3x3 and an output channel that is k times the number of input channels is used to magnify the channel dimension details. Then, a pointwise convolution with a size of 1x1 is used to fuse the channels.

3. The hyperspectral image classification method based on MetaFormer and deep extended feature fusion network according to claim 2, characterized in that, For Deep Dilated Separate Convolutional (DDSC), the first step is to start from the input hyperspectral image... Selected from Patches Where C, H, and W refer to the number of spectral channels, height, and width of the hyperspectral image, respectively. To select the number of patches, The patches after depthwise dilation and separation convolution can be represented as: （1） in The trainable weight matrix of the convolution kernel. Is input HSI The patches in and express The coordinates of the pixel value, As the expansion factor, For channel dimension, and These are the two dimensions of the convolution kernel; Then, pointwise convolution is used to fuse the features from each layer. The formula for the output patches can be expressed as: （2） in The trainable weight matrix of the convolution kernel. For channel dimension, and express The coordinates of the pixel value.

Images