A visual recognition method based on MLP without block embedding mapping
By stacking fully connected layers to create a visual MLP model without the need for block embedding, the problems of large parameter count and altered feature distribution in existing visual MLP models are solved, achieving efficient feature extraction and improved recognition accuracy.
Patent Information
- Application Number
- CN202310401160.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-15
- Publication Date
- 2026-01-16
- Estimated Expiration
- 2043-04-15
AI Technical Summary
Existing visual MLP models perform block embedding mapping through convolution operations, resulting in a large number of parameters and high storage space requirements, and also altering the initial spatial feature distribution.
A visual MLP model consisting of stacked fully connected layers without the need for block embedding is adopted. Feature extraction is achieved by using batch normalization and multilayer perceptron modules to independently interact with features in the width, height, and channel dimensions.
The number of model parameters was reduced while maintaining the model's representational capabilities. Furthermore, the recognition accuracy was improved by visually observing the information interaction and long-term dependency phenomena on the feature maps.
Smart Images
Figure CN116563613B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of deep learning model computer vision recognition, and particularly relates to a visual recognition method based on MLP, which is a deep learning visual multi-layer perceptron model for extreme decoupling of features and can be used as a backbone model for computer vision tasks. BACKGROUND
[0002] As the first end-to-end visual model, convolutional neural networks (CNNs) have become the de facto standard tool in the field of computer vision for a long time and have been successfully applied to many practical scene tasks. Generally speaking, CNNs have a pyramid design, that is, the deeper the convolutional layer, the more feature channels it has but the lower the resolution, and typical representatives are CIFAR-QUICK and VGGNet. The research on CNNs mainly focuses on two aspects: first, the structural design of the model. A representative model is ResNet which introduces residual connections. Residual connections have become an essential design element of convolutional neural networks, and ResNet has derived many variants, such as ResNeSt, ResNeXt, etc. The other aspect is the form of the convolution kernel. Deep separable convolution decouples the standard convolution kernel into depth-wise convolution and point-wise convolution, which extract spatial and channel features respectively.
[0003] With the use of the self-attention mechanism originally proposed in the field of natural language processing, visual Transformers (ViTs) have also achieved excellent results in many visual tasks.
[0004] Recent research shows that when using modern learning strategies such as pre-training on large datasets and a large number of data augmentation methods, vision multi-layer perceptron (Vision MLP) is proven to be a visual model that can achieve comparable performance to CNNs and ViTs. As a highly competitive but conceptually and technically simpler alternative to convolution operations and self-attention mechanism operations, vision MLP inherits the trend of end-to-end learning mode to discard hand-designed visual features and inductive bias. Therefore, in the field of deep learning models, from the original MLP using fully connected layers to CNN using convolution layers, and then to ViTs using self-attention mechanisms, a closed loop of returning to MLP has been successfully established.
[0005] Visual MLPs are designed to rely entirely on fully connected layers (i.e., feedforward neural networks) and mimic the structure of ViTs, i.e., using patch-embedded images as input. However, for existing visual MLPs (Plain-MLP, MLP-Mixer, and ResMLP, etc.), the common practice is to use convolutional operations to implement the patch-embedding mapping operation on the input image, which is contrary to the purpose of establishing a visual MLP model purely stacked by fully connected layers. SUMMARY
[0006] For existing visual MLPs that mimic the structure of visual Transformers, using convolutional mapping to embed three-dimensional input tensors into two-dimensional matrices through patch-embedding results in technical defects such as the need for a large number of parameters and storage space and changes in the initial spatial feature distribution. The purpose of the present application is to use a visual MLP model that is completely free of convolutional operations and does not require patch-embedding, which is purely stacked by fully connected layers, to implement a visual recognition method. Here, patch-embedding refers to uniformly dividing an image into multiple blocks and mapping the image from three dimensions to two dimensions through convolution or other encoding methods.
[0007] The technical solutions adopted by the present application are as follows:
[0008] A visual recognition method based on MLP without patch-embedding mapping, comprising the following steps:
[0009] Step 1: stack N layers of MLP into a backbone model; directly input the image into the backbone model without convolutional mapping encoding or patch-embedding operation;
[0010] Step 2: for each MLP layer, given an input feature map X with a size of W*H*C, W, H and C represent the width, height and channel dimensions respectively, first perform batch normalization on the feature map BN1(X) through a batch normalization layer BN1;
[0011] Step 3: use a width multi-layer perceptron module composed of W different linear mappings to interactively extract the width dimension feature information of the feature map BN1(X) obtained in step 2 at each pixel position point in the H*C plane, to obtain a feature map U;
[0012] Step 4: use a height multi-layer perceptron module composed of H different linear mappings to interactively extract the height dimension feature information of the feature map U obtained in step 3 at each pixel position point in the C*W plane, to obtain a feature map V; perform batch normalization on the feature map V through a batch normalization layer BN2 to obtain a feature map BN2(V); add BN2(V) and the feature map BN1(X) obtained in step 2, and then perform batch normalization again through a batch normalization layer BN3 to obtain a feature map S;
[0013] Step 5, stack the first fully connected layer composed of E*C different linear mappings, the first activation layer, the batch normalization layer BN4, the second fully connected layer composed of C different linear mappings, the second activation layer and the batch normalization layer BN5 to obtain the channel multi-layer perception module, wherein E is an expansion coefficient for adjusting the output feature channel number of the first fully connected layer; interactively extract the channel dimension feature information of the feature map S obtained in step 4 at each pixel position point in the W*H plane to obtain a feature map O;
[0014] Step 6, add the feature map O in step 5 and the feature map S in step 4 to obtain an output feature map X' of the current MLP layer, which is input as an input feature map to the next MLP layer for continuing feature extraction; repeat steps 2-6 until the last MLP layer.
[0015] Step 7, input the output of the backbone model to a global average pooling layer and a linear fully connected layer for classification to perform image recognition classification and output a final prediction result.
[0016] Further, in step 3, the W different linear mappings are a fully connected layer with W as input and output, which has parameters PW with a size of W*W, and each pixel position point in the H*C plane shares the parameters.
[0017] Further, in step 4, the H different linear mappings are a fully connected layer with H as input and output, which has parameters PH with a size of H*H, and each pixel position point in the C*W plane shares the parameters.
[0018] Further, in step 5, in the first fully connected layer, the E*C different linear mappings are a fully connected layer with C as input and E*C as output, which has parameters PC1 with a size of C*E*C, and each pixel position point in the W*H plane shares the parameters; in the second fully connected layer, the C different linear mappings are a fully connected layer with E*C as input and C as output, which has parameters PC2 with a size of E*C*C, and each pixel position point in the W*H plane shares the parameters.
[0019] Further, in step 5, the first activation layer and the second activation layer are activation functions used in deep learning models.
[0020] The visual recognition method based on the MLP provided in the present application completely decouples the visual feature extraction, and performs feature extraction alternately and sequentially from three dimensions of width, height and channel number, thereby greatly reducing the parameter quantity of the model under the premise of ensuring the model representation capability. The present application is the first visual multi-layer perception model using a stack of fully connected layers, and uses the original image as the input without performing convolution mapping coding or block embedding on the input image. In addition, by reconstructing the global spatial weight by combining the width and height dimensions, the information interaction between any pixel pairs on the feature mapping can be observed visually, and the phenomenon of capturing long-term dependencies can be observed. BRIEF DESCRIPTION OF DRAWINGS
[0021] Figure 1 The flowchart of the method of the present application is shown in the figure.
[0022] Figure 2 The MLP layer schematic diagram of various specific embodiments of the present application is shown in the figure, (a) is a basic MLP layer, (b) is an expanded MLP layer, (c) is an alternating MLP layer, and (d) is a super MLP layer.
[0023] Figure 3 The performance comparison between the present application and the prior art is shown in the figure.
[0024] Figure 4 The visualization of the reconstructed spatial weight of the present application is shown in the figure. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
[0026] The existing visual MLP model is improved in the present application, which is referred to as an X-MLP model in the embodiments. Figure 1 The flowchart of the visual recognition method using the X-MLP model of the present application is shown in the figure.
[0027] Firstly, the original image is used as the input of the X-MLP model without dividing the image into uniform blocks and performing convolution mapping embedding. The X-MLP model is repeatedly stacked by multiple layers (12 layers in the embodiment). Each layer of the MLP layer completely decouples the features, and independently alternately performs information interaction and feature extraction from three dimensions of width, height and channel number. The feature mapping generated by each layer of the MLP layer is a three-dimensional real-valued tensor. For the image recognition and classification task, a global average pooling layer and a linear fully connected layer for classification are added after the X-MLP model.
[0028] For each MLP layer of the layers, assuming that a given input feature map is X, with size W*H*C, first batch normalization is performed on the feature map X by a batch normalization layer BN to obtain a feature map BN(X).
[0029] Then, the width dimension feature information of the feature map BN(X) is extracted interactively at each pixel position point in the H*C plane using a width-cross MLP block composed of W different linear mappings, to obtain a feature map U. The W different linear mappings are a fully connected layer with input and output both being W, which has parameters PW (i.e. weights) with size W*W (i.e. PW 11 to PW ww ), and each pixel position point in the H*C plane shares the parameters.
[0030] Then, the height dimension feature information of the feature map U is extracted interactively at each position point in the C*W plane using a height-cross MLP block composed of H different linear mappings, to obtain a feature map V. The H different linear mappings are a fully connected layer with input and output both being H, which has parameters PH (i.e. weights) with size H*H (i.e. PH 11 to PH HH ), and each pixel position point in the C*W plane shares the parameters.
[0031] Then, the feature map V is batch normalized by a batch normalization layer BN to obtain a feature map BN(V);
[0032] Then, the feature map BN(V) is added to the feature map BN(X), and then batch normalized by a batch normalization layer BN to obtain a feature map S;
[0033] Then stack E*C different linear mapping layers (first fully connected layer), PReLU activation layer, batch normalization layer, C different linear mapping layers (second fully connected layer), PReLU activation layer and batch normalization layer to obtain a channel-cross MLP block. E is an expansion coefficient, used to adjust the output feature channel number of the fully connected layer, and E is 4 in this embodiment. The first fully connected layer of the channel-cross MLP block has E*C different linear mappings, which is a fully connected layer with an input of C and an output of E*C, has parameters PC1 (i.e. weights) with a size of C*E*C, and each position point in the W*H plane shares the parameters. The second fully connected layer of the channel-cross MLP block has C different linear mappings, which is a fully connected layer with an input of E*C and an output of C, has parameters PC2 (i.e. weights) with a size of E*C*C, and each pixel position point in the W*H plane shares the parameters.
[0034] Then the channel-cross MLP block is used to interactively extract channel dimension feature information of the feature map S at each position point in the W*H plane to obtain a feature map O.
[0035] Then the feature map O is added to the feature map S, i.e. an identity mapping residual connection from the feature map S to the feature map O, to obtain an output feature map X' of the MLP layer.
[0036] Then the obtained output feature map X' is input as an input feature map into a next MLP layer to continue feature extraction. When the X-MLP model is used for other visual tasks, neural network layers required for the specific task can be added after the X-MLP model, and the output feature map of the X-MLP model is input into the neural network layers. For example, a classification module is connected in an image classification task.
[0037] The core innovation of the MLP-based visual recognition method is that the MLP layer completely decouples the features, independently and alternately interacts information and extracts features from the width, height and channel dimensions, and different combinations of components conforming to the core innovation are within the scope of the application, so that the application derives many variant embodiments, such as Figure 2 Fig. 1 shows MLP layer schematic diagrams of various embodiments of the application, including basic MLP layers and improved MLP layers.
[0038] Figure 2 (a) is a basic MLP layer, corresponding to "X-Basic" in Figure 3 , which is built according to the above steps.
[0039] Figure 2 (b) is an expanded MLP layer, corresponding to Figure 3 “X-Exp”. Based on the basic MLP layer, an additional fully connected layer is added in the width multi-layer perceptron module and the height multi-layer perceptron module, and a nonlinear activation function is added between the two fully connected layers. The model aims to capture nonlinear relationships and improve the extraction ability of spatial features.
[0040] Figure 2 (c) is an alternating MLP layer, corresponding to Figure 3 “X-Alt”. This model separates the two fully connected layers in the width multi-layer perceptron module and the height multi-layer perceptron module of the expanded MLP layer, and alternates the four layers in order. This method can more fully extract and merge the information of width and height.
[0041] Figure 2 (d) is a super MLP layer, corresponding to Figure 3 “X-Sup”. This model builds more skip connections and channel cross MLP blocks in the basic MLP layer to enhance feature extraction. In addition, we aggregate channel information after mixing three-dimensional features. This complex de-symbolization enhances the extraction of spatial and channel features and achieves superior performance on most benchmark datasets. The beneficial effects of the present embodiment can be verified by the following experiments:
[0042] I. Model accuracy
[0043] According to the four embodiments X-MLP models stacked by Figure 2 four kinds of MLP layers proposed in the present application, i.e. X-Basic, X-Exp, X-Alt, X-Sup, are tested on classification tasks, and the test includes ten challenging computer vision image classification datasets, respectively: Tiny-ImageNet-200, Caltech-256, Food-101, Flowers-102, CIFAR-100, CIFAR-10, SVHN, MNIST, KMNIST and FashionMNIST. As shown in Figure 4 , all of them achieve excellent test accuracy, proving that the X-MLP model proposed in the present application has excellent representation ability.
[0044] and compared with existing deep learning visual multi-layer perceptron models and convolutional neural network models, as shown in Figure 3 . In order to ensure the fairness of comparison, all visual models are trained to convergence on the ten datasets using the same optimization configuration.
[0045] When the method proposed in this invention is compared with other existing visual MLP methods, the method proposed in this invention is based on... Figure 2 All four implementations of the MLP layer achieve better results. Although MLP Mixer-12 is the best-performing MLP model among other existing visual MLPs, the super MLP layer proposed in this invention ( Figure 2 (d) The accuracy on the Tiny-ImageNet-200 and Food-101 datasets significantly outperforms MLMIXer-12, by 9.22% and 10.34%, respectively. Furthermore, the proposed embodiment X-Sup has 1.5 times fewer parameters than MLMIXer-12. The proposed embodiment X-Alt significantly outperforms the optimal MLMIXer-12 on the Flowers-102 and CIFAR-100 datasets, by 5.77% and 5.99%, respectively, but the proposed embodiment X-Alt has only 54.62% of the number of parameters of MLMIXer-24.
[0046] like Figure 3 The test results show the accuracy differences between the best implementation of the X-MLP model proposed in this invention and the best existing visual MLP: The accuracy difference between the proposed X-Alt implementation on the Caltech-256 dataset and the best existing implementation MLMPixer-24 is 3.25%; the accuracy difference between the proposed X-Alt implementation on the CIFAR-10 dataset and the best existing implementation MLMPixer-12 is 5.99%; the accuracy difference between the proposed X-Sup implementation on the MNIST dataset and the best existing implementation MLMPixer-12 is 0.40%; the accuracy difference between the proposed X-Sup implementation on the KMNIST dataset and the best existing implementation MLMPixer-24 is 2.33%; and the accuracy difference between the proposed X-Sup implementation on the Fashion-MNIST dataset and the best existing implementation MLMPixer-12 is 0.84%. The increase in accuracy is still very significant.
[0047] When compared with CNN models, the X-MLP embodiment proposed in this invention still achieves competitive or even better results on these challenging computer vision image classification datasets. For example, the proposed embodiment X-Sup outperforms VGGNet on the Tiny-ImageNet-200 and Food-101 datasets by 5.66% and 3.92%, respectively, while having only 28.52% of the parameters of VGGNet.
[0048] It can be seen that the application has superior comprehensive performance in model accuracy and model size (parameter amount).
[0049] II. Visualization results
[0050] By reconstructing the spatial weight, the spatial weight can be visualized, so that the information interaction between any pixel pairs on the feature map and the capture of long-term dependence phenomenon can be observed.
[0051] The steps of reconstructing the spatial weight are:
[0052] (1) Given the input X, the size is H*W*C, since only the spatial weight is considered, the channel dimension of X is ignored, and the size of X is H*W;
[0053] (2) The weight PW of the width-cross MLP block is used to extract the feature information of the width dimension, and the size is W*W;
[0054] (3) The weight PH of the height-cross MLP block is used to extract the feature information of the height dimension, and the size is H*H;
[0055] (4) The width-cross MLP block is applied to X to obtain the output feature map Y', and the expression is:
[0056]
[0057] Wherein, the subscript represents the coordinates of the point. (i, j) is the point in the i-th row and j-th column, the value range of i is 1~H, and the value range of j is 1~W.
[0058] (5) The height-cross MLP block is applied to the feature map Y' to obtain the output feature map Y, and the expression is:
[0059]
[0060] (6) The expression of the feature map Y is rewritten as:
[0061]
[0062] (7) The reconstructed spatial weight W' has a size of H*W*H*W, and the corresponding mapping can be expressed as:
[0063]
[0064]
[0065] Wherein, a and b are subscripts, and the meaning is the specific size (H and W).
[0066] As Figure 4 shows the visualization of the reconstructed spatial weights of the present application. It can be observed that the information interaction between any pair of pixels on the feature map and the capture of long-term dependence phenomenon.
[0067] The above description is merely that of the preferred embodiments of the present application and is not intended to limit the application and its applications. Any modification, equivalent replacements, improvements, and the like made within the spirit and principle of the present application shall fall within the protection scope of the present application.
Claims
1. A visual recognition method based on MLP without patching embedding mapping, characterized in that, The method comprises the following steps: Step 1, stack N MLP layers into a backbone model; directly input the image into the backbone model without convolutional mapping coding or block embedding operation; Step 2, for each MLP layer, given an input feature map X with a size of W*H*C, W, H and C represent width, height and channel dimension respectively, first batch normalization is performed on the feature map X through a batch normalization layer BN1 to obtain a feature map BN1(X); Step 3, width multi-layer perception module composed of W different linear mappings is used to interactively extract width dimension feature information of the feature map BN1(X) obtained in step 2 at each pixel position point in the H*C plane to obtain a feature map U; Step 4, height multi-layer perception module composed of H different linear mappings is used to interactively extract height dimension feature information of the feature map U obtained in step 3 at each pixel position point in the C*W plane to obtain a feature map V; The feature map V is batch normalized through a batch normalization layer BN2 to obtain a feature map BN2(V); the BN2(V) is added to the feature map BN1(X) obtained in step 2, and then batch normalized through a batch normalization layer BN3 to obtain a feature map S; Step 5, a first fully connected layer composed of E*C different linear mappings, a first activation layer, a batch normalization layer BN4, a second fully connected layer composed of C different linear mappings, a second activation layer and a batch normalization layer BN5 are stacked to obtain a channel multi-layer perception module, wherein E is a dilution coefficient used to adjust the output feature channel number of the first fully connected layer; the channel dimension feature information of the feature map S obtained in step 4 is interactively extracted at each pixel position point in the W*H plane to obtain a feature map O; Step 6, the feature map O in step 5 is added to the feature map S in step 4 to obtain an output feature map X' of the current MLP layer, which is input as an input feature map into the next MLP layer for continuous feature extraction; steps 2-6 are repeated until the last MLP layer; Step 7, the output of the backbone model is input into a global average pooling layer and a linear fully connected layer for classification to perform image recognition classification and output a final prediction result. 2.The visual recognition method based on MLP without block embedding mapping according to claim 1, wherein, In step 3, the W different linear mappings are a fully connected layer with an input and an output both being W, which has parameters PW with a size of W*W, and each pixel position point in the H*C plane shares the parameters. 3.The visual recognition method based on MLP without patch embedding mapping according to claim 1, wherein, In step 4, the H different linear mappings are a fully connected layer with an input and an output both being H, which has parameters PH with a size of H*H, and each pixel position point in the C*W plane shares the parameters.
4. The visual recognition method based on MLP without block embedding mapping according to claim 1, characterized in that, In step 5, in the first fully connected layer, the E*C different linear mappings are a fully connected layer with an input of C and an output of E*C, which has parameters PC1 with a size of C*E*C, and each pixel position point in the W*H plane shares the parameters; in the second fully connected layer, the C different linear mappings are a fully connected layer with an input of E*C and an output of C, which has parameters PC2 with a size of E*C*C, and each pixel position point in the W*H plane shares the parameters.
5. The visual recognition method based on MLP without block embedding mapping according to claim 1, characterized in that, In the step 5, the first activation layer and the second activation layer are activation functions used in a deep learning model.