Lightweight window pyramid network model and application thereof

Abstract

The application belongs to the field of image recognition, and particularly relates to a light-weight window pyramid network model and application thereof, which comprises a Patchembedding module, a Patchmerging module and a Transformerblock module. The Patchembedding module performs average division on an input picture, and takes each block obtained as a vector for subsequent attention calculation. The Patchmerging module performs down-sampling on an input feature map, so that the network can perform feature calculation of different scales, thereby obtaining multiple feature maps of different resolution sizes. The Transformerblock firstly performs window attention calculation of different sizes on an input feature map, so that the network can pay attention to features of different scales, then performs light-weight attention calculation on the features, so that information in different windows can be interacted, and finally uses a linear layer to fuse the features. The network architecture proposed in the application realizes the best balance between performance and efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image recognition, and in particular, it is a lightweight window pyramid network model and its application. The network model is based on Transformer and uses deep learning to achieve lightweight image recognition. BACKGROUND

[0002] Image classification is the most basic task in the field of image recognition. Computers classify pictures based on their semantic information to achieve the smallest classification error. Since the AlexNet network was proposed in 2012, deep learning methods based on convolutional neural networks have become the mainstream method for solving image classification problems. In the field of computer vision, image classification is not only the basis for research on dense detection tasks such as object detection, semantic segmentation, and instance segmentation, but also has broad application scenarios in the fields of autonomous driving, facial recognition, and industrial control.

[0003] Transformer was first applied in the field of natural language processing. Due to its excellent performance, more and more workers have begun to try to apply Transformer to the field of computer vision. The first to apply Transformer to the field of computer vision was Detection Transformer, which applied Transformer as a neck to the target detection network. Under the condition of realizing end-to-end detection, it achieved results comparable to Faster RCNN, demonstrating the powerful expression ability of Transformer. The first to apply Transformer as a backbone network to the field of computer vision was Vision Transformer, which divided the picture into several patches, each patch as a token (representing a feature) to perform attention calculation, and achieved good results. Swin Transformer (Transformer based on sliding window) divided the picture into multiple windows and limited attention calculation within the window, greatly reducing the computational complexity of Transformer and achieving good results. PyramidVision Transformer also reduced the number of tokens participating in attention calculation in a subsampling manner based on Vision Transformer, achieving good results.

[0004] Although the above-mentioned Transformer-based backbone network achieves good results, they are still limited by the following shortcomings: (1) In each attention calculation, only single-scale features are involved, but the size of the target is not uniform, so it is necessary to introduce multi-scale information into each attention calculation. (2) Although the above-mentioned method reduces the computational complexity of the Transformer, it also brings a performance decline, so it is necessary to explore an efficient attention calculation method. SUMMARY

[0005] In view of the defects and deficiencies of the prior art, the purpose of the present application is to propose a lightweight window pyramid network model and its application to achieve the best balance between model recognition accuracy and computational efficiency.

[0006] To achieve the above goal, the specific solution of the present application is:

[0007] A lightweight window pyramid network model, which consists of three basic modules: Patch embedding, Patch merging and Transformer block. The core of the Patch embedding module is a 7x7 convolution with a step size of 4, which maps every 4x4 pixels in the input image to a feature (token), and the 7x7 convolution kernel preserves the adjacent information between patches to a great extent; The core of the Patch merging module is a 3x3 convolution with a step size of 2, which performs 2 times down-sampling on the input feature map to obtain feature maps of different scales; The Transformer block mainly consists of an attention module, a feedforward network and a normalization layer, the attention module has local attention and global attention, the former is mainly realized by window attention, and the latter is realized by down-sampling attention (ADQK, Attention based on Down-sampled QK). In order to introduce multi-scale information in attention calculation, a feature map is divided into several parts in the channel dimension, and different size window attention and corresponding down-sampling attention calculation are performed on them. The feedforward network consists of two linear layers. The normalization layer consists of Layer normalization.

[0008] In the technical scheme, the Patch embedding module is composed of a convolution layer and a normalization layer, the convolution layer is composed of 64 convolution kernels with a size of 7*7 and a step of 4, can reduce the resolution of the input picture to 1 / 4 of the original, and the channel number to 64, the normalization layer adopts Layer normalization to perform normalization operation on the input features in the channel dimension; the Patch merging module is composed of a convolution layer and a normalization layer, the convolution layer is composed of a plurality of convolution kernels with a size of 3*3 and a step of 2, can reduce the resolution of the input picture to 1 / 2 of the original, and the channel number to twice of the original, the normalization layer adopts Layer normalization to perform normalization operation on the input features in the channel dimension; the Transformer block is composed of an attention module, a feedforward network and a normalization layer; in the attention module, the input feature x is firstly subjected to three different linear mappings to generate q1, k1 and v1 matrices, then they are averagely divided into two groups in the channel dimension to perform attention calculation of two different scales; the attention calculation is composed of a window attention module and an ADQK module: the window attention module limits the attention calculation in a w*w window and outputs x1, at this time, x1 has not realized the fusion of global information; in the ADQK module, the input x1 is firstly subjected to a linear mapping to generate a v2 matrix, then x1 is subjected to w times downsampling, normalization and two linear mappings to generate q2 and k2 matrices to obtain the output x2; the outputs of the two branches are spliced in the channel dimension, and the multi-scale information fusion is performed through a linear layer to obtain the output of the attention module; then, the normalization layer, the feedforward network and the normalization layer are sequentially passed to obtain the final output.

[0009] The application further discloses an application of the lightweight window pyramid network model, which is used for realizing lightweight image recognition and specifically comprises the following steps.

[0010] S1, inputting an image into a Patch embedding module to obtain a first feature map with a resolution of 1 / 4 of an original image and a channel number of 64;

[0011] S2, extracting features of the first feature map by using a Transformer block to obtain a second feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0012] S3, inputting the second feature map into a Patch merging module to obtain a third feature map with a resolution of 1 / 8 of the original image and a channel number of 128;

[0013] S4, extracting features of the third feature map by using the Transformer block to obtain a fourth feature map with a resolution of 1 / 8 of the original image and a channel number of 64;

[0014] S5, input the fourth feature map into the Patch merging module, obtain the fifth feature map with a resolution of 1 / 16 of the original image and a channel number of 256;

[0015] S6, continuously use four Transformer blocks to extract features of the fifth feature map, obtain the sixth feature map with a resolution of 1 / 16 of the original image and a channel number of 256;

[0016] S7, input the sixth feature map into the Patch merging module, obtain the seventh feature map with a resolution of 1 / 32 of the original image and a channel number of 512;

[0017] S8, continuously use two Transformer blocks to extract features of the seventh feature map, obtain the eighth feature map with a resolution of 1 / 32 of the original image and a channel number of 512.

[0018] The beneficial effects of the present application are:

[0019] Compared with the existing model, the present application has significant progress: the present application designs a backbone network based on Transformer for lightweight image recognition, which provides an efficient method to solve the problem of high computational complexity of Transformer. The experimental results show that the network architecture designed achieves the best balance between detection performance and efficiency. The lightweight window pyramid backbone network based on Transformer designed by the present application is composed of three basic components, including Patch embedding, Patch merging and Transformer block. The Patch embedding module divides the input picture by averaging, and each block obtained is used as a vector (token) for subsequent attention calculation. The Patch merging module down-samples the input feature map, so that the network can calculate features of different scales, thereby obtaining multiple feature maps of different resolutions. The Transformer block first performs window attention calculation of different sizes on the input feature map, so that the network can pay attention to features of different scales, then performs lightweight attention calculation (P-ADQK, Pyramid Attention Based on Down-sampled QK) on these features, so that the information inside different windows can be interacted, and finally uses a linear layer to fuse these features. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 It is a lightweight window pyramid network model diagram of the present application.

[0021] Figure 2 Figure for pyramid attention module based on QK down-sampling in the present application.

[0022] Figure 3 Figure for Patch embedding in the present application.

[0023] Figure 4 Figure for Patch merging in the present application. DETAILED DESCRIPTION

[0024] In order to deepen the understanding of the present application, the present application will be further described in detail below in combination with the accompanying drawings and examples, which are only used to explain the present application and do not constitute a limitation on the protection scope of the present application.

[0025] As shown in Figure 1 A lightweight window pyramid network model, which is composed of three basic modules: Patch embedding, Patch merging and Transformer block. The core of Patch embedding module is 7x7 convolution with a stride of 4, which maps every 4x4 pixels in the input picture into a token, and the 7x7 convolution kernel greatly preserves the information between adjacent patches; The core of Patch merging module is 3x3 convolution with a stride of 2, which performs 2 times down-sampling on the input feature map to obtain feature maps of different scales; Transformer block is mainly composed of attention module, feedforward network and normalization layer, the attention module has local attention and global attention, the former is mainly realized by window attention, and the latter is realized by down-sampling attention (ADQK, Attention based on Down-sampled QK). In order to introduce multi-scale information in attention calculation, a feature map is divided into several parts in the channel dimension, and different size window attention and corresponding ratio down-sampling attention calculation are performed. The feedforward network is composed of two linear layers. The normalization layer is composed of Layer normalization.

[0026] As shown in Figure 2As shown, the down-sampling window pyramid attention is based on. The attention is composed of multiple branches, the number of branches depends on how many scales are calculated. Each branch is composed of two modules, which are window attention module and ADQK module respectively. Window attention can extract local information, but it cannot realize global information interaction, while the latter exactly solves the defects of window attention. The ADQK module takes into account that the window attention has realized the information interaction between local tokens, so it down-samples the input feature map to represent all local tokens with a token to calculate the attention map, and each value in the attention map represents the meaning: when predicting a certain local feature, the influence degree of a certain local feature on the local feature. Then use the local attention map to weight the input global feature map as shown in the figure, which can realize the interaction of global information and keep low computational complexity.

[0027] As shown in Figure 3 , the Patch embedding module is composed of a convolution layer and a normalization layer, the convolution layer is composed of 64 convolution kernels with a size of 7*7 and a step of 4, which can reduce the resolution of the input picture to 1 / 4 of the original, and the channel number to 64, the normalization layer adopts Layer normalization, which normalizes the input features in the channel dimension.

[0028] As shown in Figure 4 , the Patch merging module is composed of a convolution layer and a normalization layer, the convolution layer is composed of several convolution kernels with a size of 3*3 and a step of 2, which can reduce the resolution of the input picture to 1 / 2 of the original, and the channel number to twice the original, the normalization layer adopts Layer normalization, which normalizes the input features in the channel dimension.

[0029] In combination Figure 1 , the application also discloses an application of a lightweight window pyramid network model, which is used to realize lightweight image recognition, specifically including the following steps, step (Step, hereinafter abbreviated as S) as follows:

[0030] S1, input the image into the Patch embedding module to obtain a first feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0031] S2, using the Transformer block to extract features from the first feature map, obtaining a second feature map with a resolution of 1 / 4 of the original image and a channel number of 64;

[0032] S3, input the second feature map into the Patch merging module, obtain a third feature map with a resolution of 1 / 8 of the original image and a channel number of 128;

[0033] S4, use the Transformer block to extract features of the third feature map, obtain a fourth feature map with a resolution of 1 / 8 of the original image and a channel number of 64;

[0034] S5, input the fourth feature map into the Patch merging module, obtain a fifth feature map with a resolution of 1 / 16 of the original image and a channel number of 256;

[0035] S6, use four Transformer blocks in succession to extract features of the fifth feature map, obtain a sixth feature map with a resolution of 1 / 16 of the original image and a channel number of 256;

[0036] S7, input the sixth feature map into the Patch merging module, obtain a seventh feature map with a resolution of 1 / 32 of the original image and a channel number of 512;

[0037] S8, use two Transformer blocks in succession to extract features of the seventh feature map, obtain an eighth feature map with a resolution of 1 / 32 of the original image and a channel number of 512;

[0038] Special note: the above steps S1-S8 are combined with the attached Figure 1 The overall network structure designed by the application is described, and the steps and the attached Figure 1 may be mutually confirmed. It should be pointed out that the overall network designed by the application is tested on the image classification public benchmark imagenet1k.

[0039] The above is an exemplary embodiment of the application, and does not limit the patent protection scope of the application, any equivalent structure or equivalent process transformation using the content of the application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the application.

Claims

1. A method for constructing a lightweight window pyramid network model, characterized in that, It consists of three basic modules: Patchembedding, Patchmerging and Transformerblock; the Patchembedding module maps every 4x4 pixels in the input picture into a feature, and the 7x7 convolution kernel greatly preserves the information between adjacent patches; the Patchmerging module performs 2 times down-sampling on the input feature map to obtain feature maps of different scales; the Transformerblock first performs window attention calculation of different sizes on the input feature map to construct local features of different scales, and then uses the proposed down-sampling attention module to make information interaction between these local features, and finally obtains a feature map with global information; the Patchembedding module consists of a convolution layer and a normalization layer; the convolution layer consists of 64 convolution kernels with a size of 7x7 and a step of 4, which can reduce the resolution of the input picture to 1 / 4 of the original, and the channel number to 64; the normalization layer adopts Layernormalization to normalize the input features in the channel dimension.

2. The method of claim 1, wherein, The Patchmerging module consists of a convolution layer and a normalization layer; the convolution layer consists of several convolution kernels with a size of 3x3 and a step of 2, which can reduce the resolution of the input picture to 1 / 2 of the original, and the channel number to twice the original; the normalization layer adopts Layernormalization to normalize the input features in the channel dimension.

3. The method of claim 2, wherein, The Transformerblock consists of an attention module, a feedforward network and a normalization layer; in the attention module, the input feature x first undergoes three different linear mappings to generate q1, k1 and v1 matrices, which are then divided into two groups in the channel dimension and subjected to two different scale attention calculations; The attention calculation consists of a window attention module and an ADQK module: the window attention module limits the attention calculation within a w x w window and outputs x1, which does not realize global information fusion at this time; in the ADQK module, the input x1 first undergoes a linear mapping to generate a v2 matrix, then x1 undergoes w times down-sampling, normalization and two linear mappings to generate q2 and k2 matrices, and finally obtains the output x2; the outputs of the two branches are spliced in the channel dimension, and the multi-scale information is fused through a linear layer to obtain the output of the attention module; then it sequentially passes through a normalization layer, a feedforward network and a normalization layer to obtain the final output.

4. The method of claim 3, wherein the method further comprises: Specifically, the following steps are included: S1, inputting an image into the Patchembedding module to obtain a first feature map with a resolution of 1 / 4 of the original image and a channel number of 64; S2, using the Transformerblock to extract features from the first feature map to obtain a second feature map with a resolution of 1 / 4 of the original image and a channel number of 64; S3, input the second feature map into the Patchmerging module, obtain a third feature map with a resolution of 1 / 8 of the original image and a channel number of 128; S4, use the Transformerblock to extract features from the third feature map, obtain a fourth feature map with a resolution of 1 / 8 of the original image and a channel number of 64; S5, input the fourth feature map into the Patchmerging module, obtain a fifth feature map with a resolution of 1 / 16 of the original image and a channel number of 256; S6, use four Transformerblocks in succession to extract features from the fifth feature map, obtain a sixth feature map with a resolution of 1 / 16 of the original image and a channel number of 256; S7, input the sixth feature map into the Patchmerging module, obtain a seventh feature map with a resolution of 1 / 32 of the original image and a channel number of 512; S8, use two Transformerblocks in succession to extract features from the seventh feature map, obtain an eighth feature map with a resolution of 1 / 32 of the original image and a channel number of 512.

Images