Lightweight image super-resolution method and device based on hybrid pooling transformer

Abstract

The application discloses a kind of lightweight image super-resolution method and device based on mixed pooling Transform, it is related to image processing field, comprising: constructing lightweight image super-resolution model based on mixed pooling Transform and training, obtain the trained lightweight image super-resolution model;Acquire the low-resolution image to be reconstructed and scale factor and input to the trained lightweight image super-resolution model, low-resolution image input first convolutional layer, obtain the output feature of first convolutional layer, after the output feature of first convolutional layer passes through several Transform modules based on mixed pooling, obtain the output feature of last Transform module based on mixed pooling, the output feature of last Transform module based on mixed pooling is added with the output feature of first convolutional layer and successively passes through subpixel convolutional layer and second convolutional layer, obtain high-resolution reconstruction image.The application solves the problem that the existing Transform method is too high in calculation complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and particularly relates to a lightweight image super-resolution method and device based on mixed pooling Transformer. BACKGROUND

[0002] Single Image Super-Resolution (SISR) is a key task in the field of computer vision, which aims to restore high-resolution images from degraded low-resolution images. Since image SR can bring more abundant and valuable details to images, it is widely used in various computer vision applications, including remote sensing images, medical imaging, face recognition and video surveillance.

[0003] In recent years, Transformer, as a popular deep learning backbone network, has been widely used in the field of image SR. As an alternative to convolutional neural networks, Transformer captures long-range dependencies by computing the similarity between all query features and key features. Related research has shown that this global context modeling method has good potential in image SR. However, directly applying dense global self-attention is computationally infeasible because its computational complexity is quadratic in the length of the token sequence. For this reason, window-based self-attention is proposed, which limits self-attention computation within non-overlapping windows to reduce computational burden. Unfortunately, this window-based self-attention paradigm, which needs to be repeatedly stacked, is proven to be inefficient for methods that model context interactions due to the abandonment of global reasoning. This limitation is exacerbated in lightweight SR models, as these models do not have enough budget to build deep enough networks. Therefore, further reducing the computational complexity of self-attention is an urgent need for current image super-resolution methods. SUMMARY

[0004] The present application aims to propose a lightweight image super-resolution method and device based on mixed pooling Transformer to solve the above technical problems.

[0005] In a first aspect, the present application provides a lightweight image super-resolution method based on mixed pooling Transformer, comprising the following steps:

[0006] The trained lightweight image super-resolution model is obtained by constructing and training a lightweight image super-resolution model based on a hybrid pooling Transformer, the lightweight image super-resolution model comprising a first convolutional layer, a plurality of hybrid pooling-based Transformer modules, a sub-pixel convolutional layer and a second convolutional layer connected in sequence; the hybrid pooling-based Transformer module comprises a first normalization layer, a hybrid pooling-based self-attention module, a second normalization layer and a frequency domain channel hybrid feedforward network connected in sequence, the input feature of the hybrid pooling-based Transformer module and the output feature of the hybrid pooling-based self-attention module are added to obtain the input feature of the second normalization layer, the input feature of the second normalization layer is input into the second normalization layer and the frequency domain channel hybrid feedforward network to obtain the output feature of the frequency domain channel hybrid feedforward network, and the output feature of the frequency domain channel hybrid feedforward network and the input feature of the second normalization layer are added to obtain the output feature of the hybrid pooling-based Transformer module;

[0007] The low-resolution image to be reconstructed and the scale factor are input into the trained lightweight image super-resolution model, the low-resolution image is input into the first convolutional layer to obtain the output feature of the first convolutional layer, the output feature of the first convolutional layer is input into the plurality of hybrid pooling-based Transformer modules to obtain the output feature of the last hybrid pooling-based Transformer module, and the output feature of the last hybrid pooling-based Transformer module and the output feature of the first convolutional layer are added and then input into the sub-pixel convolutional layer and the second convolutional layer in sequence to obtain the high-resolution reconstructed image.

[0008] Preferably, the hybrid pooling-based self-attention module comprises a first linear layer, a second linear layer, a third linear layer, a hybrid pooling feature embedding module and a self-attention layer.

[0009] The input feature of the hybrid pooling-based self-attention module is divided into N first non-overlapping windows with a size of SxS The input feature of the hybrid pooling-based self-attention module is divided into N first non-overlapping windows with a size of SxS wherein N=HxW / S 2 wherein C, H and W represent the number of channels, length and width of the input feature of the hybrid pooling-based self-attention module, respectively, denotes a set of real numbers, and the first linear layer L Q The first non-overlapping window X win is converted into a query matrix

[0010] Q=L Q (X win );

[0011] The input features X of the self-attention module based on hybrid pooling are processed by the hybrid pooling feature embedding module to generate hybrid feature embeddings. Its expression is as follows:

[0012] X PM =D PM (X);

[0013] Among them, D PM (·) represents the function corresponding to the hybrid pooling feature embedding module;

[0014] Embedding hybrid features into X PM Divide into N windows with a size of Second non-overlapping window Where N = H × W / S 2 Then, using the second linear layer L K and the third linear layer L V The second non-overlapping window Convert them into bond matrices respectively Sum matrix Its expression is as follows:

[0015]

[0016] The self-attention layer performs self-attention computation based on the query matrix, key matrix, and value matrix as inputs to obtain enhanced features. The expression is as follows:

[0017]

[0018] Where T represents the transpose operation, and Softmax(·) represents the Softmax normalization function;

[0019] Finally, the enhanced feature Y will be... EN The shape is N×S 2 ×C is converted to C×H×W and used as the output feature of the self-attention module based on hybrid pooling.

[0020] As a preferred embodiment, the hybrid pooling feature embedding module includes a channel downsampling convolutional layer, a global max pooling layer, a global average pooling layer, a channel splicing layer, and a third convolutional layer;

[0021] Input features of the self-attention module based on hybrid pooling The input is fed into the hybrid pooling feature embedding module, and then passes through a channel downsampling convolutional layer with a scaling factor of 2 to reduce the number of channels of X by a factor of 2, thus obtaining the first intermediate feature map. Its expression is as follows:

[0022]

[0023] wherein, denotes a function corresponding to a channel down-sampling convolutional layer with a scaling factor of 2, whose kernel size is 1x1;

[0024] The first intermediate feature map X' is down-sampled by a global max pooling layer and a global average pooling layer, respectively, to obtain a first feature and a second feature The expressions are as follows:

[0025] X' GMP = GMP(X');

[0026] X' GAP = GAP(X');

[0027] wherein, GMP(·) denotes a function corresponding to a global max pooling layer, and GAP(·) denotes a function corresponding to a global average pooling layer;

[0028] The first feature X' GMP and the second feature X' GAP are input into a channel concatenation layer to be concatenated along the channel dimension, and then a third convolutional layer is used to perform a convolution operation with a kernel size of 1x1 to obtain a hybrid feature embedding The expression is as follows:

[0029] X PM = Conv 1×1 (Concat(X' GMP , X' GAP ))

[0030] wherein, Conv 1×1 (·) denotes a convolution operation with a kernel size of 1x1, and Concat(·) denotes a concatenation operation along the channel dimension.

[0031] As a preferred, the frequency domain channel hybrid feedforward network comprises a fourth convolutional layer and a channel attention layer.

[0032] The input feature of the frequency domain channel hybrid feedforward network is first subjected to a convolution operation with a kernel size of 1x1 by the fourth convolutional layer to obtain a second intermediate feature map The expression is as follows:

[0033] Z' = Conv 1×1 (Z);

[0034] wherein, Conv 1×1(·) represents a convolution operation with a kernel size of 1x1, C, H and W represent the number of channels, length and width of the input feature of the frequency domain channel mixed feedforward network, respectively, which are the same as the number of channels, length and width of the input feature of the mixed pooling based self-attention module, denotes a set of real numbers;

[0035] The second intermediate feature map Z' is converted from the spatial domain to the frequency domain by using the discrete Fourier transform to obtain a frequency domain feature map denotes a set of real numbers, and its expression is as follows:

[0036]

[0037] wherein x and y are the indices of the frequency domain feature map in the vertical direction and the horizontal direction, respectively, and the value ranges are 0, 1, 2,..., H-1 and 0, 1, 2,..., W-1, respectively, and u and v are the indices of the second intermediate feature map in the vertical direction and the horizontal direction, respectively, and the value ranges are 0, 1, 2,..., H-1 and 0, 1, 2,..., W-1, respectively;

[0038] The frequency domain feature map is input into the channel attention layer to obtain the frequency domain feature map H enhanced by attention, and its expression is as follows:

[0039]

[0040] wherein CAB(·) represents a function corresponding to the channel attention layer;

[0041] The frequency domain feature map H enhanced by attention is converted from the frequency domain to the spatial domain by using the inverse discrete Fourier transform to obtain the frequency enhanced feature its expression is as follows:

[0042]

[0043] The frequency enhanced feature Z FE is added to the second intermediate feature map Z' to obtain the output feature of the frequency domain channel mixed feedforward network its expression is as follows:

[0044] Z EN = Z FE + Z'.

[0045] As preferred, the input feature of the mixed pooling based Transformer module is given as The calculation process of the mixed pooling based Transformer module is as follows:

[0046] W' = f HWTSA (LN(W))+W;

[0047] W" = f GNFN (LN(W')) + W';

[0048] where LN(·) denotes a layer normalization operation, f HWTSA (·) denotes a function corresponding to the hybrid-pooling-based self-attention module, f GNFN (·) denotes a function corresponding to the frequency domain channel hybrid feedforward network, W' denotes an intermediate feature map, and W" denotes an output feature of the hybrid-pooling-based Transformer module.

[0049] As a preference, the calculation process of the lightweight image super-resolution model is as follows:

[0050] The low-resolution image is input into a first convolutional layer with a convolution kernel size of 1x1 to extract shallow features The expression is as follows:

[0051] F0 = Conv 1×1 (I LR );

[0052] where Conv 1×1 (·) denotes a convolution operation with a convolution kernel size of 1x1;

[0053] A number of A hybrid-pooling-based Transformer modules and a residual connection are used to extract deep features The expression is as follows:

[0054]

[0055] F D = F A +F0;

[0056] where f i denotes a function corresponding to the i-th hybrid-pooling-based Transformer module, i ∈ [1, A], F i-1 i-1 denotes an output feature of the i-1-th hybrid-pooling-based Transformer module, F i i denotes an output feature of the i-th hybrid-pooling-based Transformer module, A denotes a function corresponding to the A-th hybrid-pooling-based Transformer module, F A-1 A-1 denotes an output feature of the A-1-th hybrid-pooling-based Transformer module, F A A denotes an output feature of the A-th hybrid-pooling-based Transformer module;

[0057] The depth feature F is processed by a sub-pixel convolution layer with a scale factor of scale and a second convolution layer with a kernel size of 1*1 D Upsampling and reconstruction are performed to generate a high-resolution reconstructed image The expression is as follows:

[0058] I SR =Conv 1×1 (f up (F D ))

[0059] Wherein, f up (·) represents the function corresponding to the sub-pixel convolution layer.

[0060] In a second aspect, the present application provides a lightweight image super-resolution device based on hybrid pooling Transformer, comprising:

[0061] A model construction module is configured to construct and train a lightweight image super-resolution model based on hybrid pooling Transformer to obtain a trained lightweight image super-resolution model, wherein the lightweight image super-resolution model comprises a first convolution layer, a plurality of hybrid pooling based Transformer modules, a sub-pixel convolution layer and a second convolution layer connected in sequence; the hybrid pooling based Transformer module comprises a first normalization layer, a hybrid pooling based self-attention module, a second normalization layer and a frequency domain channel hybrid feedforward network connected in sequence, the input feature of the hybrid pooling based Transformer module is added to the output feature of the hybrid pooling based self-attention module to obtain the input feature of the second normalization layer, the input feature of the second normalization layer is processed by the second normalization layer and the frequency domain channel hybrid feedforward network to obtain the output feature of the frequency domain channel hybrid feedforward network, and the output feature of the frequency domain channel hybrid feedforward network is added to the input feature of the second normalization layer to obtain the output feature of the hybrid pooling based Transformer module;

[0062] A reconstruction module is configured to obtain a low-resolution image to be reconstructed and a scale factor and input them to the trained lightweight image super-resolution model, the low-resolution image is input to the first convolution layer to obtain the output feature of the first convolution layer, the output feature of the first convolution layer is processed by the plurality of hybrid pooling based Transformer modules to obtain the output feature of the last hybrid pooling based Transformer module, and the output feature of the last hybrid pooling based Transformer module is added to the output feature of the first convolution layer and then sequentially processed by the sub-pixel convolution layer and the second convolution layer to obtain a high-resolution reconstructed image.

[0063] In a third aspect, the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementation manners of the first aspect.

[0064] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, when the computer program is executed by a processor, the method described in any of the implementation manners of the first aspect is implemented.

[0065] In a fifth aspect, the present application provides a computer program product, comprising a computer program, when the computer program is executed by a processor, the method described in any of the implementation manners of the first aspect is implemented.

[0066] Compared with the prior art, the present application has the following beneficial effects:

[0067] (1) The lightweight image super-resolution method based on mixed-pooling Transformer proposed in the present application combines the global max-pooling layer and the average pooling layer to compress the original image label sequence into a compressed label sequence containing beneficial features, thereby effectively avoiding dense self-attention calculation, so that the image super-resolution model can achieve better performance with less calculation overhead.

[0068] (2) The lightweight image super-resolution method based on mixed-pooling Transformer proposed in the present application realizes efficient feature aggregation by cross-attention to migrate the compressed label to the original image label, overcoming the problem of high calculation complexity of existing Transformer methods.

[0069] (3) The lightweight image super-resolution method based on mixed-pooling Transformer proposed in the present application adopts a frequency domain channel mixed feedforward network to enhance beneficial high-frequency information, thereby enhancing the feature representation capability of the image super-resolution model. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0071] Figure 1 Flowchart of the lightweight image super-resolution method based on mixed-pooling Transformer of the embodiments of the present application;

[0072] Figure 2A schematic diagram of a mixed-pooling feature embedding module of a mixed-pooling based lightweight image super-resolution method according to an embodiment of the present application;

[0073] Figure 3 A schematic diagram of a mixed-pooling based self-attention module of a mixed-pooling based lightweight image super-resolution method according to an embodiment of the present application;

[0074] Figure 4 A schematic diagram of a frequency domain channel mixed feed-forward network of a mixed-pooling based lightweight image super-resolution method according to an embodiment of the present application;

[0075] Figure 5 A schematic diagram of a mixed-pooling based Transformer module of a mixed-pooling based lightweight image super-resolution method according to an embodiment of the present application;

[0076] Figure 6 A schematic diagram of a lightweight image super-resolution model of a mixed-pooling based lightweight image super-resolution method according to an embodiment of the present application;

[0077] Figure 7 A schematic diagram of a mixed-pooling based lightweight image super-resolution device according to an embodiment of the present application;

[0078] Figure 8 A hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0080] Figure 1 A mixed-pooling based lightweight image super-resolution method according to an embodiment of the present application is shown, which includes the following steps:

[0081] S1. Construct and train a lightweight image super-resolution model based on hybrid pooling Transformer to obtain the trained lightweight image super-resolution model. The lightweight image super-resolution model includes a first convolutional layer, several hybrid pooling-based Transformer modules, a sub-pixel convolutional layer, and a second convolutional layer connected in sequence. The hybrid pooling-based Transformer module includes a first normalization layer, a hybrid pooling-based self-attention module, a second normalization layer, and a frequency domain channel hybrid feedforward network connected in sequence. The input features of the hybrid pooling-based Transformer module and the output features of the hybrid pooling-based self-attention module are added together to obtain the input features of the second normalization layer. After passing through the second normalization layer and the frequency domain channel hybrid feedforward network, the input features of the second normalization layer are used to obtain the output features of the frequency domain channel hybrid feedforward network. The output features of the frequency domain channel hybrid feedforward network and the input features of the second normalization layer are added together to obtain the output features of the hybrid pooling-based Transformer module.

[0082] In a specific embodiment, the hybrid pooling feature embedding module includes a channel downsampling convolutional layer, a global max pooling layer, a global average pooling layer, a channel splicing layer, and a third convolutional layer;

[0083] Input features of the self-attention module based on hybrid pooling The input is fed into the hybrid pooling feature embedding module, and then passes through a channel downsampling convolutional layer with a scaling factor of 2 to reduce the number of channels of X by a factor of 2, thus obtaining the first intermediate feature map. Its expression is as follows:

[0084]

[0085] in, This represents the function corresponding to the channel downsampling convolutional layer with a scaling factor of 2 and a kernel size of 1×1;

[0086] The first intermediate feature map X′ is downsampled using both global max pooling and global average pooling layers to obtain the first feature map. Second feature Its expression is as follows:

[0087] X′ GMP =GMP(X′);

[0088] X′ GAP =GAP(X′);

[0089] Where GMP(·) represents the function corresponding to the global max pooling layer, and GAP(·) represents the function corresponding to the global average pooling layer;

[0090] The first feature X′ GMP Second feature X′ GAP The input channel concatenation layer concatenates along the channel dimension, and then performs a 1×1 convolution operation on the third convolutional layer to obtain the hybrid feature embedding. Its expression is as follows:

[0091] X PM =Conv 1×1 (Concat(X′ GMP ,X′ GAP ))

[0092] Among them, Conv 1×1 (·) represents a convolution operation with a kernel size of 1×1, and Concat(·) represents a concatenation operation along the channel dimension.

[0093] For details, please refer to Figure 2 First, a hybrid pooling feature embedding module is constructed. In the embodiments of this application, the global max pooling layer and the global average pooling layer are combined in the hybrid pooling feature embedding module to compress the original image label sequence into a compressed label sequence containing beneficial features, thereby effectively avoiding dense self-attention calculations, enabling the lightweight image super-resolution model to achieve better performance with less computational overhead.

[0094] In a specific embodiment, the self-attention module based on hybrid pooling includes a first linear layer, a second linear layer, a third linear layer, a hybrid pooling feature embedding module, and a self-attention layer;

[0095] Input features of the self-attention module based on hybrid pooling Divide the window into N non-overlapping windows of size S×S. Where N = H × W / S 2 Where C, H, and W represent the number of channels, length, and width of the input features of the self-attention module based on hybrid pooling, respectively. Representing the set of real numbers, using the first linear layer L Q X win Transform into a query matrix

[0096] Q = L Q (X win );

[0097] The input features X of the self-attention module based on hybrid pooling are processed by the hybrid pooling feature embedding module to generate hybrid feature embeddings. Its expression is as follows:

[0098] X PM =DPM (X);

[0099] wherein, D PM denotes the function corresponding to the mixed-pooling feature embedding module;

[0100] The mixed feature embedding X PM is divided into N second non-overlapping windows of size wherein N = H x W / S 2 , and then the second non-overlapping windows K are respectively converted into key matrixes V and value matrixes by using a second linear layer L and a third linear layer L The expressions are as follows:

[0101]

[0102] According to the query matrix, the key matrix and the value matrix, self-attention calculation is performed in the self-attention layer to obtain enhanced features The expression is as follows:

[0103]

[0104] wherein, T denotes the transpose operation, and Softmax(·) denotes the function of Softmax normalization;

[0105] Finally, the shape of the enhanced features Y EN is converted from N x S 2 x C to C x H x W, and is taken as the output feature of the mixed-pooling based self-attention module.

[0106] Specifically, referring to Figure 3 , the mixed-pooling based self-attention module is constructed on the basis of the mixed-pooling feature embedding module, and efficient feature aggregation is realized by cross-attention to transfer the compressed label to the original image label.

[0107] In specific embodiments, the frequency domain channel mixed feedforward network comprises a fourth convolutional layer and a channel attention layer.

[0108] The input feature of the frequency domain channel mixed feedforward network is first subjected to a convolution operation with a convolution kernel size of 1 x 1 by the fourth convolutional layer to obtain a second intermediate feature map as shown in the following formula:

[0109] Z' = Conv 1×1 (Z);

[0110] wherein, Conv 1×1 ​(·) represents a convolution operation with a kernel size of 1x1, C, H and W represent the number of channels, length and width of the input feature of the frequency domain channel mixed feedforward network, respectively, which are the same as the number of channels, length and width of the input feature of the mixed pooling based self-attention module, denotes a set of real numbers;

[0111] The second intermediate feature map Z' is converted from the spatial domain to the frequency domain by using a discrete Fourier transform to obtain a frequency domain feature map denotes a set of real numbers, and its expression is as follows:

[0112]

[0113] wherein x and y are indexes of the frequency domain feature map in the vertical direction and the horizontal direction, respectively, and the value ranges are 0, 1, 2,..., H-1 and 0, 1, 2,..., W-1, respectively, u and v are indexes of the second intermediate feature map in the vertical direction and the horizontal direction, respectively, and the value ranges are 0, 1, 2,..., H-1 and 0, 1, 2,..., W-1, respectively;

[0114] The frequency domain feature map is input into a channel attention layer to obtain an attention enhanced frequency domain feature map H, and its expression is as follows:

[0115]

[0116] wherein CAB(·) represents a function corresponding to the channel attention layer;

[0117] The attention enhanced frequency domain feature map H is converted from the frequency domain to the spatial domain by using an inverse discrete Fourier transform to obtain a frequency enhanced feature and its expression is as follows:

[0118]

[0119] The frequency enhanced feature Z FE is added to the second intermediate feature map Z' to obtain the output feature of the frequency domain channel mixed feedforward network and its expression is as follows:

[0120] Z EN =Z FE +Z′。

[0121] In specific embodiments, given the input feature of the mixed pooling based Transformer module The calculation process of the mixed pooling based Transformer module is as follows:

[0122] W' = f HWTSA (LN(W))+W;

[0123] W" = f GNFN (LN(W')) + W';

[0124] wherein LN(·) represents a layer normalization operation, f HWTSA (·) represents a function corresponding to the mixed-pooling based self-attention module, f GNFN (·) represents a function corresponding to the frequency domain channel mixed feedforward network, W' represents an intermediate feature map, and W" represents an output feature of the mixed-pooling based Transformer module.

[0125] Specifically, referring to Figure 4 , the embodiment of the present application also proposes a frequency domain channel mixed feedforward network to enhance beneficial high-frequency information, thereby enhancing the feature representation capability of the lightweight image super-resolution model. The fourth convolutional layer in the frequency domain channel mixed feedforward network can remove redundant channel features. In order to adaptively adjust the frequency to emphasize high-frequency features beneficial to detail restoration, a channel attention mechanism is applied to , thereby obtaining an attention-enhanced frequency domain feature map H. Referring to Figure 5 , the mixed-pooling based self-attention module and the frequency domain channel mixed feedforward network are integrated to construct a mixed-pooling based Transformer module. The mixed-pooling based Transformer module is composed of a first layer normalization layer, a mixed-pooling based self-attention module, a second layer normalization layer, and a frequency domain channel mixed feedforward network. A mixed-pooling Transformer based lightweight image super-resolution model is constructed on the basis of the mixed-pooling based Transformer module. In the mixed-pooling Transformer based lightweight image super-resolution model, A mixed-pooling based Transformer modules are stacked, and the A-th mixed-pooling based Transformer module is the last mixed-pooling based Transformer module.

[0126] S2, obtaining a low-resolution image to be reconstructed and a scale factor and inputting them to the trained lightweight image super-resolution model, inputting the low-resolution image to the first convolutional layer to obtain an output feature of the first convolutional layer, obtaining an output feature of the last mixed-pooling based Transformer module after the output feature of the first convolutional layer passes through a plurality of mixed-pooling based Transformer modules, and obtaining a high-resolution reconstructed image after the output feature of the last mixed-pooling based Transformer module and the output feature of the first convolutional layer are added and sequentially pass through a sub-pixel convolutional layer and a second convolutional layer.

[0127] In specific embodiments, the calculation process of the lightweight image super-resolution model is as follows:

[0128] The low-resolution image A first convolutional layer with a kernel size of 1x1 is input to extract shallow features The expression is as follows:

[0129] F0=Conv 1×1 (I LR );

[0130] Where Conv 1×1 (·) represents a convolution operation with a kernel size of 1x1;

[0131] A total of A hybrid pooling-based Transformer modules and a residual connection are used to extract deep features The expression is as follows:

[0132]

[0133] F D =F A +F0;

[0134] Where, represents the function corresponding to the i-th hybrid pooling-based Transformer module, i∈[1,A], F i-1 represents the output feature of the i-1-th hybrid pooling-based Transformer module, F i represents the output feature of the i-th hybrid pooling-based Transformer module, represents the function corresponding to the A-th hybrid pooling-based Transformer module, F A-1 represents the output feature of the A-1-th hybrid pooling-based Transformer module, F A represents the output feature of the A-th hybrid pooling-based Transformer module;

[0135] A sub-pixel convolutional layer with a scale factor of scale and a second convolutional layer with a kernel size of 1x1 are used to upsample and reconstruct the deep feature F D to generate a high-resolution reconstructed image The expression is as follows:

[0136] I SR =Conv 1×1 (f up (F D ))

[0137] Where f up(·) represents a function corresponding to the sub-pixel convolutional layer.

[0138] Specifically, referring to Figure 6 , a low-resolution image to be reconstructed is given The scale factor scale is used as the input of the trained lightweight image super-resolution model, where the value of scale is the required magnification, for example, scale is 2, 3, 4, etc. The trained lightweight image super-resolution model is used to reconstruct the low-resolution image to be reconstructed, and the reconstruction result, i.e. a high-resolution reconstructed image, is obtained.

[0139] Further referring to Figure 7 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a lightweight image super-resolution device based on mixed pooling Transformer. The device embodiment corresponds to the method embodiment shown in Figure 1 , and the device can be specifically applied to various electronic devices.

[0140] The present application embodiment provides a lightweight image super-resolution device based on mixed pooling Transformer, which comprises:

[0141] The model construction module 1 is configured to construct and train a lightweight image super-resolution model based on mixed pooling Transformer to obtain a trained lightweight image super-resolution model. The lightweight image super-resolution model comprises a first convolutional layer, a plurality of mixed-pooling-based Transformer modules, a sub-pixel convolutional layer and a second convolutional layer connected in sequence. The mixed-pooling-based Transformer module comprises a first normalization layer, a mixed-pooling-based self-attention module, a second normalization layer and a frequency domain channel mixed feedforward network connected in sequence. The input feature of the mixed-pooling-based Transformer module is added to the output feature of the mixed-pooling-based self-attention module to obtain the input feature of the second normalization layer. The input feature of the second normalization layer is processed by the second normalization layer and the frequency domain channel mixed feedforward network to obtain the output feature of the frequency domain channel mixed feedforward network. The output feature of the frequency domain channel mixed feedforward network is added to the input feature of the second normalization layer to obtain the output feature of the mixed-pooling-based Transformer module.

[0142] The reconstruction module 2 is configured to obtain a low-resolution image to be reconstructed and a scale factor and input to the trained lightweight image super-resolution model, the low-resolution image is input to a first convolutional layer to obtain output features of the first convolutional layer, the output features of the first convolutional layer are input to a plurality of mixed-pooling-based Transformer modules to obtain output features of the last mixed-pooling-based Transformer module, and the output features of the last mixed-pooling-based Transformer module are added to the output features of the first convolutional layer and sequentially input to a sub-pixel convolutional layer and a second convolutional layer to obtain a high-resolution reconstructed image.

[0143] Figure 8 The hardware structure schematic diagram of the electronic device provided by the embodiment of the present application is shown in the figure. Figure 8 As shown in the figure, the electronic device of the embodiment includes a processor 801 and a memory 802; the memory 802 is used for storing computer execution instructions; the processor 801 is used for executing the computer execution instructions stored in the memory to realize each step executed by the electronic device in the above-mentioned embodiment. For details, please refer to the related description in the foregoing method embodiment.

[0144] Optionally, the memory 802 can be independent or integrated with the processor 801.

[0145] When the memory 802 is independently arranged, the electronic device further includes a bus 803 for connecting the memory 802 and the processor 801.

[0146] The embodiment of the present application further provides a computer storage medium, and the computer storage medium stores computer execution instructions; when the processor 801 executes the computer execution instructions, the method as above is realized.

[0147] The embodiment of the present application further provides a computer program product, and the computer program product includes a computer program; when the computer program is executed by the processor 801, the method as above is realized.

[0148] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules is only a logical function division. There can be another division manner in actual implementation, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0149] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to implement the embodiments of the present application.

[0150] In addition, each functional module in various embodiments of the present application can be integrated in one processing unit, or each module can be physically present alone, or two or more modules can be integrated in one unit. The unit formed by the above modules can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0151] The integrated modules realized in the form of software functional modules can be stored in a computer readable storage medium. The software functional modules stored in a storage medium include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 801 to execute part of the steps of the various embodiment methods of the present application.

[0152] It should be understood that the above processor 801 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor 801 can also be any conventional processor 801, etc. The steps of the method disclosed in the application can be directly embodied as hardware processor 801 execution, or executed by a combination of hardware and software modules in the processor 801.

[0153] The memory 802 can include a high-speed RAM memory, and can also include a non-volatile storage NVM, for example, at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0154] The bus 803 can be an Industry Standard Architecture (ISA), Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 803 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 803 in the drawings of the present application does not limit to only one bus 803 or one type of bus 803.

[0155] The storage medium described above can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0156] An exemplary storage medium is coupled to the processor 801, so that the processor 801 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor 801. The processor 801 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 801 and the storage medium can also exist as discrete components in an electronic device or a host device.

[0157] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program, when executed, executes steps including the above-mentioned method embodiments; and the foregoing storage medium includes ROM, RAM, magnetic disk or optical disk and various storage media that can store program codes.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A lightweight image super-resolution method based on mixed-pooling Transformer, characterized in that, The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The input feature of the mixed-pooling-based self-attention module The input feature of the mixed-pooling-based self-attention module The expression is as follows: ； wherein, denotes a function corresponding to a channel down-sampling convolutional layer with a scaling factor of 2, whose kernel size is 1 x 1; respectively utilize a global max pooling layer and a global average pooling layer on the first intermediate feature map to perform down-sampling, respectively obtaining first feature and second feature , expressions of which are as follows: ； ； wherein, represents a function corresponding to a global max pooling layer, represents a function corresponding to a global average pooling layer; The first feature and the second feature The channel concatenation layer is input along the channel dimension, and a convolution operation with a kernel size of 1*1 is performed through the third convolution layer to obtain a mixed feature embedding The expression is as follows: ； wherein, denotes a convolution operation with a kernel size of 1x1, denotes a concatenation operation along the channel dimension; the frequency domain channel mixing feedforward network comprises a fourth convolution layer and a channel attention layer; Input features of the frequency domain channel mixing feedforward network The convolution operation with a kernel size of 1x1 is first performed on the fourth convolution layer to obtain a second intermediate feature map As shown in the following formula: ； wherein, denotes a convolution operation with a kernel size of 1x1, C, H and W denote the number of channels, length and width of the input feature of the frequency domain channel mixing feedforward network, respectively, which are the same as the number of channels, length and width of the input feature of the mixing-pooling-based self-attention module, denotes a set of real numbers; using a discrete Fourier transform on the second intermediate feature map from the spatial domain to the frequency domain to obtain a frequency domain feature map , denotes a set of real numbers, whose expression is as follows: ； The method comprises the following steps: The frequency domain feature map Inputting the channel attention layer yields the attention-enhanced frequency domain feature map H, whose expression is as follows: ； wherein, denotes a function corresponding to the channel attention layer; The attention-enhanced frequency domain feature map H is converted from the frequency domain to the spatial domain by using an inverse discrete Fourier transform to obtain a frequency-enhanced feature The expression is as follows: ； The frequency enhancement feature is added to the second intermediate feature map to obtain an output feature of the frequency domain channel mixing feedforward network , and its expression is as follows: ； The method comprises the following steps:

2. The lightweight image super-resolution method based on mixed-pooling Transformer according to claim 1, characterized in that, The method comprises the following steps: input features of the mixed-pooling based self-attention module performing partitioning into N first non-overlapping windows of size SxS wherein wherein C, H and W denote the number of channels, length and width of the input features of the mixed-pooling based self-attention module, respectively, denotes a set of real numbers, the first linear layer being configured to transform the first non-overlapping windows into a query matrix : ； The input feature X of the mixed-pooling-based self-attention module is generated into a mixed feature embedding by the mixed-pooling feature embedding module The expression is as follows: ； wherein, represents a function corresponding to the mixed-pooling feature embedding module; Embed the hybrid features Divide into N windows with a size of Second non-overlapping window ,in Then, using the second linear layer and the third linear layer The second non-overlapping window Convert them into bond matrices respectively Sum matrix Its expression is as follows: ； ； The self-attention layer performs a self-attention calculation according to the query matrix, the key matrix, and the value matrix input, to obtain enhanced features The expression is as follows: ； where T denotes the transpose operation, denotes a function of Softmax normalization. Finally, the enhanced features of shape N x S 2 x C are converted to C x H x W and serve as the output features of the hybrid pooling-based self-attention module.

3. The lightweight image super-resolution method based on mixed-pooling Transformer according to claim 1, characterized in that, Given the input features of the hybrid-pooling-based Transformer module The calculation process of the hybrid-pooling-based Transformer module is as follows: ； ； wherein, denotes a layer normalization operation, denotes a function corresponding to the hybrid pooling based self-attention module, denotes a function corresponding to the frequency domain channel mixed feed-forward network, denotes an intermediate feature map, denotes an output feature of the hybrid pooling based Transformer module.

4. The lightweight image super-resolution method based on mixed-pooling Transformer according to claim 1, characterized in that, The method comprises the following steps: extracting low-resolution images a first convolutional layer with a kernel size of 1x1 is input to extract shallow features whose expression is as follows: ； wherein denotes a convolution operation with a kernel size of 1 x 1 ; A number of A hybrid pooling based Transformer modules and one residual connection are used to extract deep features whose expression is as follows: ； ； ； wherein, denotes a function corresponding to the i-th mixed-pooling-based Transformer module, , denotes an output feature of the i-1-th mixed-pooling-based Transformer module, denotes an output feature of the i-th mixed-pooling-based Transformer module, denotes a function corresponding to the A-th mixed-pooling-based Transformer module, denotes an output feature of the A-1-th mixed-pooling-based Transformer module, denotes an output feature of the A-th mixed-pooling-based Transformer module; The deep feature is processed by a sub-pixel convolution layer with a scale factor of scale and a second convolution layer with a kernel size of 1x1 Upsampling and reconstruction are performed to generate a high-resolution reconstructed image The expression is as follows: ； wherein, denotes the function corresponding to the subpixel convolutional layer.

5. A lightweight image super-resolution device based on hybrid-pooling Transformer, adopting the lightweight image super-resolution method based on hybrid-pooling Transformer in any one of claims 1-4, characterized in that, The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the The model construction module is configured to construct a lightweight image super-resolution model based on a hybrid pooling Transformer and train the same to obtain a trained lightweight image super-resolution model, the lightweight image super-resolution model comprising a first convolutional layer, a plurality of hybrid pooling based Transformer modules, a sub-pixel convolutional layer and a second convolutional layer connected in sequence; the hybrid pooling based Transformer module comprises a first layer normalization layer, a hybrid pooling based self-attention module, a second layer normalization layer and a frequency domain channel hybrid feedforward network connected in sequence, an input feature of the hybrid pooling based Transformer module being added to an output feature of the hybrid pooling based self-attention module to obtain an input feature of the second layer normalization layer, the input feature of the second layer normalization layer being input to the second layer normalization layer and the frequency domain channel hybrid feedforward network to obtain an output feature of the frequency domain channel hybrid feedforward network, the output feature of the frequency domain channel hybrid feedforward network being added to the input feature of the second layer normalization layer to obtain an output feature of the hybrid pooling based Transformer module; The reconstruction module is configured to obtain a low-resolution image to be reconstructed and a scale factor and input the same to the trained lightweight image super-resolution model, the low-resolution image being input to the first convolutional layer to obtain an output feature of the first convolutional layer, the output feature of the first convolutional layer being input to the plurality of hybrid pooling based Transformer modules to obtain an output feature of the last hybrid pooling based Transformer module, the output feature of the last hybrid pooling based Transformer module being added to the output feature of the first convolutional layer and input to the sub-pixel convolutional layer and the second convolutional layer in sequence to obtain a high-resolution reconstructed image.

6. An electronic device, comprising: one or more processors; storage configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method according to any one of claims 1-4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1-4.

Images