Lightweight image super-resolution reconstruction method and system

Abstract

The invention discloses a lightweight image super-resolution reconstruction method and system. A lightweight super-resolution neural network based on a partial convolution information distillation mechanism module and a multi-scale attention mechanism module is constructed; the lightweight super-resolution neural network comprises a shallow layer extraction module, a deep layer feature extraction module and a reconstruction module. The deep feature extraction module comprises five multi-scale partial convolution residual distillation modules, a connection fusion layer, a second convolution layer, a pixel attention module and a second partial convolution layer; and the reconstruction module comprises a first sub-pixel convolution module, a second sub-pixel convolution module and a third convolution layer. According to the method, channel redundancy can be effectively avoided, the feature extraction process is simplified, more accurate spatial information distribution can be obtained, module parameters are reduced by adopting partial convolution, and better balance between parameters and performance of the network is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically provides a lightweight image super-resolution method and system based on information distillation and multi-scale attention. Background Art

[0002] Single image super-resolution is an important task in image processing, which aims to reconstruct a high-resolution image from a low-resolution image, while optimizing details and textures to improve the quality of visual perception. It is currently widely used in various scenarios in real life, including security monitoring, remote sensing, video enhancement, image segmentation, etc., and has therefore attracted widespread attention from academia and industry.

[0003] In recent years, with the gradual development of deep learning technology, super-resolution technology has gradually matured, and there are many powerful super-resolution networks that can achieve high-quality image restoration. However, in order to achieve high-quality image restoration effects, these networks often have very large parameters and high model complexity, which makes them difficult to deploy on mobile terminals or under conditions where device performance is not very friendly. In order to solve this problem, people began to study efficient super-resolution reconstruction algorithms. Summary of the invention

[0004] Purpose of the invention: In order to solve the problem that the existing super-resolution network parameters are too large and it is difficult to achieve a good balance between the parameters and network performance. The present invention provides a lightweight image super-resolution reconstruction method, which can enhance the resolution of the image with a smaller model and calculation amount, and can be used in the actual industry.

[0005] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A lightweight image super-resolution reconstruction method comprises the following steps:

[0007] Step 1: Obtain lightweight images and preprocess them as training sets.

[0008] Step 2: Construct a lightweight super-resolution neural network based on partial convolution information distillation mechanism module and multi-scale attention mechanism module.

[0009] The lightweight super-resolution neural network includes a shallow extraction module, a deep feature extraction module and a reconstruction module.

[0010] The shallow extraction module includes a first convolutional layer.

[0011] The deep feature extraction module includes five multi-scale partial convolution residual distillation modules, a connection fusion layer, a second convolution layer, a pixel attention module, and a second partial convolution layer. The five multi-scale partial convolution residual distillation modules are connected in sequence, and the output of the first convolution layer enters the first multi-scale partial convolution residual distillation module among the five multi-scale partial convolution residual distillation modules. At the same time, the outputs of the five multi-scale partial convolution residual distillation modules respectively enter the connection fusion layer. The connection fusion layer, the second convolution layer, the pixel attention module, and the second partial convolution layer are connected in sequence.

[0012] The reconstruction module includes a first sub-pixel convolution module, a second sub-pixel convolution module and a third convolution layer, the first sub-pixel convolution module includes a first upsampling operation and a first addition layer. The second sub-pixel convolution module includes a second addition layer and a second upsampling operation, the output of the first convolution layer enters the third convolution layer and the second addition layer respectively, the third convolution layer, the first upsampling operation, and the first addition layer are connected in sequence, and the second addition layer, the second upsampling operation, and the first addition layer are connected in sequence.

[0013] Step 3: Train the constructed lightweight super-resolution neural network through the training set to obtain a trained lightweight super-resolution neural network.

[0014] Step 4: Reconstruct the low-resolution image through the trained lightweight super-resolution neural network to obtain a reconstructed super-resolution image.

[0015] Preferably: the multi-scale partial convolution residual distillation module includes 4 fourth partial residual modules, 4 fourth convolutional layers, a fourth connection fusion layer, a channel attention module, a multi-scale enhanced spatial attention module, and a fourth addition layer one. The 4 fourth partial residual modules are respectively the fourth partial residual module one, the fourth partial residual module two, the fourth partial residual module three, and the fourth partial residual module four. The 4 fourth convolutional layers are respectively the fourth convolutional layer one, the fourth convolutional layer two, the fourth convolutional layer three, and the fourth convolutional layer four. The fourth partial residual module one, the fourth partial residual module two, the fourth partial residual module three, the fourth partial residual module four, the fourth connection fusion layer, the fourth convolutional layer four, the channel attention module, the multi-scale enhanced spatial attention module, and the fourth addition layer one are connected in sequence. The input of the multi-scale partial convolution residual distillation module enters the fourth partial residual module one, the fourth convolutional layer one, and the fourth addition layer one respectively, and the output of the fourth convolutional layer one is connected to the fourth connection fusion layer. The input of the fourth convolutional layer 2 is connected to the output of the fourth partial residual module 1, and the output of the fourth convolutional layer 2 is connected to the fourth connection fusion layer. The input of the fourth convolutional layer 3 is connected to the output of the fourth partial residual module 2, and the output of the fourth convolutional layer 3 is connected to the fourth connection fusion layer.

[0016] Preferably: the multi-scale enhanced spatial attention module includes the fifth convolution layer one, the fifth compressed convolution layer one, the fifth maximum pooling layer one, the fifth compressed convolution layer two, the fifth maximum pooling layer two, the fifth compressed convolution layer three, the fifth maximum pooling layer three, the fifth compressed convolution layer four, the fifth local variance module, the fifth addition layer one, the fifth partial convolution group one, the fifth partial convolution group two, the fifth partial convolution group three, the fifth upsampling module, the fifth addition layer two, the fifth convolution layer two, the fifth activation function module, the element multiplication operation module, and the fifth addition layer three. The input of the multi-scale enhanced spatial attention module enters the fifth convolution layer one, the element multiplication operation module, and the fifth addition layer three respectively. The output of the fifth convolution layer one enters the fifth compressed convolution layer one, the fifth compressed convolution layer two, the fifth compressed convolution layer three, and the fifth compressed convolution layer four respectively. The fifth compressed convolution layer one, the fifth maximum pooling layer one, and the fifth partial convolution group one are connected in sequence. The fifth compressed convolution layer 2, the fifth maximum pooling layer 2, and the fifth partial convolution group 2 are connected in sequence. The fifth compressed convolution layer 3, the fifth maximum pooling layer 3, the fifth addition layer 1, and the fifth partial convolution group 3 are connected in sequence. The fifth compressed convolution layer 4, the fifth local variance module, and the fifth addition layer 1 are connected in sequence. The outputs of the fifth partial convolution group 1, the fifth partial convolution group 2, and the fifth partial convolution group 3 respectively enter the fifth upsampling module, and the fifth upsampling module, the fifth addition layer 2, the fifth convolution layer 2, the fifth activation function module, the element multiplication operation module, and the fifth addition layer 3 are connected in sequence. The output of the fifth convolution layer 1 enters the fifth addition layer 2.

[0017] Preferably: the fourth partial residual module includes a fourth partial convolution layer, a fourth addition layer 2, and a Gelu activation layer connected in sequence, and the input of the fourth partial residual module enters the fourth partial convolution layer and the fourth addition layer 2 respectively. The fourth partial convolution layer includes a fourth input layer and a fourth output layer, the input of the fourth input layer enters the fourth output layer through a convolution operation, and the input of the fourth input layer enters the fourth output layer through direct mapping.

[0018] Preferably: the information distillation method of the multi-scale partial convolution residual distillation module is:

[0019] Step S101, the input F of the multi-scale partial convolution residual distillation module in It is divided into two paths, one path is input into the fourth residual module 1, and the other path is input into the fourth convolutional layer 1 for feature refinement, which is recorded as:

[0020] F c1 =PRB1(F in ),

[0021] F d1 =C1(F in )

[0022] Where C1 represents the convolution operation of the fourth convolutional layer, PRB1 represents the residual operation of the fourth residual module, and F d1 represents the features extracted by the fourth convolutional layer, F c1 Represents the features of the fourth part of the residual module being sent to the next step.

[0023] Step S102: the obtained F c1 Further refinement, the process is expressed as:

[0024] F c2 =PRB2(PRB1(F in ))

[0025] F d2 =C2(PRB1(F in ))

[0026] Where C2 represents the second convolution operation of the fourth convolutional layer, PRB2 represents the second residual operation of the fourth residual module, and F d2 represents the features extracted by the fourth convolutional layer 2, F c2 It indicates that the fourth part of the residual module 2 is sent to the next step of the feature.

[0027] Step S103: c2 Continuing with the refinement, the process is expressed as:

[0028] F c3 =PRB3(PRB2(PRB1(F in ))),

[0029] F d3 =C3(PRB2(PRB1(F in )))

[0030] Where C3 represents the third convolution operation of the fourth convolutional layer, PRB3 represents the third residual operation of the fourth residual module, and F d3 represents the features extracted by the fourth convolutional layer, F c3 It indicates that the fourth part of the residual module 3 is sent to the next step of the feature.

[0031] Step S104: obtain the F c3 The fourth part is sent to the residual module 4 for the final refinement. The process is expressed as:

[0032] F c4 =PRB4(PRB3(PRB2(PRB1(F in ))))

[0033] PRB4 represents the fourth residual module, the four-part residual operation, Fc4 Represents the features of the fourth part of the residual module four being sent to the next step of operation.

[0034] Step S105, the features extracted in the above steps are fused, and the process is expressed as:

[0035] F fused

[0036] =Concat(C1(F in ), C2(PRB1(F in )), C3(PRB2(PRB1(F in ))),PRB4(PRB3(PRB2(PRB1(F in )))))

[0037] In the formula, F fused Represents the fusion feature, and Concat represents the fusion operation of the fourth connection fusion layer.

[0038] Preferably: the multi-scale attention method of the multi-scale partial convolution residual distillation module is:

[0039] Step S201: First, the input F of the multi-scale attention mechanism is s Through a fifth convolutional layer to reduce the number of channels to reduce the amount of network parameters, the process can be expressed as:

[0040] F in,1 =C 1×1 (F S )

[0041] Among them, C 1×1 represents the convolution operation of the fifth convolutional layer, F S represents the input of the multi-scale attention mechanism, F in,1 Represents the convolution operation result of the fifth convolutional layer.

[0042] Step S202: the obtained F in,1 Compression is performed on three scales and local variance is introduced. The process is expressed as:

[0043]

[0044] in, represents compressed local features, B3 represents the maximum pooling operation after the fifth compressed convolution layer 1, B5 represents the maximum pooling operation after the fifth compressed convolution layer 2, B7 represents the maximum pooling operation after the fifth compressed convolution layer 3, B L-vatIt means that after the fourth compressed convolution operation of the fifth compressed convolution layer, the fifth local variance module is operated.

[0045] Step S203, the obtained The features are further refined by sending them to the convolution group constructed by partial convolution. The process is expressed as:

[0046]

[0047] in, Indicates the result obtained after partial convolution group, B g Represents a partial convolution group operation.

[0048] Step S204, the obtained Up-sampling is performed to obtain a high-resolution feature map, and the high-resolution feature map is added to the result obtained in step S201. The process is expressed as:

[0049]

[0050] in, represents the obtained high-resolution feature map, B up Represents an upsampling operation.

[0051] Step S205: The high-resolution feature map obtained in step S204 is After the fifth convolutional layer, it is adjusted to F S The number of channels is the same, and then processed by the Sigmoid function and added to F S Add. The process is expressed as:

[0052]

[0053] in, Represents element-wise multiplication, H conv1-1 represents the second convolution operation of the fifth convolutional layer, and Sigmoid represents the Sigmoid function.

[0054] Preferably: the shallow extraction process of the shallow extraction module is:

[0055] F0=B SF (I LR )

[0056] Among them, F0 represents the result of shallow feature extraction, B SF represents the shallow feature extraction process, I LR Represents the input image.

[0057] The feature extraction process of the deep feature extraction module is as follows:

[0058] F1=B DF (BSF (I LR ))

[0059] Among them, F1 represents the deep feature extraction result, B DF represents the deep feature extraction process, B SF represents the shallow feature extraction process, I LR Represents the input image.

[0060] The reconstruction process of the reconstruction module is:

[0061] I SR =B SR (B DF (B SF (I LR )))+B up (I LR )

[0062] Among them, I SR Indicates the final result, B SR represents the second sub-pixel convolution process, B DF represents the deep feature extraction process, B SF represents the shallow feature extraction process, I LR represents the input image, B up Represents the first sub-pixel convolution process.

[0063] Preferably: the channel attention module combines the fusion feature F by comparing the channel attention fused Processing is performed to obtain the input F of the multi-scale attention mechanism s .

[0064] Another object of the present invention is to provide a lightweight image super-resolution reconstruction system for implementing the lightweight image super-resolution reconstruction method, comprising an input unit, a lightweight super-resolution neural network unit, and an output unit, wherein:

[0065] The input unit is used to input a low-resolution image to be reconstructed.

[0066] The lightweight super-resolution neural network unit is used to store the trained lightweight super-resolution neural network. The low-resolution image is reconstructed by the trained lightweight super-resolution neural network to obtain a reconstructed super-resolution image.

[0067] The output unit is used to output the reconstructed super-resolution image.

[0068] Another object of the present invention is to provide a computer system, comprising a memory and a processor, wherein the memory is used to store a computer program / instruction, and the processor is used to execute the computer program / instruction to implement the lightweight image super-resolution reconstruction method.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] The information distillation mechanism based on partial convolution can effectively reduce the number of network parameters. The multi-scale enhanced spatial attention mechanism can effectively extract and fuse features of different scales to obtain more useful image features by introducing multi-scale information. Therefore, the present invention has a lightweight effect, and the number of parameters of the model is only 550K. At the same time, the designed super-resolution network has a good balance between network parameters and performance. This solves the problem that the number of parameters of the original super-resolution network is too large and it is difficult to achieve a good balance between the number of parameters and network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A specific schematic diagram of the super-resolution network structure proposed in the present invention;

[0072] Figure 2 It is a schematic diagram of the structure of partial convolution;

[0073] Figure 3 Schematic diagram of some residual modules mentioned in the present invention;

[0074] Figure 4 Schematic diagram of the information distillation mechanism based on partial convolution proposed in the present invention;

[0075] Figure 5 Schematic diagram of the multi-scale enhanced spatial attention mechanism proposed in this invention. DETAILED DESCRIPTION

[0076] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0077] The present invention designs a lightweight image super-resolution reconstruction method, such as Figure 1-5 As shown, the following steps are included:

[0078] Step 1: Obtain lightweight images and preprocess them as training sets.

[0079] In another embodiment, the selected training data set is specifically the DF2K data set, which includes 3550 images. The test sets used are the publicly used Set5, Set14, Urban100, B100, and Manga109. The low-resolution images in DF2K are cropped into image blocks of 96×96, 144×144, and 192×192 sizes and randomly rotated by 90°, 180°, 270°, and horizontally flipped as network inputs of ×2, ×3, and ×4 scales, respectively.

[0080] Step 2: Construct a lightweight super-resolution neural network based on partial convolution information distillation mechanism module and multi-scale attention mechanism module.

[0081] The lightweight super-resolution neural network includes a shallow extraction module, a deep feature extraction module and a reconstruction module.

[0082] The shallow extraction module includes a first convolution layer, which is a 3×3 convolution layer, and the purpose is to expand the number of channels of the input image from the original RGB three channels to a feature map with 64 channels.

[0083] Given an input I LR , B SF and F0 represent the shallow feature extraction process and the result of shallow feature extraction respectively. The shallow extraction process of the shallow extraction module is:

[0084] F0=B SF (I LR )

[0085] Among them, F0 represents the result of shallow feature extraction, B SF represents the shallow feature extraction process, I LR Represents the input image.

[0086] The purpose of the deep feature extraction module is to further process the features extracted by the shallow feature extraction module. The deep feature extraction module includes five multi-scale partial convolution residual distillation modules, a connection fusion layer, a second convolution layer, a pixel attention module, and a second partial convolution layer. The five multi-scale partial convolution residual distillation modules are connected in sequence, and the output of the first convolution layer enters the first multi-scale partial convolution residual distillation module among the five multi-scale partial convolution residual distillation modules. At the same time, the outputs of the five multi-scale partial convolution residual distillation modules enter the connection fusion layer respectively. The connection fusion layer, the second convolution layer, the pixel attention module, and the second partial convolution layer are connected in sequence. The second convolution layer is a 1×1 convolution, and the second partial convolution layer is a 3×3 partial convolution with a ratio of 1 / 4.

[0087] Assume that the deep feature extraction process is B DF, the input feature map is F0, and the output result is F1, then the feature extraction process of the deep feature extraction module is:

[0088] F1=B DF (B SF (I LR ))

[0089] Among them, F1 represents the deep feature extraction result, B DF represents the deep feature extraction process, B SF represents the shallow feature extraction process, I LR Represents the input image.

[0090] In another embodiment, the multi-scale partial convolution residual distillation module includes 4 fourth partial residual modules, 4 fourth convolution layers, a fourth connection fusion layer, a channel attention module, a multi-scale enhanced spatial attention module, and a fourth addition layer 1. The 4 fourth partial residual modules are respectively the fourth partial residual module 1, the fourth partial residual module 2, the fourth partial residual module 3, and the fourth partial residual module 4. The 4 fourth partial residual modules are, from front to back, a 3×3 partial residual module with a ratio of 1, a 3×3 partial residual module with a ratio of 1 / 2, a 3×3 partial residual module with a ratio of 1 / 4, and a 3×3 partial residual module with a ratio of 1. , the four fourth convolutional layers are respectively the fourth convolutional layer 1, the fourth convolutional layer 2, the fourth convolutional layer 3, and the fourth convolutional layer 4, and the four fourth convolutional layers are all 1×1 convolutional layers. The fourth partial residual module 1, the fourth partial residual module 2, the fourth partial residual module 3, the fourth partial residual module 4, the fourth connection fusion layer, the fourth convolutional layer 4, the channel attention module, the multi-scale enhanced spatial attention module, and the fourth addition layer 1 are connected in sequence. The input of the multi-scale partial convolution residual distillation module enters the fourth partial residual module 1, the fourth convolutional layer 1, and the fourth addition layer 1 respectively, and the output of the fourth convolutional layer 1 is connected to the fourth connection fusion layer. The input of the fourth convolutional layer 2 is connected to the output of the fourth partial residual module 1, and the output of the fourth convolutional layer 2 is connected to the fourth connection fusion layer. The input of the fourth convolutional layer 3 is connected to the output of the fourth partial residual module 2, and the output of the fourth convolutional layer 3 is connected to the fourth connection fusion layer.

[0091] The fourth partial residual module includes a fourth partial convolution layer, a fourth addition layer 2, and a Gelu activation layer connected in sequence, and the input of the fourth partial residual module enters the fourth partial convolution layer and the fourth addition layer 2 respectively. The fourth partial convolution layer includes a fourth input layer and a fourth output layer, and the input of the fourth input layer enters the fourth output layer through a convolution operation, and the input of the fourth input layer enters the fourth output layer through direct mapping. The fourth partial convolution layer is a 3×3 partial convolution layer.

[0092] In another embodiment, the information distillation method of the multi-scale partial convolution residual distillation module is:

[0093] Step S101, the input F of the multi-scale partial convolution residual distillation module in It is divided into two paths, one path is input into the fourth residual module 1, and the other path is input into the fourth convolutional layer 1 for feature refinement, which is recorded as:

[0094] F c1 =PRB1(F in ),

[0095] F d1 =C1(F in )

[0096] In the formula, C1 represents the convolution operation of the fourth convolutional layer, which is used to refine the features. The fourth convolutional layer is a 1×1 convolutional layer. PRB1 represents the residual operation of the fourth residual module. F d1 represents the features extracted by the fourth convolutional layer, F c1 Represents the features of the fourth part of the residual module being sent to the next step.

[0097] Step S102: the obtained F c1 Further refinement, the process is expressed as:

[0098] F c2 =PRB2(PRB1(F in ))

[0099] F d2 =C2(PRB1(F in ))

[0100] In the formula, C2 represents the convolution operation of the fourth convolutional layer 2, the fourth convolutional layer 2 is a 1×1 convolutional layer used to refine the features, PRB2 represents the residual operation of the fourth residual module 2, and F d2 represents the features extracted by the fourth convolutional layer 2, F c2 It indicates that the fourth part of the residual module 2 is sent to the next step of the feature.

[0101] Step S103: c2 Continuing with the refinement, the process is expressed as:

[0102] F c3 =PRB3(PRB2(PRB1(F in ))),

[0103] F d3 =C3(PRB2(PRB1(F in )))

[0104] Where, C3 represents the third convolution operation of the fourth convolutional layer, the fourth convolutional layer is a 1×1 convolutional layer used to refine the features, PRB3 represents the third residual operation of the fourth residual module, and F d3 represents the features extracted by the fourth convolutional layer, F c3 It indicates that the fourth part of the residual module 3 is sent to the next step of the feature.

[0105] Step S104: obtain the F c3 The fourth part is sent to the residual module 4 for the final refinement. The process is expressed as:

[0106] F c4 =PRB4(PRB3(PRB2(PRB1(F in ))))

[0107] PRB4 represents the fourth residual module, the four-part residual operation, F c4 Represents the features of the fourth part of the residual module four being sent to the next step of operation.

[0108] Step S105, the features extracted in the above steps are fused, and the process is expressed as:

[0109] F fused

[0110] =Concat(C1(F in ), C2(PRB1(F in )), C3(PRB2(PRB1(F in ))),PRB4(PRB3(PRB2(PRB1(F in )))))

[0111] In the formula, F fused Represents the fusion feature, and Concat represents the fusion operation of the fourth connection fusion layer.

[0112] In another embodiment, the multi-scale enhanced spatial attention module includes a fifth convolutional layer one, a fifth compressed convolutional layer one, a fifth maximum pooling layer one, a fifth compressed convolutional layer two, a fifth maximum pooling layer two, a fifth compressed convolutional layer three, a fifth maximum pooling layer three, a fifth compressed convolutional layer four, a fifth local variance module, a fifth addition layer one, a fifth partial convolution group one, a fifth partial convolution group two, a fifth partial convolution group three, a fifth upsampling module, a fifth addition layer two, a fifth convolutional layer two, a fifth activation function module, an element multiplication operation module, and a fifth addition layer three. The inputs of the multi-scale enhanced spatial attention module enter the fifth convolutional layer one, the element multiplication operation module, and the fifth addition layer three respectively. The outputs of the fifth convolutional layer one enter the fifth compressed convolutional layer one, the fifth compressed convolutional layer two, the fifth compressed convolutional layer three, and the fifth compressed convolutional layer four respectively. The fifth compressed convolutional layer one, the fifth maximum pooling layer one, and the fifth partial convolution group one are connected in sequence. The fifth compressed convolution layer 2, the fifth maximum pooling layer 2, and the fifth partial convolution group 2 are connected in sequence. The fifth compressed convolution layer 3, the fifth maximum pooling layer 3, the fifth addition layer 1, and the fifth partial convolution group 3 are connected in sequence. The fifth compressed convolution layer 4, the fifth local variance module, and the fifth addition layer 1 are connected in sequence. The outputs of the fifth partial convolution group 1, the fifth partial convolution group 2, and the fifth partial convolution group 3 respectively enter the fifth upsampling module, and the fifth upsampling module, the fifth addition layer 2, the fifth convolution layer 2, the fifth activation function module, the element multiplication operation module, and the fifth addition layer 3 are connected in sequence. The output of the fifth convolution layer 1 enters the fifth addition layer 2.

[0113] In another embodiment, the channel attention module combines the fusion feature F by comparing the channel attention fused Processing is performed to obtain the input F of the multi-scale attention mechanism s .

[0114] In another embodiment, the multi-scale attention method of the multi-scale partial convolution residual distillation module is:

[0115] Step S201: First, the input F of the multi-scale attention mechanism is s Through a fifth convolutional layer to reduce the number of channels to reduce the amount of network parameters, the process can be expressed as:

[0116] F in,1 =C 1×1 (F S )

[0117] Among them, C 1×1 Represents the convolution operation of the fifth convolutional layer, the fifth convolutional layer is a 1×1 convolutional layer, F S represents the input of the multi-scale attention mechanism, F in,1 Represents the convolution operation result of the fifth convolutional layer.

[0118] Step S202: the obtained F in,1 Compression is performed on three scales and local variance is introduced. The process is expressed as:

[0119]

[0120] in, represents compressed local features, B3 represents the maximum pooling operation after the fifth compressed convolution layer 1, B5 represents the maximum pooling operation after the fifth compressed convolution layer 2, B7 represents the maximum pooling operation after the fifth compressed convolution layer 3, B3, B5, B7 represent the maximum pooling operation after the fifth maximum pooling layer 3, B3, B5, B7 represent the maximum pooling operation after the 3×3 convolution with a step size of 2 and the step sizes of 3, 5, and 7 respectively. L-vat It means that after the operation of the fifth compressed convolution layer and the fourth compressed convolution operation, it is operated by the fifth local variance module. L-vat Represents a local variance operation with scale 7.

[0121] Step S203, the obtained The features are further refined by sending them to the convolution group constructed by partial convolution. The process is expressed as:

[0122]

[0123] in, Indicates the result obtained after the partial convolution group, which consists of two partial convolutions, B g Represents a partial convolution group operation.

[0124] Step S204: obtain the low-resolution feature map Up-sampling is performed to obtain a high-resolution feature map, and the high-resolution feature map is added to the result obtained in step S201. The process is expressed as:

[0125]

[0126] in, represents the obtained high-resolution feature map, B up Represents an upsampling operation.

[0127] Step S205: The high-resolution feature map obtained in step S204 is After the fifth convolutional layer, it is adjusted to F S The number of channels is the same, and then processed by the Sigmoid function and added to F S Add. The process is expressed as:

[0128]

[0129] Among them, F out Indicates the output, Represents element-wise multiplication, H conv1-1 Represents the convolution operation of the fifth convolutional layer 2. The fifth convolutional layer 2 is a 1×1 convolutional layer. Sigmoid represents the Sigmoid function.

[0130] The reconstruction module includes a first sub-pixel convolution module, a second sub-pixel convolution module and a third convolution layer. The first sub-pixel convolution module includes a first upsampling operation and a first addition layer. The second sub-pixel convolution module includes a second addition layer and a second upsampling operation. The output of the first convolution layer enters the third convolution layer and the second addition layer respectively. The third convolution layer, the first upsampling operation and the first addition layer are connected in sequence. The second addition layer, the second upsampling operation and the first addition layer are connected in sequence. The third convolution layer is a 3×3 convolution.

[0131] Assume that the process of the sub-pixel convolution module is B SR The initial input feature map undergoes a 3×3 convolution and sub-pixel convolution process to be B up , the input feature map is F1, and the final result is I SR , then the reconstruction process of the reconstruction module is:

[0132] I SR =B SR (B DF (B SF (I LR )))+B up (I LR )

[0133] Among them, I SR Indicates the final result, B SR represents the second sub-pixel convolution process, B DF represents the deep feature extraction process, B SF represents the shallow feature extraction process, I LR represents the input image, B up Represents the first sub-pixel convolution process.

[0134] The loss function is chosen as:

[0135] I loss =∥I SR -I HR ∥1+α∥F(I SR )-F(I HR )∥

[0136] Among them, I loss Represents the loss value, I SR Indicates that IHR Indicates that ∥I SR -I HR ∥1 means, F(.) means, ∥F(I SR )-F(I HR )∥ represents the frequency domain loss, and α is the adaptive weight. The gradient descent method is used to iterate the model parameters during the training process; the optimizer uses Adam, and the parameters are set as follows: the batch size is 16, the exponential decay rate β1 of the first-order moment estimate is 0.9, the exponential decay rate β2 of the second-order moment estimate is 0.999, and the short floating point type value ε to maintain numerical stability is 10 -8 , the initial learning rate is set to 1e -3 , using the cosine annealing training strategy, the number of training iterations is 1×10 6 .

[0137] Step 3: Train the constructed lightweight super-resolution neural network through the training set to obtain a trained lightweight super-resolution neural network.

[0138] Step 4: Reconstruct the low-resolution image through the trained lightweight super-resolution neural network to obtain a reconstructed super-resolution image.

[0139] In the reconstruction process, assume that the input image is F in , the output image is F out , then the multi-scale partial convolution residual distillation module process involved can be described as:

[0140] (a) Input F in It is divided into two paths, one of which is extracted through 1×1 convolution to obtain F di ; The other path is further refined to obtain F ci .

[0141] (b) The extracted F di Perform Concat operation to get F fused .

[0142] (c) The fused feature F fused After 1×1 convolution, it is refined and the input F of the multi-scale enhanced spatial attention mechanism is obtained by comparing channel attention. s .

[0143] (d) F s Perform compression with scales of 3, 5, and 7 and a local variance operation with a scale of 7, and sum them to get

[0144] (e) Upsample the low-resolution attention map obtained in (d) to a high-resolution attention map and perform a Sigmoid activation operation to obtain the weights

[0145] (f) The weights are combined with the input F of the multi-scale enhanced spatial attention mechanism s Multiply and then add to get the output F out .

[0146] In another embodiment, a lightweight image super-resolution reconstruction system is provided, which is used to implement the lightweight image super-resolution reconstruction method, including an input unit, a lightweight super-resolution neural network unit, and an output unit, wherein:

[0147] The input unit is used to input a low-resolution image to be reconstructed.

[0148] The lightweight super-resolution neural network unit is used to store the trained lightweight super-resolution neural network. The low-resolution image is reconstructed by the trained lightweight super-resolution neural network to obtain a reconstructed super-resolution image.

[0149] The output unit is used to output the reconstructed super-resolution image.

[0150] In another embodiment, a computer system is provided, comprising a memory and a processor, wherein the memory is used to store a computer program / instruction, and the processor is used to execute the computer program / instruction to implement the lightweight image super-resolution reconstruction method.

[0151] In the embodiment, we propose a multi-scale partial convolution residual distillation super-resolution network (MPCRDN), and the multi-scale partial convolution residual distillation module (MPCRDB) is used as the basic block of MPCRDN. This module includes our multi-level information distillation mechanism based on partial convolution design and multi-scale enhanced spatial attention mechanism. The multi-level information distillation mechanism based on partial convolution design can effectively avoid channel redundancy and simplify the feature extraction process. Multi-scale enhanced spatial attention introduces multi-scale information and local variance information to obtain a more accurate spatial information distribution, and uses partial convolution to reduce module parameters. A good balance between network parameters and performance is achieved.

[0152] To demonstrate the effectiveness of our proposed module, we designed several sets of ablation experiments.

[0153] As shown in Table 1, the information distillation mechanism with a weight of 1×2×4 has an average PSNR reduction of 0.05dB compared to the information distillation mechanism with a weight of 1×1×1, while the number of parameters is reduced by nearly half. This is mainly because redundant feature channels appear during the convolution process. These redundant channels do not have a significant effect on performance improvement, but they will bring a considerable amount of parameters. Compared with the information distillation mechanisms with weights of 2×2×2 and 4×4×4, although the network models of these two weights have fewer parameters, the PSNR and SSIM results are too low. This is mainly because the feature extraction capability is not strong enough and the feature refinement is not enough. Therefore, we use a weight ratio of 1×2×4 as the weight of the information distillation module in the final MPCRDN.

[0154] In addition, we designed an ablation experiment on the attention mechanism. We used MPCRDN as the basic network and used ESA, which is commonly used in efficient super-resolution, to replace the proposed MESA. The results are shown in Table 2. It can be seen that the proposed MESA is better than ESA. Compared with ESA, MESA can better utilize the spatial information in the image and achieve better performance with smaller parameters. Specifically, the combination of MESA and CCA increases the number of parameters by 2K compared to the combination of ESA and CCA, but increases the average PSNR of Set5 by 0.07dB.

[0155] Table 1 Experimental data of information distillation mechanism with different weights

[0156]

[0157] Table 2 Ablation experiment based on attention mechanism

[0158]

[0159] The neural network, gradient descent, and Adam optimizer in this embodiment are all professional terms in this field, which are prior art and are not the main improvement points of the present invention, so they will not be described in detail.

[0160] This example achieves super-resolution reconstruction of low-resolution images by designing a lightweight image super-resolution method based on information distillation and multi-scale attention, so that the generated image can restore the high-frequency details and basic texture shape of the image while improving the resolution and increasing the image size. At the same time, the network model has a small number of parameters and is suitable for industrial scenarios. This example uses an information distillation mechanism based on partial convolution and multi-scale enhanced spatial attention to avoid channel redundancy, effectively reduce the amount of network calculations and parameters, and ensure the efficiency of the model.

[0161] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A lightweight image super-resolution reconstruction method, characterized in that: The following steps are involved: Step 1, obtain lightweight images and preprocess them as training sets; Step 2: construct a lightweight super-resolution neural network based on partial convolution information distillation mechanism module and multi-scale attention mechanism module; The lightweight super-resolution neural network includes a shallow layer extraction module, a deep layer feature extraction module and a reconstruction module, and the shallow layer extraction module includes a first convolutional layer; The deep feature extraction module includes five multi-scale partial convolution residual distillation modules, a connection fusion layer, a second convolution layer, a pixel attention module, and a second partial convolution layer. The five multi-scale partial convolution residual distillation modules are connected in sequence, and the output of the first convolution layer enters the first multi-scale partial convolution residual distillation module among the five multi-scale partial convolution residual distillation modules. At the same time, the outputs of the five multi-scale partial convolution residual distillation modules respectively enter the connection fusion layer. The connection fusion layer, the second convolution layer, the pixel attention module, and the second partial convolution layer are connected in sequence; The reconstruction module includes a first sub-pixel convolution module, a second sub-pixel convolution module and a third convolution layer, the first sub-pixel convolution module includes a first upsampling operation and a first addition layer; the second sub-pixel convolution module includes a second addition layer and a second upsampling operation, the output of the first convolution layer enters the third convolution layer and the second addition layer respectively, the third convolution layer, the first upsampling operation, and the first addition layer are connected in sequence, and the second addition layer, the second upsampling operation, and the first addition layer are connected in sequence; Step 3, training the constructed lightweight super-resolution neural network through the training set to obtain a trained lightweight super-resolution neural network; Step 4: Reconstruct the low-resolution image through the trained lightweight super-resolution neural network to obtain a reconstructed super-resolution image.

2. The lightweight image super-resolution reconstruction method according to claim 1, characterized in that: The multi-scale partial convolution residual distillation module includes 4 fourth partial residual modules, 4 fourth convolution layers, a fourth connection fusion layer, a channel attention module, a multi-scale enhanced spatial attention module, and a fourth addition layer 1. The 4 fourth partial residual modules are respectively the fourth partial residual module 1, the fourth partial residual module 2, the fourth partial residual module 3, and the fourth partial residual module 4. The 4 fourth convolution layers are respectively the fourth convolution layer 1, the fourth convolution layer 2, the fourth convolution layer 3, and the fourth convolution layer 4. The fourth partial residual module 1, the fourth partial residual module 2, the fourth partial residual module 3, the fourth partial residual module 4, and the fourth connection The fusion layer, the fourth convolutional layer four, the channel attention module, the multi-scale enhanced spatial attention module, and the fourth addition layer one are connected in sequence, the inputs of the multi-scale partial convolutional residual distillation module enter the fourth partial residual module one, the fourth convolutional layer one and the fourth addition layer one respectively, and the output of the fourth convolutional layer one is connected to the fourth connection fusion layer; the input of the fourth convolutional layer two is connected to the output of the fourth partial residual module one, and the output of the fourth convolutional layer two is connected to the fourth connection fusion layer; the input of the fourth convolutional layer three is connected to the output of the fourth partial residual module two, and the output of the fourth convolutional layer three is connected to the fourth connection fusion layer.

3. The lightweight image super-resolution reconstruction method according to claim 2, characterized in that: The multi-scale enhanced spatial attention module includes the fifth convolution layer one, the fifth compressed convolution layer one, the fifth maximum pooling layer one, the fifth compressed convolution layer two, the fifth maximum pooling layer two, the fifth compressed convolution layer three, the fifth maximum pooling layer three, the fifth compressed convolution layer four, the fifth local variance module, the fifth addition layer one, the fifth partial convolution group one, the fifth partial convolution group two, the fifth partial convolution group three, the fifth upsampling module, the fifth addition layer two, the fifth convolution layer two, the fifth activation function module, the element multiplication operation module, and the fifth addition layer three. The input of the multi-scale enhanced spatial attention module enters the fifth convolution layer one, the element multiplication operation module, and the fifth addition layer three respectively, and the output of the fifth convolution layer one enters the fifth compressed convolution layer one, the fifth compressed convolution layer two, the fifth compressed convolution layer three, The fifth compressed convolution layer four, the fifth compressed convolution layer one, the fifth maximum pooling layer one, and the fifth partial convolution group one are connected in sequence; the fifth compressed convolution layer two, the fifth maximum pooling layer two, and the fifth partial convolution group two are connected in sequence; the fifth compressed convolution layer three, the fifth maximum pooling layer three, the fifth addition layer one, and the fifth partial convolution group three are connected in sequence; the fifth compressed convolution layer four, the fifth local variance module, and the fifth addition layer one are connected in sequence; the outputs of the fifth partial convolution group one, the fifth partial convolution group two, and the fifth partial convolution group three respectively enter the fifth upsampling module, the fifth upsampling module, the fifth addition layer two, the fifth convolution layer two, the fifth activation function module, the element multiplication operation module, and the fifth addition layer three are connected in sequence; the output of the fifth convolution layer one enters the fifth addition layer two.

4. The lightweight image super-resolution reconstruction method according to claim 3, characterized in that: The fourth partial residual module includes a fourth partial convolutional layer, a fourth addition layer 2, and a Gelu activation layer connected in sequence, and the input of the fourth partial residual module enters the fourth partial convolutional layer and the fourth addition layer 2 respectively; the fourth partial convolutional layer includes a fourth input layer and a fourth output layer, and the input of the fourth input layer enters the fourth output layer through a convolution operation, and the input of the fourth input layer enters the fourth output layer through direct mapping.

5. The lightweight image super-resolution reconstruction method according to claim 4, characterized in that: The information distillation method of the multi-scale partial convolution residual distillation module is: Step S101, the input F of the multi-scale partial convolution residual distillation module in It is divided into two paths, one path is input into the fourth residual module 1, and the other path is input into the fourth convolutional layer 1 for feature refinement, which is recorded as: F c1 =PRB1(F in ), F d1 =C1(F in ) Where C1 represents the convolution operation of the fourth convolutional layer, PRB1 represents the residual operation of the fourth residual module, and F d1 represents the features extracted by the fourth convolutional layer, F c1 Indicates the feature that the fourth part of the residual module is sent to the next step; Step S102: the obtained F c1 Further refinement, the process is expressed as: F c2 =PRB2(PRB1(F in )) F d2 =C2(PRB1(F in )) Where C2 represents the second convolution operation of the fourth convolutional layer, PRB2 represents the second residual operation of the fourth residual module, and F d2 represents the features extracted by the fourth convolutional layer 2, F c2 Indicates the features of the fourth part of the residual module 2 being sent to the next step; Step S103: c2 Continuing with the refinement, the process is expressed as: F c3 =PRB3(PRB2(PRB1(F in ))), F d3 =C3(PRB2(PRB1(F in ))) Where C3 represents the third convolution operation of the fourth convolutional layer, PRB3 represents the third residual operation of the fourth residual module, and F d3 represents the features extracted by the fourth convolutional layer, F c3 It indicates that the fourth part of the residual module 3 is sent to the next step of the operation; Step S104: obtain the F c3 The fourth part is sent to the residual module 4 for the final refinement. The process is expressed as: F c4 =PRB4(PRB3(PRB2(PRB1(F in )))) PRB4 represents the fourth residual module, the four-part residual operation, F c4 Indicates the feature that the fourth part of the residual module 4 is sent to the next step of operation; Step S105, the features extracted in the above steps are fused, and the process is expressed as: F fused =Concat(C1(F in ), C2(PRB1(F in )), C3(PRB2(PRB1(F in ))),PRB4(PRB3(PRB2(PRB1(F in )))))In the formula, F fused Represents the fusion feature, and Concat represents the fusion operation of the fourth connection fusion layer.

6. The lightweight image super-resolution reconstruction method according to claim 5, characterized in that: The multi-scale attention method of the multi-scale partial convolution residual distillation module is: Step S201: First, the input F of the multi-scale attention mechanism is s Through a fifth convolutional layer to reduce the number of channels to reduce the amount of network parameters, the process can be expressed as: F in,1 =C 1×1 (F S ) Among them, C 1×1 represents the convolution operation of the fifth convolutional layer, F S represents the input of the multi-scale attention mechanism, F in,1 Represents the convolution operation result of the fifth convolutional layer. Step S202: the obtained F in,1 Compression is performed on three scales and local variance is introduced. The process is expressed as: F s 3 in =B3(F in,1 )+B5(F in,1 )+B7(F in,1 )+B L-vat (C 1×1 (F S )) Among them, F s 3 in represents compressed local features, B3 represents the maximum pooling operation after the fifth compressed convolution layer 1, B5 represents the maximum pooling operation after the fifth compressed convolution layer 2, B7 represents the maximum pooling operation after the fifth compressed convolution layer 3, B L-vat It means that after the operation of the fifth compressed convolution layer and the fourth compressed convolution operation, it passes through the operation of the fifth local variance module; Step S203, the obtained The features are further refined by sending them to the convolution group constructed by partial convolution. The process is expressed as: in, Indicates the result obtained after some convolution groups, B g Represents a partial convolution group operation; Step S204, the obtained Up-sampling is performed to obtain a high-resolution feature map, and the high-resolution feature map is added to the result obtained in step S201. The process is expressed as: in, represents the obtained high-resolution feature map, B up Represents an upsampling operation; Step S205: The high-resolution feature map obtained in step S204 is After the fifth convolutional layer, it is adjusted to F S The number of channels is the same, and then processed by the Sigmoid function and added to F S Add; the process is expressed as: Among them, F out Indicates the output, Represents element-wise multiplication, H conv1-1 represents the second convolution operation of the fifth convolutional layer, and Sigmoid represents the Sigmoid function.

7. The lightweight image super-resolution reconstruction method according to claim 6, characterized in that: The shallow extraction process of the shallow extraction module is as follows: F0=B SF (IN LR ) Among them, F0 represents the result of shallow feature extraction, B SF represents the shallow feature extraction process, I LR represents the input image; The feature extraction process of the deep feature extraction module is as follows: F1=B DF (B SF (I LR )) Among them, F1 represents the deep feature extraction result, B DF represents the deep feature extraction process, B SF represents the shallow feature extraction process, I LR represents the input image; The reconstruction process of the reconstruction module is: I SR =B SR (B DF (B SF (I LR )))+B up (I LR ) Among them, I SR Indicates the final result, B SR represents the second sub-pixel convolution process, B DF represents the deep feature extraction process, B SF represents the shallow feature extraction process, I LR represents the input image, B up Represents the first sub-pixel convolution process.

8. The lightweight image super-resolution reconstruction method according to claim 7, characterized in that: The channel attention module combines the fusion feature F by comparing the channel attention fused Processing is performed to obtain the input F of the multi-scale attention mechanism s .

9. A lightweight image super-resolution reconstruction system, characterized by: The method for implementing the lightweight image super-resolution reconstruction method according to any one of claims 1 to 8 comprises an input unit, a lightweight super-resolution neural network unit, and an output unit, wherein: The input unit is used to input the low-resolution image to be reconstructed; The lightweight super-resolution neural network unit is used to store the trained lightweight super-resolution neural network; the low-resolution image is reconstructed by the trained lightweight super-resolution neural network to obtain a reconstructed super-resolution image; The output unit is used to output the reconstructed super-resolution image.

10. A computer system, characterized in that: It comprises a memory and a processor, wherein the memory is used to store computer programs / instructions; and the processor is used to execute the computer programs / instructions to implement the lightweight image super-resolution reconstruction method according to any one of claims 1 to 8.

Images