Image reconstruction method based on cross iteration back projection network

Abstract

The invention relates to an image reconstruction method based on a cross iteration back projection network. The method comprises the following steps: constructing a cross iteration back projection joint attention network model; wherein the cross iteration back projection joint attention network model comprises a shallow layer extraction module, a plurality of deep layer extraction modules, an upward sampling module and a reconstruction module; training the cross iteration back projection joint attention network model based on the loss function to obtain a trained cross iteration back projection joint attention network model; and inputting a target low-resolution image into the trained cross iteration back projection joint attention network model to obtain a high-resolution image. According to the method, the target low-resolution image is input into the trained cross iteration back projection joint attention network model, so that reconstruction of the target low-resolution image is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image reconstruction method based on a cross-iterative back-projection network. Background Art

[0002] There are generally two ways to obtain high-resolution images: hardware improvement and software processing. Hardware improvement generally involves improving the optical hardware in the acquisition system to improve the quality of the image acquisition process, but this approach is limited by manufacturing processes, manufacturing costs, and other constraints. It is not only difficult to make significant improvements, but also difficult to popularize in actual production and life. Software processing generally obtains high-resolution images through algorithm processing based on the original image. Compared with the hardware improvement method, the software processing method is not only easy to upgrade, but also low-cost. There is no need to replace additional imaging equipment. It only needs to process the image through the design of the algorithm to improve the image quality.

[0003] In the related art, the software processing methods for obtaining high-resolution images mainly include interpolation-based image reconstruction methods, degradation model-based image reconstruction methods, learning-based image reconstruction methods, etc. Although the interpolation-based image reconstruction method has the advantages of small computational complexity and no additional data requirements, it also has disadvantages such as poor reconstruction effect when facing rich texture image information and easy to produce artifacts. The image reconstruction method based on the degradation model mainly models the degradation process and reconstructs the high-resolution image in reverse. This method is to reconstruct the original image with low-resolution images after blurring, noise and downsampling, combined with prior information. Although the reconstruction effect has been changed to a certain extent, the high-resolution image obtained is still not clear enough. The learning-based image reconstruction method mainly learns the image feature information of a given image data set, establishes a sequence prior relationship between low-resolution images and high-resolution images, and realizes image reconstruction through algorithms. It mainly adopts a deep learning algorithm. Although image reconstruction can be realized through a super-resolution convolutional neural network, there are still problems such as insufficient image processing details caused by the network, large network computational complexity, and low network computing efficiency. Although the accelerated super-resolution convolutional neural network has improved this, there are still problems such as insufficient depth resulting in incomplete reconstruction effect. Although the residual channel attention network can continue to increase the network depth, integrate the attention mechanism, increase the network's feature processing capabilities, and improve network performance, a network that is too deep will lead to a sharp increase in the number of parameters and computational costs.

[0004] In the above-mentioned related technologies, although the software processing method for obtaining high-resolution images can realize image reconstruction, there are still problems such as unclear image reconstruction, incomplete image reconstruction effect, large network calculation amount, and low network operation efficiency.

[0005] Therefore, it is necessary to provide a new technical solution to improve one or more problems existing in the above solutions.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] The purpose of the present application is to provide an image reconstruction method based on a cross-iterative back-projection network, thereby overcoming one or more problems caused by the limitations and defects of related technologies to a certain extent.

[0008] According to an embodiment of the present application, an image reconstruction method based on a cross-iterative back-projection network is provided, the method comprising: Constructing a cross-iterative back-projection joint attention network model; wherein the cross-iterative back-projection joint attention network model comprises: a shallow extraction module, multiple deep extraction modules, an upsampling module and a reconstruction module; Training the cross-iterative back-projection joint attention network model based on the loss function to obtain the trained cross-iterative back-projection joint attention network model; Inputting the target low-resolution image into the trained cross-iterative back-projection joint attention network model to obtain a high-resolution image; wherein the step of inputting the target low-resolution image into the trained cross-iterative back-projection joint attention network model to obtain a high-resolution image includes: Inputting the target low-resolution image into the shallow layer extraction module to extract shallow features, thereby obtaining shallow features of the target low-resolution image; Performing deep feature extraction on the shallow features through a plurality of the deep extraction modules to obtain deep features of the target low-resolution image; Performing up-sampling processing on the deep features through the up-sampling module to obtain a sampled feature image; The sampling feature image is reconstructed by the reconstruction module to obtain a high-resolution image.

[0009] In an embodiment of the present application, the step of training the cross-iterative back-projection joint attention network model based on the loss function to obtain the trained cross-iterative back-projection joint attention network model includes: Input n low-resolution images and reconstruct n super-resolution images; Calculate the loss function of n super-resolution images and the real high-resolution image.

[0010] In an embodiment of the present application, the loss function is: (1) In the formula, n represents the number of training samples, represents the network parameters, i represents the sequence number of the training sample during the training process, represents the loss function, X i represents a real high-resolution image, Represents a super-resolution image.

[0011] In an embodiment of the present application, the step of training the cross-iterative back-projection joint attention network model based on the loss function to obtain the trained cross-iterative back-projection joint attention network model further includes: The Adam optimization algorithm is used to optimize the network parameters of the cross-iterative back-projection joint attention network model to obtain the trained cross-iterative back-projection joint attention network model.

[0012] In an embodiment of the present application, the step of optimizing the network parameters of the cross-iterative back-projection joint attention network model using the Adam optimization algorithm to obtain the trained cross-iterative back-projection joint attention network model includes: (2) (3) (4) (5) (6) (7) (8) In the formula, express time, express The loss function at the moment Network parameters The gradient of represents the gradient, express The first moment estimate of the gradient at that moment, represents the exponential decay rate of the first-order moment estimate, express The exponential decay rate of the first-order moment estimate at time , express The first moment estimate of the gradient at that moment, express The second moment estimate of the gradient at that moment, express The second moment estimate of the gradient at that moment, represents the exponential decay rate of the second-order moment estimate, express The exponential decay rate of the second-order moment estimate at time, express time The deviation correction of express time The deviation correction of represents the step length, represents a constant, express The network parameters at the time, express The updated value of the network parameters at time , express Network parameters at the time.

[0013] In an embodiment of the present application, the deep extraction module includes a cross-upper projection module, an improved encoding and decoding attention module and a cross-down projection module connected in series in sequence; wherein the cross-upper projection module and the cross-down projection module both include a convolution extraction block and a deconvolution extraction block.

[0014] In an embodiment of the present application, the step of performing deep feature extraction on the shallow features through a plurality of deep extraction modules to obtain the deep features of the target low-resolution image includes: Upsampling the shallow features through the deconvolution extraction block of the cross-up-projection module to generate a first image feature; Processing the first image feature by the convolution extraction block of the cross-upper projection module to generate a first compression feature; Acquire a first error feedback feature according to the shallow feature and the first compression feature; Processing the first error feedback feature by the convolution extraction block of the cross-upper projection module to obtain a second compression feature; Processing the second compressed feature by the deconvolution extraction block of the cross-upper projection module to obtain a second image feature; The second image feature is superimposed on the first image feature to obtain a fine restoration feature, and the fine restoration feature is input into the arrangement convolution of the cross-upper projection module to perform feature arrangement to obtain an arrangement feature.

[0015] In an embodiment of the present application, the step of performing deep feature extraction on the shallow features through a plurality of deep extraction modules to obtain the deep features of the target low-resolution image includes: Processing the sorting features through the convolution extraction block of the cross-under-projection module to generate a third compressed feature; Processing the third compression feature by the deconvolution extraction block of the cross-under-projection module to generate a third image feature; Acquire a second error feedback feature according to the sorting feature and the third image feature; Processing the second error feedback feature block by convolution of the cross-under-projection module to obtain a fourth compression feature; Processing the fourth compression feature by the convolution extraction block of the cross-under-projection module to obtain a fifth compression feature; Superimposing the fifth compression feature with the third compression feature to obtain a fine compression feature; The fine compressed features are input into the arranging convolution of the cross-down projection module for feature arrangement to obtain the deep features.

[0016] In an embodiment of the present application, the improved codec attention module includes average pooling, maximum pooling, 1x1 convolution, channel shuffling, 1x1 deconvolution and activation function connected in series.

[0017] In an embodiment of the present application, the shallow extraction module includes a 3x3 convolution, the up-sampling module includes a sub-pixel convolution layer, the sub-pixel convolution layer includes a 3x3 convolution, the input channel of the sub-pixel convolution layer is 64, and the output channel of the sub-pixel convolution layer is 256.

[0018] The technical solution provided by the embodiments of the present application may have the following beneficial effects: In one embodiment of the present application, a cross-iterative back-projection joint attention network model is designed through the above method, and is trained to obtain a trained cross-iterative back-projection joint attention network model. By inputting the target low-resolution image into the trained cross-iterative back-projection joint attention network model, reconstruction of the target low-resolution image is achieved. And because the designed cross-iterative back-projection joint attention network model is trained based on the loss function, the trained cross-iterative back-projection joint attention network model is more accurate, so that when the trained cross-iterative back-projection joint attention network model is used to reconstruct the target low-resolution image, the image reconstruction effect is good and the reconstruction efficiency is high.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart schematically showing a method for image reconstruction based on a cross-iterative back-projection network in an exemplary embodiment of the present application; Figure 2 A schematic diagram schematically illustrates a framework of an image reconstruction method based on a cross-iterative back-projection network in an exemplary embodiment of the present application; Figure 3 Schematically showing the structure of the cross-iterative back-projection joint attention network model in the exemplary embodiment of the present application; Figure 4 A schematic diagram schematically illustrates a cross-upper projection module in an exemplary embodiment of the present application; Figure 5 A schematic diagram schematically illustrates a cross-down projection module in an exemplary embodiment of the present application; Figure 6 A schematic diagram schematically illustrates an improved encoding and decoding attention module in an exemplary embodiment of the present application; Figure 7 A schematic diagram schematically illustrates an original image in an exemplary embodiment of the present application; Figure 8 A schematic diagram schematically shows the reconstruction of the original image using SRCNN in an exemplary embodiment of the present application; Fig. 9 A schematic diagram schematically shows the reconstruction of the original image using FSRCNN in an exemplary embodiment of the present application; Fig.10 A schematic diagram schematically shows the reconstruction of the original image using VDSR in an exemplary embodiment of the present application; Fig.11 A schematic diagram schematically shows the reconstruction of the original image using DRCN in an exemplary embodiment of the present application; Fig.12 A schematic diagram schematically shows the reconstruction of the original image using LapSRN in an exemplary embodiment of the present application; Fig.13 The schematic diagram schematically shows the reconstruction of the original image using the method of the present application in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0022] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0023] This example embodiment provides an image reconstruction method based on a cross-iterative back-projection network. Figure 1 As shown, the method includes: step S100 to step S300.

[0024] Among them, step S100: construct a cross-iterative back-projection joint attention network model; wherein, the cross-iterative back-projection joint attention network model includes: a shallow extraction module, multiple deep extraction modules, an upsampling module and a reconstruction module.

[0025] Step S200: Train the cross-iterative back-projection joint attention network model based on the loss function to obtain a trained cross-iterative back-projection joint attention network model.

[0026] Step S300: Input the target low-resolution image into the trained cross-iterative back-projection joint attention network model to obtain a high-resolution image.

[0027] Wherein, step S300 includes: Step S310: inputting the target low-resolution image into a shallow layer extraction module to extract shallow features, thereby obtaining shallow features of the target low-resolution image; Step S320: extracting deep features from shallow features through multiple deep extraction modules to obtain deep features of the target low-resolution image; Step S330: performing up-sampling processing on the deep features through an up-sampling module to obtain a sampled feature image; Step S340: reconstruct the sampled feature image through a reconstruction module to obtain a high-resolution image.

[0028] It should be noted that the purpose of constructing the cross-iterative back-projection joint attention network model in this application is to facilitate better image reconstruction in the future. After the cross-iterative back-projection joint attention network model is constructed, it needs to be trained so that the trained cross-iterative back-projection joint attention network model can be used for image reconstruction in the future. Because the initially constructed cross-iterative back-projection joint attention network model includes a shallow extraction module, multiple deep extraction modules, an upsampling module, and a reconstruction module. Therefore, Figure 2 and Figure 3As shown in FIG. 1 , the trained cross-iteration back-projection joint attention network model also includes a shallow extraction module, multiple deep extraction modules, an up-sampling module, and a reconstruction module. By inputting the target low-resolution image into the trained cross-iteration back-projection joint attention network model, the target low-resolution image is reconstructed.

[0029] In one embodiment of the present application, a cross-iterative back-projection joint attention network model is designed through the above method, and is trained to obtain a trained cross-iterative back-projection joint attention network model. By inputting the target low-resolution image into the trained cross-iterative back-projection joint attention network model, reconstruction of the target low-resolution image is achieved. And because the designed cross-iterative back-projection joint attention network model is trained based on the loss function, the trained cross-iterative back-projection joint attention network model is more accurate, so that when the trained cross-iterative back-projection joint attention network model is used to reconstruct the target low-resolution image, the image reconstruction effect is good and the reconstruction efficiency is high.

[0030] Next, we will refer to Figures 4 to 6 Each step of the above method in this example implementation is described in more detail.

[0031] In step S100, the purpose of constructing a cross-iterative back-projection joint attention network model in the present application is to facilitate subsequent image reconstruction.

[0032] In step S200, the present application selects the public DIV2K dataset as the training dataset, of which 800 images are used for the training set, 100 images are used for the validation set, and 100 images are used for the test set. The high-resolution images of the DIV2K dataset are degraded, and the bicubic interpolation algorithm is used for downsampling and blurring to obtain 1 / N downsampled images as low-resolution images, where N represents a specified scaling factor. The training set is used to train the constructed cross-iteration back projection joint attention network model, the test set is used to test the trained cross-iteration back projection joint attention network model to test the performance of the trained cross-iteration back projection joint attention network model, and the validation set is used to test the tested cross-iteration back projection joint attention network model to verify the true performance of the tested cross-iteration back projection joint attention network model. The DIV2K dataset is a popular single image super-resolution dataset.

[0033] In addition, the publicly available Set5, Set14, B100, and Urban100 can also be used as test sets to test the performance of the cross-iterative back-projection joint attention network model constructed in this application. In order to expand the samples of the training data set, the DIV2K data set can be rotated by 90°, 180°, and 270° and scaled by 0.9, 0.8, 0.7, and 0.6. Among them, Set5, Set14, B100, and Urban100 are all commonly used super-resolution data sets.

[0034] The low-resolution image obtained by the above-mentioned degradation of the high-resolution image is cropped into an image block of size H×W as the input of the cross-iterative back-projection joint attention network model, and the image of the training dataset is cropped into an image block of size kH×kW, which is used as the label corresponding to the input low-resolution image; the obtained input low-resolution image and its corresponding high-resolution image are used as a sample pair as the training dataset file. It should be noted that H represents the width of the image block, W represents the height of the image block, and k represents the scale factor of the image block.

[0035] In the process of training the constructed cross-iterative back-projection joint attention network model through the training data set, the cross-iterative back-projection joint attention network model is also trained based on the loss function to obtain a trained cross-iterative back-projection joint attention network model.

[0036] In one embodiment, the step of training the cross-iterative back-projection joint attention network model based on the loss function to obtain the trained cross-iterative back-projection joint attention network model includes: Input n low-resolution images and reconstruct n super-resolution images; Calculate the loss function of n super-resolution images and the true high-resolution image.

[0037] In one embodiment, the loss function is: (1) In the formula, n represents the number of training samples, represents the network parameters, i represents the sequence number of the training sample during the training process, represents the loss function, X i represents a real high-resolution image, Represents a super-resolution image.

[0038] In one embodiment, the step of training the cross-iterative back-projection joint attention network model based on the loss function to obtain the trained cross-iterative back-projection joint attention network model further includes: The Adam optimization algorithm is used to optimize the network parameters of the cross-iterative back-projection joint attention network model to obtain a trained cross-iterative back-projection joint attention network model.

[0039] It should be noted that the Adam optimization algorithm is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It can iteratively update the weights of the neural network based on the training data. The stochastic gradient descent algorithm maintains a single learning rate to update all weights, and the learning rate does not change during the network training process. The Adam optimization algorithm designs independent adaptive learning rates for different parameters by calculating the first-order moment estimate and the second-order moment estimate of the gradient. The Adam optimization algorithm has high computational efficiency and low memory requirements, and the diagonal rescaling of the Adam optimization algorithm gradient is invariant. Among them, Adam is an adaptive moment estimation, and its full English name is Adaptive moment estimation.

[0040] In one embodiment, the step of optimizing the network parameters of the cross-iterative back-projection joint attention network model using the Adam optimization algorithm to obtain a trained cross-iterative back-projection joint attention network model includes: (2) (3) (4) (5) (6) (7) (8) In the formula, express time, express The loss function at the moment Network parameters The gradient of represents the gradient, express The first moment estimate of the gradient at that moment, represents the exponential decay rate of the first-order moment estimate, express The exponential decay rate of the first-order moment estimate at time , express The first moment estimate of the gradient at that moment, express The second moment estimate of the gradient at that moment, express The second moment estimate of the gradient at that moment, represents the exponential decay rate of the second-order moment estimate, express The exponential decay rate of the second-order moment estimate at time, express time The deviation correction of express time The deviation correction of represents the step length, represents a constant, express The network parameters at the time, express The updated value of the network parameters at time , express Network parameters at the time.

[0041] It should be noted that the Adam optimization algorithm first initializes the parameter vector, the first-order moment vector, and the second-order moment vector, and then iteratively updates each part to make the network parameters Convergence. That is, Add 1, update the first-order moment estimate and the second-order moment estimate of the deviation, then calculate the deviation correction of the first-order moment estimate and the deviation correction of the second-order moment estimate, and then update the deviation correction at that time. Network parameters The required gradient is finally calculated using the above Network parameters at this moment Update value of , update the network parameters of the cross-iterative back-projection joint attention network model . Network parameters of the cross-iterative back-projection joint attention network model The updating process is the process of optimizing it. The network parameters of the cross-iterative back-projection joint attention network model are The optimization process is actually the process of training the cross-iterative back-projection joint attention network model. When the optimized network parameters are obtained When , the trained cross-iterative back-projection joint attention network model is obtained.

[0042] In one embodiment, the deep extraction module includes a cross-upper projection module, an improved codec attention module and a cross-down projection module connected in series; wherein the cross-upper projection module and the cross-down projection module both include a convolution extraction block and a deconvolution extraction block.

[0043] In one embodiment, the step of extracting deep features from shallow features by using multiple deep extraction modules to obtain deep features of the target low-resolution image includes: Upsampling the shallow features through the deconvolution extraction block of the cross-up-projection module to generate a first image feature; Processing the first image feature by a convolution extraction block of a cross-up projection module to generate a first compression feature; Acquire a first error feedback feature according to the shallow feature and the first compression feature; Processing the first error feedback feature by a convolution extraction block of a cross-up projection module to obtain a second compression feature; Processing the second compressed feature by a deconvolution extraction block of a cross-upper projection module to obtain a second image feature; The second image feature is superimposed on the first image feature to obtain a fine restoration feature, and the fine restoration feature is input into the arrangement convolution of the cross projection module for feature arrangement to obtain an arrangement feature.

[0044] It should be noted that in the process of extracting deep features from shallow features through multiple deep extraction modules to obtain deep features of the target low-resolution image, the cross-upper projection module is first used to perform primary extraction and secondary extraction. Figure 4 As shown, the cross-up projection module of the present application includes a convolution extraction block and a deconvolution extraction block. The deconvolution extraction block of the cross-up projection module is used for upsampling twice to generate a first image feature and a second image feature. The convolution extraction block of the cross-up projection module is used for downsampling twice to generate a first compression feature and a second compression feature.

[0045] For the shallow features input to the cross-upper projection module, the initial extraction is first performed on them. The process of performing the initial extraction through the cross-upper projection module includes: The shallow features are upsampled through the deconvolution extraction block one of the cross-up projection module to obtain the first image feature; the first image feature is passed into the convolution extraction block one of the cross-up projection module to obtain the first compression feature to extract information from different receptive fields, and then the shallow feature is subtracted from the first compression feature to obtain the first error feedback feature, and the error information is secondary extracted to obtain a better reconstruction effect.

[0046] For the secondary extraction process of the cross-upper projection module, the convolution extraction block 2 of the upper cross-projection module is first performed to extract the first error feedback feature to obtain the second compression feature. Then, the second image feature is obtained through the deconvolution extraction block 2 of the upper cross-projection module, and the second image feature is superimposed with the first image feature to obtain a fine restoration feature. Finally, the fine restoration feature is sent to the sorting convolution of the upper cross-projection module for feature sorting. In particular, the first image feature and the first compression feature obtained by the deconvolution extraction block 1 and the convolution extraction block 1 of the cross-upper projection module are passed to the convolution extraction block 2 of the cross-upper projection module and the dimension corresponding to the sorting convolution for feature compensation, with the aim of obtaining more high-frequency components.

[0047] Specifically, the convolution extraction block uses a standard convolution extraction with a convolution kernel size of 1x1, and an end-to-end activation function to increase the nonlinearity of its extracted information. The deconvolution extraction block uses a deconvolution extraction with a convolution kernel of 1x1, and an end-to-end activation function to increase the nonlinearity of its extracted information. A 3x3 hole convolution is added between the convolution extraction block and the deconvolution extraction block to expand the receptive field of the extracted information. The extraction dimension is also adjusted by compression and expansion. The specific extraction process is as follows: First, the deconvolution extraction block with 64-dimensional feature information input obtains 64-dimensional feature information at the output. , the feature information is passed into the aforementioned hole convolution, and the 64-dimensional feature information is output. Then the information is passed into the convolution extraction block, and the 64-dimensional feature information is output. The feature information is passed into the second hole convolution. The difference is that the hole convolution compresses the 64-dimensional feature into 32 dimensions, and expands the 32-dimensional feature into 64 dimensions through the deconvolution extraction block that is input later. The entire upper projection extraction is completed through the terminal hole convolution. In this extraction process, residual connections and jump connections are introduced to maintain the extracted features and prevent gradient disappearance or gradient explosion, so as to build a deeper network.

[0048] In one embodiment, the step of extracting deep features from shallow features by using multiple deep extraction modules to obtain deep features of the target low-resolution image includes: The sorted features are processed by the convolution extraction block of the cross-under-projection module to generate a third compressed feature; Processing the third compressed feature by a deconvolution extraction block of a cross-under-projection module to generate a third image feature; Acquire a second error feedback feature according to the sorting feature and the third image feature; Processing the second error feedback feature by a convolution extraction block of a cross-under-projection module to obtain a fourth compression feature; Processing the fourth compressed feature by a convolution extraction block of a cross-under-projection module to obtain a fifth compressed feature; Superimposing the fifth compression feature with the third compression feature to obtain a refined compression feature; The refined compressed features are input into the tidying convolution of the cross-down projection module for feature tidying to obtain deep features.

[0049] It should be noted that the reference Figure 5 As shown, the cross-down projection module of the present application also includes a convolution extraction block and a deconvolution extraction block. The deconvolution extraction block of the cross-down projection module is also used for upsampling, and the upsampling obtains high-resolution information. The convolution extraction block of the cross-down projection module is also used for downsampling, and the downsampling obtains compressed information. The specific extraction process is divided into primary extraction and secondary extraction. The convolution extraction block is used for downsampling three times in succession to generate the third compressed feature, the fourth compressed feature and the fifth compressed feature. The deconvolution extraction block of the cross-down projection module is used for upsampling to generate the third image feature.

[0050] For the sorting features input to the cross-under-projection module, firstly, the first extraction is performed, and the third compressed features are obtained through the convolution extraction block 1, and then the features are passed to the deconvolution extraction block 1 of the cross-under-projection module to obtain the third image features, so as to extract information from different receptive fields, and then the sorting features are subtracted from the third image features to obtain the second error feedback features, and the error information is extracted twice to obtain a better reconstruction effect; for the second extraction process, the second error feedback features are first extracted through the convolution extraction block 2 to obtain compressed feature information, and then the fifth compressed features are obtained through the convolution extraction block 3 of the cross-under-projection module, and the fifth compressed features are superimposed on the third compressed features to obtain fine compressed features, and finally the fine compressed features are sent to the sorting convolution of the cross-under-projection module for feature sorting, in order to restore the feature dimension; in particular, the third compressed features and the third image features obtained by the convolution extraction block 1 of the cross-under-projection module and the deconvolution extraction block 1 of the cross-under-projection module are passed to the convolution extraction block 2 of the cross-under-projection module and the dimensions corresponding to the sorting convolution of the cross-under-projection module, and feature compensation is performed, in order to obtain more high-frequency components.

[0051] It should be noted that Figure 4 and Figure 5 In the network, convolution (Convolution, Conv) and deconvolution (Deconvolution, DeConv) are a mathematical method of integral transformation, and the leaky linear rectified function (Leaky Rectified Linear Unit, LReLU) is a commonly used activation function.

[0052] In one embodiment, reference Figure 6 As shown in the figure, the improved encoder-decoder attention module includes average pooling, maximum pooling, 1x1 convolution, channel shuffling, 1x1 deconvolution and activation function connected in series.

[0053] It should be noted that by improving the encoding and decoding attention module to learn different weights, the parameters of the attention weights are allocated to the features obtained by the cross-upper projection module and the cross-down projection module; the specific weight allocation process is to learn the features of the cross-upper projection module and the cross-down projection module, learn the output feature information, and use the idea of ​​attention to obtain the mutual dependence between image features, realize adaptive adjustment of its output features, and better utilize the features obtained by the cross-upper projection module and the cross-down projection module. At the same time, compared with other attentions, convolution and deconvolution are introduced in the attention for encoding and decoding, and global feature extraction and local feature extraction are considered at the same time, so as to enhance the network's ability to learn features of interest and better obtain high-resolution images.

[0054] In one embodiment, the shallow extraction module includes 3x3 convolution, the up-sampling module includes a sub-pixel convolution layer, the sub-pixel convolution layer includes 3x3 convolution, the input channel of the sub-pixel convolution layer is 64, and the output channel of the sub-pixel convolution layer is 256.

[0055] It should be noted that the upsampling module of this application is composed of a sub-pixel convolution layer, which uses a convolution kernel size of 3x3, an input channel of 64, and an output channel of 256. The sub-pixel convolution layer can upsample the image with a size of The feature maps are rearranged into The feature map of the reconstruction output is a convolution with a convolution kernel size of 3x3, which converts the output 64 channels into 3 channels. Indicates the number of channels, Indicates the zoom ratio.

[0056] The present application realizes image reconstruction through the designed cross-back projection network model, which can not only extract differential information through encoding and decoding operations, but also extract information of different levels through cross-back projection, avoiding information loss during the transmission process. The cross-upper projection module and cross-lower projection module designed in the present application extract information of different spatial dimensions of the image, and the cross-jump connection retains and reuses this information. The present application improves the extraction function of the encoding and decoding attention module to better adapt to the cascading of multiple cross-upper projection modules and cross-lower projection modules, increase the extraction richness while selectively extracting information of interest, and more effectively focus on contextual features.

[0057] In order to verify the effectiveness and reliability of the method of the present application, it was compared with multiple existing reconstruction methods at different magnifications. At low magnification (2x, 3x, 4x), the present application was compared with 5 existing methods.

[0058] The present application is further described below through the following examples.

[0059] Get test set: This application uses the public Set5, Set14, BSD100, and Urban100 as four test sets to verify the image reconstruction effect of this application, and uses peak signal-to-noise ratio PSNR and structural similarity SSIM as evaluation indicators to evaluate the image reconstruction effect of this application. It can be seen from Table 1 that the image reconstruction effect of this application is significantly better than the five existing methods.

[0060] Table 1 PSNR / SSIM under 2x, 3x, and 4x magnification on four test sets Combination Figures 7 to 13 It can be seen that: Figure 7 For the original image, the method of the present application, the Super Resolution Convolutional Neural Networks (SRCNN), the Fast Super-Resolution Convolutional Neural Network (FSRCNN), the Very Deep Super Resolution (VDSR), the Deeply-Recurisive Convolutional Network (DRCN), and the Laplacian Pyramid Super-Resolution Network (LapSRN) are used to respectively Figure 7 After the reconstruction, by Figures 8 to 13 By comparison, it can be seen that the image reconstruction effect using the method of the present application is good.

[0061] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

Claims

1. An image reconstruction method based on a cross-iterative back-projection network, characterized in that: The method includes: Constructing a cross-iterative back-projection joint attention network model; wherein the cross-iterative back-projection joint attention network model comprises: a shallow extraction module, multiple deep extraction modules, an upsampling module and a reconstruction module; Training the cross-iterative back-projection joint attention network model based on the loss function to obtain the trained cross-iterative back-projection joint attention network model; Inputting the target low-resolution image into the trained cross-iteration back-projection joint attention network model to obtain a high-resolution image; wherein the step of inputting the target low-resolution image into the trained cross-iteration back-projection joint attention network model to obtain a high-resolution image includes: Inputting the target low-resolution image into the shallow layer extraction module to extract shallow features, thereby obtaining shallow features of the target low-resolution image; Performing deep feature extraction on the shallow features through a plurality of the deep extraction modules to obtain deep features of the target low-resolution image; Performing up-sampling processing on the deep features through the up-sampling module to obtain a sampled feature image; The sampling feature image is reconstructed by the reconstruction module to obtain a high-resolution image.

2. The image reconstruction method based on cross-iterative back-projection network according to claim 1, characterized in that: The step of training the cross-iterative back-projection joint attention network model based on the loss function to obtain the trained cross-iterative back-projection joint attention network model includes: Input n low-resolution images and reconstruct n super-resolution images; Calculate the loss function of n super-resolution images and the real high-resolution image.

3. The image reconstruction method based on cross-iterative back-projection network according to claim 2, characterized in that: The loss function is: （1） In the formula, n represents the number of training samples, represents the network parameters, i represents the sequence number of the training sample during the training process, represents the loss function, X i represents a real high-resolution image, Represents a super-resolution image.

4. The image reconstruction method based on cross-iterative back-projection network according to claim 3, characterized in that: The step of training the cross-iterative back-projection joint attention network model based on the loss function to obtain the trained cross-iterative back-projection joint attention network model also includes: The Adam optimization algorithm is used to optimize the network parameters of the cross-iterative back-projection joint attention network model to obtain the trained cross-iterative back-projection joint attention network model.

5. The image reconstruction method based on cross-iterative back-projection network according to claim 4, characterized in that: The step of optimizing the network parameters of the cross-iterative back-projection joint attention network model by using the Adam optimization algorithm to obtain the trained cross-iterative back-projection joint attention network model includes: （2） （3） （4） （5） （6） （7） （8） In the formula, express time, express The loss function at the moment Network parameters The gradient of represents the gradient, express The first moment estimate of the gradient at that moment, represents the exponential decay rate of the first-order moment estimate, express The exponential decay rate of the first-order moment estimate at time , express The first moment estimate of the gradient at that moment, express The second moment estimate of the gradient at that moment, express The second moment estimate of the gradient at that moment, represents the exponential decay rate of the second-order moment estimate, express The exponential decay rate of the second-order moment estimate at time, express time The deviation correction of express time The deviation correction of represents the step length, represents a constant, express The network parameters at the time, express The updated value of the network parameters at time , express Network parameters at the time.

6. The image reconstruction method based on cross-iterative back-projection network according to claim 1, characterized in that: The deep extraction module includes a cross-upper projection module, an improved codec attention module and a cross-down projection module connected in series in sequence; wherein the cross-upper projection module and the cross-down projection module both include a convolution extraction block and a deconvolution extraction block.

7. The image reconstruction method based on cross-iterative back-projection network according to claim 6, characterized in that: The step of extracting deep features from the shallow features by using a plurality of deep extraction modules to obtain deep features of the target low-resolution image includes: Upsampling the shallow features through the deconvolution extraction block of the cross-up-projection module to generate a first image feature; Processing the first image feature by the convolution extraction block of the cross-upper projection module to generate a first compression feature; Acquire a first error feedback feature according to the shallow feature and the first compression feature; Processing the first error feedback feature by the convolution extraction block of the cross-upper projection module to obtain a second compression feature; Processing the second compressed feature by the deconvolution extraction block of the cross-upper projection module to obtain a second image feature; The second image feature is superimposed on the first image feature to obtain a fine restoration feature, and the fine restoration feature is input into the arrangement convolution of the cross-upper projection module to perform feature arrangement to obtain an arrangement feature.

8. The image reconstruction method based on cross-iterative back-projection network according to claim 7, characterized in that: The step of extracting deep features from the shallow features by using a plurality of deep extraction modules to obtain deep features of the target low-resolution image includes: Processing the sorting features through the convolution extraction block of the cross-under-projection module to generate a third compressed feature; Processing the third compression feature by the deconvolution extraction block of the cross-under-projection module to generate a third image feature; Acquire a second error feedback feature according to the sorting feature and the third image feature; Processing the second error feedback feature by the convolution extraction block of the cross-under-projection module to obtain a fourth compression feature; Processing the fourth compression feature by the convolution extraction block of the cross-under-projection module to obtain a fifth compression feature; Superimposing the fifth compression feature with the third compression feature to obtain a fine compression feature; The fine compressed features are input into the arranging convolution of the cross-down projection module for feature arrangement to obtain the deep features.

9. The image reconstruction method based on cross-iterative back-projection network according to claim 6, characterized in that: The improved codec attention module includes average pooling, maximum pooling, 1x1 convolution, channel shuffling, 1x1 deconvolution and activation function connected in series.

10. The image reconstruction method based on cross-iterative back-projection network according to claim 1, characterized in that: The shallow extraction module includes a 3x3 convolution, the up-sampling module includes a sub-pixel convolution layer, the sub-pixel convolution layer includes a 3x3 convolution, the input channel of the sub-pixel convolution layer is 64, and the output channel of the sub-pixel convolution layer is 256.

Images