Non-matching image super-resolution reconstruction method based on multi-scale Hash self-adaption

Abstract

The invention discloses a non-matching image super-resolution reconstruction method based on multi-scale Hash self-adaption, and the method comprises the steps: firstly obtaining a low-resolution LR image, carrying out the up-sampling of the LR image to a high-resolution LR image, and enabling the LR image and a high-resolution reference image Ref image to serve as a training sample; then constructing a convolutional neural network based on image super-resolution, inputting the training sample into the convolutional neural network to perform image multi-scale feature enhancement, feature decomposition, matching of different-scale LR images and Ref images, feature extraction, multi-scale feature interaction and feature fusion, and obtaining a reconstructed image; and finally, designing a loss function of the convolutional neural network. And inputting the LR image to be subjected to super-resolution reconstruction and the high-resolution Ref image into the optimized convolutional neural network based on image super-resolution to obtain a super-resolution reconstructed image. According to the method and the device, the problem of inaccurate selection of LR and Ref matching corresponding regions in the prior art is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution reconstruction, and in particular relates to a non-matching image super-resolution reconstruction method based on multi-scale hash adaptation. Background Art

[0002] With the rapid development of social economy and science and technology, my country has gradually moved towards the era of digitalization, networking, informatization and intelligence. As one of the important carriers of data storage and transmission, digital images contain rich information. However, the image quality will be seriously reduced due to the limitations of imaging equipment and the impact of storage and transmission processes, and the integrity and accuracy of information transmission cannot be guaranteed. Therefore, improving image quality has become an urgent problem to be solved.

[0003] In the study of image super-resolution, SISR is a fundamental computer vision task that aims to recover high-frequency details of HR from LR. However, the ill-posed nature of the SISR problem makes it challenging to recover high-quality details. RefSR, which uses external reference images to help super-resolve LR images and transfer detail information to LR images, can overcome the limitations of SISR and has shown good performance in the current study.

[0004] Among them, the correspondence between the reference image and the input image is the key step in the reference image texture transmission. Other reference-based methods only use LR images and Ref images for matching, using the blocks with the highest cosine similarity for matching, and do not consider that if the query texture is severely damaged, low-similarity non-local textures may provide more accurate and richer details than high-similarity non-local textures.

[0005] To solve the above problems, the present invention adds the matching of LR itself on the basis of LR and Ref matching, and proposes a global learnable attention module, which can adaptively modify the similarity score between two features during the training process, so that the global self-similar features between the image and the reference image can repair the texture features of severely damaged low-resolution images. Summary of the invention

[0006] The purpose of the present invention is to provide a non-matching image super-resolution reconstruction method based on multi-scale hash adaptation, which solves the problem of inaccurate selection of corresponding areas for LR and Ref matching in the prior art.

[0007] The technical solution adopted by the present invention is a non-matching image super-resolution reconstruction method based on multi-scale hashing adaptation, which is specifically implemented according to the following steps:

[0008] Step 1: Obtain a low-resolution LR image, upsample the LR image to a high resolution, and use the high-resolution reference image Ref image as training samples;

[0009] Step 2: Construct a convolutional neural network based on image super-resolution, input the training samples into the convolutional neural network to perform multi-scale feature enhancement, feature decomposition, matching of LR images of different scales with Ref images, feature extraction, multi-scale feature interaction, and feature fusion to obtain a reconstructed image;

[0010] Step 3: Design a loss function of the convolutional neural network, and determine the optimized convolutional neural network based on image super-resolution through the loss of the super-resolution reconstructed image;

[0011] Step 4: Input the LR image to be super-resolution reconstructed and the high-resolution Ref image into the optimized convolutional neural network based on image super-resolution to obtain a super-resolution reconstructed image.

[0012] The present invention is also characterized in that:

[0013] The original input images in step 1 are HR images X HR ∈R 3×H×W and Ref image X Ref ∈R 3×H×W , where the number of channels of the image is 3, H is the height of the image, and W is the width of the image. First, the HR image is downsampled four times by cubic spline interpolation to obtain the LR image that needs to be super-resolution reconstructed. When inputting the network, the resolution of the LR image is required to be consistent with that of the Ref image. Therefore, the network has two inputs, namely, the high-resolution input image obtained by upsampling the LR by four times by cubic spline interpolation, and the original high-resolution reference image. Each group of high-resolution input images and high-resolution Ref images is used as a group of training samples;

[0014] Follow the steps below to implement it:

[0015] Step 1.1, downsample the HR image by four times using cubic spline interpolation to obtain the LR image that needs to be super-resolution reconstructed;

[0016] Step 1.2, perform cubic spline interpolation on LR to upsample it four times to obtain a high-resolution input image;

[0017] Step 1.3: Each set of high-resolution input images and high-resolution Ref images are used as a set of training samples.

[0018] Step 2 is implemented as follows:

[0019] Step 2.1, construct a convolutional neural network based on image super-resolution, including a multi-scale image feature enhancement module, a feature decomposition module, matching of LR images of different scales with Ref images, a multi-scale feature interaction module, and a feature fusion module;

[0020] Step 2.2, extract features from the training samples, input them into the image feature enhancement module, perform preliminary enhancement on the input image and reference image in the training samples, and obtain preliminary enhanced features;

[0021] Step 2.3, input the preliminary enhanced features into the feature decomposition module, perform channel expansion on the preliminary enhanced features, and then perform three downsampling decompositions to four scales to obtain the features after four-scale decomposition;

[0022] Step 2.4, matching the features after decomposition at four scales with the same scales in pairs according to the multi-scale hash adaptive module;

[0023] Step 2.5, using the feature extraction module to perform spatial information mining and channel information mining on the features after the fourth scale decomposition;

[0024] Step 2.6: For each corresponding scale, the interaction result obtained in step 2.4 is input into the multi-scale feature interaction module for cross-scale feature interaction, and the output result is then input into the feature extraction module for feature extraction;

[0025] Step 2.7: Fuse the results obtained in step 2.5 with the results obtained in step 2.6 to obtain a reconstructed image.

[0026] Step 2.2 is implemented according to the following steps:

[0027] Step 2.2.1: Extract the features F of the high-resolution input image and the high-resolution reference image obtained in step 1 respectively input and F Ref ;

[0028] Step 2.2.2: Use channel attention to fuse the extracted features to obtain the fused features. The fusion formula is:

[0029] Z input =CA(F input ) (1)

[0030] Z Ref =CA(F Ref ) (2)

[0031] Where: CA(·) represents channel attention, Z input and Z Ref Represents the fused features;

[0032] Step 2.2.3: Use convolution operation to restore the channel of the fused features to obtain the initial enhanced features F input ∈R 3×H×W and F Ref ∈R 3×H×W .

[0033] Step 2.3 is implemented as follows:

[0034] Step 2.3.1, input the preliminary enhanced features into the feature decomposition module for channel information expansion operation, and use the convolution operation to expand the preliminary enhanced features from 3 channels to 32 channels;

[0035] Step 2.3.2: Use convolution to downsample the expanded features three times in succession, gradually reduce the height and width of the features to half of that before downsampling, and gradually expand the channels by 2 times. The specific formula is:

[0036]

[0037]

[0038] Where: Encoder(·) represents the encoder structure, and They represent the input image features and reference image features obtained after scale decomposition, specifically:

[0039] i=1,2,3,4,

[0040]

[0041] i represents four different scales, R represents the size of the feature, 32, 64, 128, 256 are the number of channels, H is the height of the image, and W is the width of the image.

[0042] Step 2.4 is implemented as follows:

[0043] Step 2.4.1: Decompose the input image features into four scales Reference image feature F i Ref Perform feature extraction, i = 1, 2, 3, 4, the feature after input image extraction is Q, the feature after reference image extraction is K, the extraction formula is:

[0044]

[0045] Where CSM(·) is the feature extraction module, Q represents the feature obtained after the input image feature processing, K represents the feature obtained after the reference image feature processing, and the size of Q and K is the same as the input F i The same size;

[0046] Step 2.4.2: Use Super-Bit Locality Sensitive Hashing (SB-LSH) to divide the features into buckets and calculate the similarity within each bucket, λ i Indicates the index of the bucket:

[0047] λ i ={x j |argmax(Mx i )=argmax(Mx j )} (7)

[0048] Where M∈R b×c is a standard orthogonal matrix, and b is the number of hash buckets. Therefore, the query feature vector x i The attention process is expressed by the following formula:

[0049]

[0050] Among them, φ v (·) is the feature embedding layer. x j and x k F′∈R 2hw×c The jth and kth eigenvectors on in ' put ,F in ' put ]. s(·,·) is used to measure the similarity between two vectors, which is represented by a learnable similarity scoring function s l (x i ) and a fixed dot product similarity score function s f (x i ,x j ) can be written as follows:

[0051]

[0052] where s f (x i ,x j )=φ q (x i ) T φ k (x j ), and set φ q (·) and φ k (·) Share the same embedding layer. Yes input (x i ), the jth component in , can be defined as follows:

[0053] s l (x i )=W2σ (W 1φl (x i )+b1)+b2 (10)

[0054] Where σ(·) is the ReLU activation, W1, W2, b1, b2 are learnable parameters. The whole process of SLA can be expressed as:

[0055] SF input =SLA(concat(F input ,F Ref )) input +F input ,

[0056] SF Ref =SLA(concat(F input ,F Ref )) Ref +F Ref (11)

[0057] Among them, SLA(·) input and SLA(·) Ref Represent the feature parts of the input image and the reference image after attention, respectively.

[0058] Step 2.4.3, after SLA self-similarity and global similarity calculation, the input image and reference image features after attention at four scales are obtained respectively;

[0059] Step 2.4.4, add the obtained feature part to the input feature to obtain the features after matching the interaction at different corresponding scales:

[0060]

[0061] in: It is the feature obtained after matching and interacting the input image and the reference image at different scales.

[0062] Step 2.5 is implemented as follows:

[0063] Step 2.5.1: For the feature input after the fourth scale decomposition, the feature extraction module uses layer normalization, uses a large convolution kernel to extract features, and then uses a gate structure to increase the nonlinearity of the network, as shown in formula (13):

[0064] SimpleGate(x)=x1⊙x2 (13)

[0065] Where: x∈R H×W×C is the input feature, x1 and The input features are divided into two parts along the channel dimension, and SimpleGate(·) is the gate structure;

[0066] Since the gate structure will reduce the number of channels of the feature, a convolution is used to restore the channels;

[0067] Step 2.5.2: For channel information extraction, the same as step 2.5.1, the result of step 2.5.1 is restored using layer normalization, and then the channel attention mechanism is used. The gate structure is also used as the activation function, and convolution is used to restore the channel.

[0068] Step 2.6 is implemented as follows:

[0069] Step 2.6.1: Upsample the features of the second scale decomposition obtained in step 2.3 once, upsample the features of the third scale decomposition twice, and upsample the features of the fourth scale decomposition three times. These three features are concatenated with the first scale along the channel, and then convolution is used to compress the number of channels to one-fourth, and then feature extraction is performed through the feature extraction module. The expression is:

[0070]

[0071] Among them: Conv1 represents 1×1 convolution, ↑ represents upsampling, It represents the first scale decomposition feature of feature extraction performed by the feature extraction module;

[0072] In step 2.6.2, the features of the first scale decomposition are downsampled once, the features of the third scale are upsampled once, and the features of the fourth scale are upsampled twice. These three features are concatenated with the features of the second scale decomposition along the channel, and then the number of channels is compressed to one-fourth using convolution. The feature extraction module is then used for feature extraction. The expression is:

[0073]

[0074] Where: ↓ represents downsampling, It represents the features of the second scale decomposition after feature extraction by the feature extraction module;

[0075] Step 2.6.3, downsample the features of the first scale decomposition twice, downsample the features of the second scale decomposition once, upsample the features of the fourth scale once, concatenate these three features with the features of the third scale decomposition along the channel, and then use convolution to compress the number of channels to one-fourth, and then extract features through the feature extraction module. The expression is:

[0076]

[0077] in: It represents the features of the third scale decomposition after feature extraction by the feature extraction module.

[0078] Step 2.6.4: Downsample the features of the first scale decomposition three times, downsample the features of the second scale decomposition twice, and downsample the features of the third scale decomposition once. These three features are concatenated with the features of the fourth scale decomposition along the channel, and then convolution is used to compress the number of channels to one-fourth, and then feature extraction is performed through the feature extraction module. The expression is:

[0079]

[0080] in: It represents the features of the fourth scale decomposition after feature extraction by the feature extraction module.

[0081] Step 2.7 is implemented as follows:

[0082] The features extracted from the feature extraction after restoring the channel in step 2.5 and the features of the four scale decompositions obtained by the feature extraction module in step 2.6 are combined into features and Upsampling, splicing and feature extraction operations are performed step by step, i = 1, 2, 3, 4, and finally the enhanced image, that is, the restored image, is obtained, which is as follows:

[0083]

[0084] Where: H input ,H Ref They represent the enhanced input image and reference image respectively, and Decoder(·) represents the decoder structure.

[0085] Step 3 is implemented as follows:

[0086] In the image super-resolution network, reconstruction loss Lrec, perceptual loss Lper and adversarial loss Ladv are used;

[0087] Step 3.1, reconstruction loss: The L1 norm is used to maintain the spatial structure of the LR image, which is defined as follows:

[0088] L rec =||I HR -I SR ||1 (19)

[0089] Step 3.2, perceptual loss: Use perceptual loss to improve visual quality, defined as follows:

[0090]

[0091] Where V and C are the volume and number of channels of the feature map, φ represents the relu5_1 feature of the VGG19 model, ||·|| F is the Frobenius norm;

[0092] Step 3.3, Adversarial loss: Adversarial loss is defined as follows:

[0093] L adv =-D(I SR ) (twenty one)

[0094] The loss for training the discriminator D is defined as follows:

[0095]

[0096] in isI SR and I HR Random convex combinations of ;

[0097] Step 3.4, the overall loss of the model is finally:

[0098] L=λ rec L rec +λ per L per +λ adv L adv (twenty three).

[0099] The beneficial effect of the present invention is that, based on the multi-scale hash adaptive non-matching image super-resolution reconstruction method, LR and Ref images are first obtained, and the input image and Ref image entering the network are obtained through bicubic interpolation, and then the features are decomposed into 4 scales through the encoder structure, and the input image features and reference image features after the 4-scale decomposition are extracted, and for the fourth scale, the feature extraction module can simultaneously mine channel information and spatial information. For these 4 scales, the input image and the reference image are first interacted at multiple scales, and then the scales of the four scales are fused respectively, and finally a decoder is used to restore the image, and the three loss functions of reconstruction loss, perception loss, and adversarial loss are used to guide them together.

[0100] This method uses attention at multiple scales through a global learnable attention module, and adds the matching of LR itself on the basis of LR and Ref matching. It adaptively modifies the similarity score between the two features during the training process, so that the global self-similar features between the image inside and the reference image can repair the texture features of the severely damaged low-resolution image, making the super-resolution reconstruction more effective and more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 This is a general architecture diagram of the non-matching image super-resolution reconstruction method based on multi-scale hash adaptation of the present invention;

[0102] Figure 2 It is an architecture diagram of a feature extraction module of a non-matching image super-resolution reconstruction method based on multi-scale hashing adaptation of the present invention;

[0103] Figure 3 It is an architecture diagram of the Hash_match module of the non-matching image super-resolution reconstruction method based on multi-scale hash adaptation of the present invention;

[0104] Figure 4 It is an architecture diagram of the MSAI module of the non-matching image super-resolution reconstruction method based on multi-scale hash adaptation of the present invention;

[0105] Figure 5 It is a low-resolution input image of the non-matching image super-resolution reconstruction method based on multi-scale hash adaptiveness of the present invention on the CUFED5 data set;

[0106] Figure 6 It is a high-resolution reference image of the non-matching image super-resolution reconstruction method based on multi-scale hash adaptiveness of the present invention on the CUFED5 data set;

[0107] Figure 7 It is an SR image after image super-resolution reconstruction based on the multi-scale hash adaptive non-matching image super-resolution reconstruction method of the present invention on the CUFED5 dataset. DETAILED DESCRIPTION

[0108] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0109] The present invention is based on a multi-scale hash adaptive non-matching image super-resolution reconstruction method, combined with Figures 1 to 4 , follow the steps below:

[0110] Step 1: Obtain a low-resolution LR image, upsample the LR image to a high resolution, and use the high-resolution reference image Ref image as training samples;

[0111] The original input images in step 1 are HR images X HR ∈R 3×H×W and Ref image X Ref ∈R 3×H×W, where the number of channels of the image is 3, H is the height of the image, and W is the width of the image. First, the HR image is downsampled four times by cubic spline interpolation to obtain the LR image that needs to be super-resolution reconstructed. When inputting the network, the resolution of the LR image is required to be consistent with that of the Ref image. Therefore, the network has two inputs, namely, the high-resolution input image obtained by upsampling the LR by four times by cubic spline interpolation, and the original high-resolution reference image. Each group of high-resolution input images and high-resolution Ref images is used as a group of training samples;

[0112] Follow the steps below to implement it:

[0113] Step 1.1, downsample the HR image by four times using cubic spline interpolation to obtain the LR image that needs to be super-resolution reconstructed;

[0114] Step 1.2, perform cubic spline interpolation on LR to upsample it four times to obtain a high-resolution input image;

[0115] Step 1.3: Each set of high-resolution input images and high-resolution Ref images are used as a set of training samples.

[0116] Step 2: Construct a convolutional neural network based on image super-resolution, input the training samples into the convolutional neural network to perform multi-scale feature enhancement, feature decomposition, matching of LR images of different scales with Ref images, feature extraction, multi-scale feature interaction, and feature fusion to obtain a reconstructed image;

[0117] Step 2 is implemented as follows:

[0118] Step 2.1, construct a convolutional neural network based on image super-resolution, including a multi-scale image feature enhancement module, a feature decomposition module, matching of LR images of different scales with Ref images, a multi-scale feature interaction module, and a feature fusion module;

[0119] Step 2.2, extract features from the training samples, input them into the image feature enhancement module, perform preliminary enhancement on the input image and reference image in the training samples, and obtain preliminary enhanced features;

[0120] Step 2.2 is implemented according to the following steps:

[0121] Step 2.2.1: Extract the features F of the high-resolution input image and the high-resolution reference image obtained in step 1 respectively input and F Ref ;

[0122] Step 2.2.2: Use channel attention to fuse the extracted features to obtain the fused features. The fusion formula is:

[0123] Z input=CA(F input ) (1)

[0124] Z Ref =CA(F Ref ) (2)

[0125] Where: CA(·) represents channel attention, Z input and Z Ref Represents the fused features;

[0126] Step 2.2.3: Use convolution operation to restore the channel of the fused features to obtain the initial enhanced features F input ∈R 3×H×W and F Ref ∈R 3×H×W .

[0127] Step 2.3, input the preliminary enhanced features into the feature decomposition module, perform channel expansion on the preliminary enhanced features, and then perform three downsampling decompositions to four scales to obtain the features after four-scale decomposition;

[0128] Step 2.3 is implemented as follows:

[0129] Step 2.3.1, input the preliminary enhanced features into the feature decomposition module for channel information expansion operation, and use the convolution operation to expand the preliminary enhanced features from 3 channels to 32 channels;

[0130] Step 2.3.2: Use convolution to downsample the expanded features three times in succession, gradually reduce the height and width of the features to half of that before downsampling, and gradually expand the channels by 2 times. The specific formula is:

[0131]

[0132] Where: Encoder(·) represents the encoder structure, and They represent the input image features and reference image features obtained after scale decomposition, specifically:

[0133] i=1,2,3,4,

[0134]

[0135] i represents four different scales, R represents the size of the feature, 32, 64, 128, 256 are the number of channels, H is the height of the image, and W is the width of the image.

[0136] Step 2.4, matching the features after decomposition at four scales with the same scales in pairs according to the multi-scale hash adaptive module;

[0137] Step 2.4 is implemented as follows:

[0138] Step 2.4.1: Decompose the input image features into four scales Reference image feature F i Ref Perform feature extraction, i = 1, 2, 3, 4, the feature after input image extraction is Q, the feature after reference image extraction is K, the extraction formula is:

[0139]

[0140] Where CSM(·) is the feature extraction module, Q represents the feature obtained after the input image feature processing, K represents the feature obtained after the reference image feature processing, and the size of Q and K is the same as the input F i The same size;

[0141] Step 2.4.2: Traditional non-local attention needs to consider all vectors and perform correlation calculations between them. Different from traditional non-local attention, in order to reduce the high computational overhead of calculating global similarity matching in non-matching image super-resolution, super-bit locality sensitive hashing (SB-LSH) is used to divide the features into buckets and calculate the similarity within each bucket, λ i Indicates the index of the bucket:

[0142] λ i ={x j |argmax(Mx i )=argmax(Mx j )} (7)

[0143] Where M∈R b×c is a standard orthogonal matrix, and b is the number of hash buckets. Therefore, the query feature vector x i The attention process is expressed by the following formula:

[0144]

[0145] Among them, φ v (·) is the feature embedding layer. x j and x k F′∈R 2hw×c The jth and kth eigenvectors on in ' put ,F in ' put ]. s(·,·) is used to measure the similarity between two vectors, which is represented by a learnable similarity scoring function s l (x i ) and a fixed dot product similarity score function s f(x i ,x j ) can be written as follows:

[0146]

[0147] where s f (x i ,x j )=φ q (x i ) T φ k (x j ), and set φ q (·) and φ k (·) Share the same embedding layer. Yes input (x i ), the jth component in , can be defined as follows:

[0148] s l (x i )=W 2σ (W 1φl (x i )+b1)+b2 (10)

[0149] Where σ(·) is the ReLU activation, W1, W2, b1, b2 are learnable parameters. The whole process of SLA can be expressed as:

[0150] SF input =SLA(concat(F input ,F Ref )) input +F input ,

[0151] SF Ref =SLA(concat(F input ,F Ref )) Ref +F Ref (11)

[0152] Among them, SLA(·) input and SLA(·) Ref Represent the feature parts of the input image and the reference image after attention, respectively.

[0153] Step 2.4.3, after SLA self-similarity and global similarity calculation, the input image and reference image features after attention at four scales are obtained respectively;

[0154] Step 2.4.4, add the obtained feature part to the input feature to obtain the features after matching the interaction at different corresponding scales:

[0155]

[0156] in: It is the feature obtained after matching and interacting the input image and the reference image at different scales.

[0157] Step 2.5, using the feature extraction module to perform spatial information mining and channel information mining on the features after the fourth scale decomposition;

[0158] Step 2.5 is implemented as follows:

[0159] Step 2.5.1: For the feature input after the fourth scale decomposition, the feature extraction module uses layer normalization, uses a large convolution kernel to extract features, and then uses a gate structure to increase the nonlinearity of the network, as shown in formula (13):

[0160] SimpleGate(x)=x1⊙x2 (13)

[0161] Where: x∈R H×W×C is the input feature, x1 and The input features are divided into two parts along the channel dimension, and SimpleGate(·) is the gate structure;

[0162] Since the gate structure will reduce the number of channels of the feature, a convolution is used to restore the channels;

[0163] Step 2.5.2: For channel information extraction, the same as step 2.5.1, the result of step 2.5.1 is restored using layer normalization, and then the channel attention mechanism is used. The gate structure is also used as the activation function, and convolution is used to restore the channel.

[0164] Step 2.6: For each corresponding scale, the interaction result obtained in step 2.4 is input into the multi-scale feature interaction module for cross-scale feature interaction, and the output result is then input into the feature extraction module for feature extraction;

[0165] Step 2.6 is implemented as follows:

[0166] Step 2.6.1: Upsample the features of the second scale decomposition obtained in step 2.3 once, upsample the features of the third scale decomposition twice, and upsample the features of the fourth scale decomposition three times. These three features are concatenated with the first scale along the channel, and then convolution is used to compress the number of channels to one-fourth, and then feature extraction is performed through the feature extraction module. The expression is:

[0167]

[0168] Among them: Conv1 represents 1×1 convolution, ↑ represents upsampling, It represents the first scale decomposition feature of feature extraction performed by the feature extraction module;

[0169] In step 2.6.2, the features of the first scale decomposition are downsampled once, the features of the third scale are upsampled once, and the features of the fourth scale are upsampled twice. These three features are concatenated with the features of the second scale decomposition along the channel, and then the number of channels is compressed to one-fourth using convolution. The feature extraction module is then used for feature extraction. The expression is:

[0170]

[0171] Where: ↓ represents downsampling, It represents the features of the second scale decomposition after feature extraction by the feature extraction module;

[0172] Step 2.6.3, downsample the features of the first scale decomposition twice, downsample the features of the second scale decomposition once, upsample the features of the fourth scale once, concatenate these three features with the features of the third scale decomposition along the channel, and then use convolution to compress the number of channels to one-fourth, and then extract features through the feature extraction module. The expression is:

[0173]

[0174] in: It represents the features of the third scale decomposition after feature extraction by the feature extraction module.

[0175] Step 2.6.4: Downsample the features of the first scale decomposition three times, downsample the features of the second scale decomposition twice, and downsample the features of the third scale decomposition once. These three features are concatenated with the features of the fourth scale decomposition along the channel, and then convolution is used to compress the number of channels to one-fourth, and then feature extraction is performed through the feature extraction module. The expression is:

[0176]

[0177] in: It represents the features of the fourth scale decomposition after feature extraction by the feature extraction module.

[0178] Step 2.7: Fuse the results obtained in step 2.5 with the results obtained in step 2.6 to obtain a reconstructed image.

[0179] Step 2.7 is implemented as follows:

[0180] The features extracted from the feature extraction after restoring the channel in step 2.5 and the features of the four scale decompositions obtained by the feature extraction module in step 2.6 are combined into features and Upsampling, splicing and feature extraction operations are performed step by step, i = 1, 2, 3, 4, and finally the enhanced image, that is, the restored image, is obtained, which is as follows:

[0181]

[0182] Where: H input ,H Ref They represent the enhanced input image and reference image respectively, and Decoder(·) represents the decoder structure.

[0183] Step 3: Design a loss function of the convolutional neural network, and determine the optimized convolutional neural network based on image super-resolution through the loss of the super-resolution reconstructed image;

[0184] Step 3 is implemented as follows:

[0185] In the image super-resolution network, reconstruction loss Lrec, perceptual loss Lper and adversarial loss Ladv are used;

[0186] Step 3.1, reconstruction loss: The L1 norm is used to maintain the spatial structure of the LR image, which is defined as follows:

[0187] L rec =||I HR -I SR ||1 (19)

[0188] Step 3.2, perceptual loss: Use perceptual loss to improve visual quality, defined as follows:

[0189]

[0190] Where V and C are the volume and number of channels of the feature map, φ represents the relu5_1 feature of the VGG19 model, ||·|| F is the Frobenius norm;

[0191] Step 3.3, Adversarial loss: Adversarial loss is defined as follows:

[0192] L adv =-D(I SR ) (twenty one)

[0193] The loss for training the discriminator D is defined as follows:

[0194]

[0195] in isI SR and I HR Random convex combinations of ;

[0196] Step 3.4, the overall loss of the model is finally:

[0197] L=λ rec L rec +λ per L per +λ adv L adv (twenty three);

[0198] Step 4: Input the LR image to be super-resolution reconstructed and the high-resolution Ref image into the optimized convolutional neural network based on image super-resolution to obtain a super-resolution reconstructed image.

[0199] In super-resolution reconstruction, the overall loss function formed by the reconstruction loss Lrec, the perceptual loss Lper, the adversarial loss Ladv and their combination together provide a clear optimization target for the model's optimization process. By optimizing this loss function, the model can adjust the weight parameters during training so that the generated super-resolution images are improved in terms of pixel accuracy, feature level and visual realism. Specifically, the reconstruction loss ensures that the model can restore the details of the high-resolution image at the pixel level, the perceptual loss helps the model generate more visually natural images through high-level feature matching, and the adversarial loss uses the generative adversarial network (GAN) to improve the authenticity and realism of the reconstructed image.

[0200] This optimization process ultimately serves step 4, that is, when the low-resolution (LR) image to be reconstructed and the high-resolution reference (Ref) image are input into the optimized convolutional neural network model, the model can output a reconstructed image with high-quality super-resolution. As the loss decreases during training, the model's ability to restore image details gradually improves, so that the quality of the output super-resolution image is closer to the real high-resolution image. By optimizing the loss function, the model continuously adjusts the weight parameters to minimize the difference. The optimized model can generate an effect as close to the real high-resolution image as possible from the input LR image, thereby ensuring the clarity and detail of the generated image. This result is directly due to the optimization of the overall loss function during the training process, which enables the model to achieve the best practical application effect in step 4.

[0201] Example 1

[0202] The present invention is based on a multi-scale hash adaptive non-matching image super-resolution reconstruction method, combined with Figure 1 , follow the steps below:

[0203] Step 1: Obtain a low-resolution LR image, upsample the LR image to a high resolution, and use the high-resolution reference image Ref image as training samples;

[0204] Step 2: Construct a convolutional neural network based on image super-resolution, input the training samples into the convolutional neural network to perform multi-scale feature enhancement, feature decomposition, matching of LR images of different scales with Ref images, feature extraction, multi-scale feature interaction, and feature fusion to obtain a reconstructed image;

[0205] Step 3: Design a loss function of the convolutional neural network, and determine the optimized convolutional neural network based on image super-resolution through the loss of the super-resolution reconstructed image;

[0206] Step 4: Input the LR image to be super-resolution reconstructed and the high-resolution Ref image into the optimized convolutional neural network based on image super-resolution to obtain a super-resolution reconstructed image.

[0207] Example 2

[0208] The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation of the present invention is used to Figure 5 The low-resolution input image shown, Figure 6 The high-resolution reference image is shown in Figure 2, and the LR image is reconstructed by super-resolution. The SR image after super-resolution is shown in Figure 2. Figure 7 shown.

[0209] according to Figure 5 , Figure 6 , Figure 7 By comparison, it can be seen that the non-matching image super-resolution reconstruction performed by the present invention makes the reconstructed image very clear in terms of details. The non-matching image super-resolution reconstruction technology of the present invention can more effectively solve the problem that the low-resolution input image has poor details and the reference image is difficult to accurately match to the appropriate position, and adaptively perform global matching to capture image detail information, thereby better improving image quality.

[0210] Example 3

[0211] The non-matching image super-resolution reconstruction method based on multi-scale hash adaptiveness of the present invention is used to perform super-resolution reconstruction on 126 pairs of LR images of the dataset CUFED5, 80 pairs of LR images of Sun80, and 100 pairs of LR images of Urban100. The objective evaluation index (PSNR) and structural similarity (SSIM) of the image are tested respectively. The results are shown in Table 1:

[0212] Table 1

[0213]

[0214] According to Table 1, the PSNR and SSIM of the non-matching image super-resolution reconstruction method based on multi-scale hash adaptation of the invention measured on three data sets have very good performance, and at the same time, have good robustness.

[0215] Through the above method, a non-matching image super-resolution reconstruction method based on multi-scale hash adaptation is invented. First, LR and Ref images are obtained, and the input image and Ref image entering the network are obtained through bicubic interpolation. Then, the features are decomposed into 4 scales through the encoder structure, and the input image features and reference image features after the 4-scale decomposition are extracted. The input image and the reference image are interacted at multiple scales, and the reference image is used to match the corresponding input image position. Then, the four scales are scale-fused respectively, and finally a decoder is used to restore the image, and the three loss functions of reconstruction loss, perception loss, and adversarial loss are jointly used for guidance.

[0216] This method effectively solves the problem that there is no reference image information available in the single-view super-resolution reconstruction method, and the matching position is not exactly corresponding when using the reference image to help the low-resolution image reconstruction. Since this method is performed at multiple scales, the robustness is greatly improved. This method can adaptively modify the similarity score between two features during the training process, so that the global self-similar features between the image and the reference image can repair the texture features of the severely damaged low-resolution image, making the super-resolution reconstruction effect better.

Claims

1. A non-matching image super-resolution reconstruction method based on multi-scale hashing adaptation, characterized in that: Follow the steps below to implement it: Step 1: Obtain a low-resolution LR image, upsample the LR image to a high resolution, and use the high-resolution reference image Ref image as training samples; Step 2: Construct a convolutional neural network based on image super-resolution, input the training samples into the convolutional neural network to perform multi-scale feature enhancement, feature decomposition, matching of LR images of different scales with Ref images, feature extraction, multi-scale feature interaction, and feature fusion to obtain a reconstructed image; Step 3: Design a loss function of the convolutional neural network, and determine the optimized convolutional neural network based on image super-resolution through the loss of the super-resolution reconstructed image; Step 4: Input the LR image to be super-resolution reconstructed and the high-resolution Ref image into the optimized convolutional neural network based on image super-resolution to obtain a super-resolution reconstructed image.

2. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 1 is characterized in that: The original input images in step 1 are HR images X HR ∈R 3×H×W and Ref image X Ref ∈R 3×H×W , where the number of channels of the image is 3, H is the height of the image, and W is the width of the image. First, the HR image is downsampled four times by cubic spline interpolation to obtain the LR image that needs to be super-resolution reconstructed. When inputting the network, the resolution of the LR image is required to be consistent with that of the Ref image. Therefore, the network has two inputs, namely, the high-resolution input image obtained by upsampling the LR by four times by cubic spline interpolation, and the original high-resolution reference image. Each group of high-resolution input images and high-resolution Ref images is used as a group of training samples; Follow the steps below to implement it: Step 1.1, downsample the HR image by four times using cubic spline interpolation to obtain the LR image that needs to be super-resolution reconstructed; Step 1.2, perform cubic spline interpolation on LR to upsample it four times to obtain a high-resolution input image; Step 1.3: Each set of high-resolution input images and high-resolution Ref images are used as a set of training samples.

3. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 2 is characterized in that: The step 2 is specifically implemented according to the following steps: Step 2.1, construct a convolutional neural network based on image super-resolution, including a multi-scale image feature enhancement module, a feature decomposition module, matching of LR images of different scales with Ref images, a multi-scale feature interaction module, and a feature fusion module; Step 2.2, extract features from the training samples, input them into the image feature enhancement module, perform preliminary enhancement on the input image and reference image in the training samples, and obtain preliminary enhanced features; Step 2.3, input the preliminary enhanced features into the feature decomposition module, perform channel expansion on the preliminary enhanced features, and then perform three downsampling decompositions to four scales to obtain the features after four-scale decomposition; Step 2.4, matching the features after decomposition at four scales with the same scales in pairs according to the multi-scale hash adaptive module; Step 2.5, using the feature extraction module to perform spatial information mining and channel information mining on the features after the fourth scale decomposition; Step 2.6: For each corresponding scale, the interaction result obtained in step 2.4 is input into the multi-scale feature interaction module for cross-scale feature interaction, and the output result is then input into the feature extraction module for feature extraction; Step 2.7: Fuse the results obtained in step 2.5 with the results obtained in step 2.6 to obtain a reconstructed image.

4. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 3 is characterized in that: The step 2.2 is specifically implemented according to the following steps: Step 2.2.1: Extract the features F of the high-resolution input image and the high-resolution reference image obtained in step 1 respectively input and F Ref ; Step 2.2.2: Use channel attention to fuse the extracted features to obtain the fused features. The fusion formula is: Z input =CA(F input ) (1) Z Ref =CA(F Ref ) (2) Where: CA(·) represents channel attention, Z input and Z Ref Represents the fused features; Step 2.2.3: Use convolution operation to restore the channel of the fused features to obtain the initial enhanced features F input ∈R 3 ×H×W and F Ref ∈R 3×H×W .

5. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 4 is characterized in that: The step 2.3 is specifically implemented according to the following steps: Step 2.3.1, input the preliminary enhanced features into the feature decomposition module for channel information expansion operation, and use the convolution operation to expand the preliminary enhanced features from 3 channels to 32 channels; Step 2.3.2: Use convolution to downsample the expanded features three times in succession, gradually reduce the height and width of the features to half of that before downsampling, and gradually expand the channels by 2 times. The specific formula is: F i input =Encoder(F input ) (3) F i Ref =Encoder(F Ref ) (4) Where: Encoder(·) represents the encoder structure, F i input and F i Ref They represent the input image features and reference image features obtained after scale decomposition, specifically: i＝1，2，3，4， F1 input ,F i Ref ∈R 32×H×W , i represents four different scales, R represents the size of the feature, 32, 64, 128, 256 are the number of channels, H is the height of the image, and W is the width of the image.

6. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 5 is characterized in that: The step 2.4 is specifically implemented according to the following steps: Step 2.4.1: Decompose the input image features F at four scales i input , reference image feature F i Ref Perform feature extraction, i = 1, 2, 3, 4, the feature after input image extraction is Q, the feature after reference image extraction is K, the extraction formula is: Q=CSM(F i input ) (5) K=CSM(F i Ref ) (6) Where CSM(·) is the feature extraction module, Q represents the feature obtained after the input image feature processing, K represents the feature obtained after the reference image feature processing, and the size of Q and K is the same as the input F i The same size; Step 2.4.2: Use Super-Bit Locality Sensitive Hashing (SB-LSH) to divide the features into buckets and calculate the similarity within each bucket, λ i Indicates the index of the bucket: λ i ={x j |argmax(Mx i )=argmax(Mx j )} (7) Where M∈R b×c is a standard orthogonal matrix, b is the number of hash buckets, so the query feature vector x i The attention process is expressed by the following formula: Among them, φ v (·) is the feature embedding layer, x j and x k F′∈R 2hw×c The jth and kth eigenvectors on input ,F′ input ], s(·,·) is used to measure the similarity between two vectors, which is represented by a learnable similarity scoring function s l (x i ) and a fixed dot product similarity score function s f (x i ,x j ) can be written as follows: where s f (x i ,x j )=φ q (x i ) T φ k (x j ), and set φ q (·) and φ k (·) share the same embedding layer, Yes input (x i ), the jth component in , can be defined as follows: s l (x i )=W 2σ (W 1φl (x i )+b1)+b2 (10) Where σ(·) is the ReLU activation, W1, W2, b1, b2 are learnable parameters, and the whole process of SLA can be expressed as: Among them, SLA(·) input and SLA(·) Ref Respectively represent the feature parts of the input image and the reference image after attention; Step 2.4.3, after SLA self-similarity and global similarity calculation, the input image and reference image features after attention at four scales are obtained respectively; Step 2.4.4, add the obtained feature part to the input feature to obtain the features after matching the interaction at different corresponding scales: in: It is the feature obtained after matching and interacting the input image and the reference image at different scales.

7. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 6 is characterized in that: The step 2.5 is specifically implemented according to the following steps: Step 2.5.1: For the feature input after the fourth scale decomposition, the feature extraction module uses layer normalization, uses a large convolution kernel to extract features, and then uses a gate structure to increase the nonlinearity of the network, as shown in formula (13): SimpleGate(x)=x1☉x2 (13) Where: x∈R H×W×C is the input feature, x1 and The input features are divided into two parts along the channel dimension, and SimpleGate(·) is the gate structure; Since the gate structure will reduce the number of channels of the feature, a convolution is used to restore the channels; Step 2.5.2: For channel information extraction, the same as step 2.5.1, the result of step 2.5.1 is restored using layer normalization, and then the channel attention mechanism is used. The gate structure is also used as the activation function, and convolution is used to restore the channel.

8. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 7 is characterized in that: The step 2.6 is specifically implemented according to the following steps: Step 2.6.1: Upsample the features of the second scale decomposition obtained in step 2.3 once, upsample the features of the third scale decomposition twice, and upsample the features of the fourth scale decomposition three times. These three features are concatenated with the first scale along the channel, and then convolution is used to compress the number of channels to one-fourth, and then feature extraction is performed through the feature extraction module. The expression is: Among them: Conv1 represents 1×1 convolution, ↑ represents upsampling, It represents the first scale decomposition feature of feature extraction performed by the feature extraction module; In step 2.6.2, the features of the first scale decomposition are downsampled once, the features of the third scale are upsampled once, and the features of the fourth scale are upsampled twice. These three features are concatenated with the features of the second scale decomposition along the channel, and then the number of channels is compressed to one-fourth using convolution. The feature extraction module is then used for feature extraction. The expression is: Where: ↓ represents downsampling, It represents the features of the second scale decomposition after feature extraction by the feature extraction module; Step 2.6.3, downsample the features of the first scale decomposition twice, downsample the features of the second scale decomposition once, upsample the features of the fourth scale once, concatenate these three features with the features of the third scale decomposition along the channel, and then use convolution to compress the number of channels to one-fourth, and then extract features through the feature extraction module. The expression is: in: It represents the features of the third scale decomposition after feature extraction by the feature extraction module; Step 2.6.4: Downsample the features of the first scale decomposition three times, downsample the features of the second scale decomposition twice, and downsample the features of the third scale decomposition once. These three features are concatenated with the features of the fourth scale decomposition along the channel, and then convolution is used to compress the number of channels to one-fourth, and then feature extraction is performed through the feature extraction module. The expression is: in: It represents the features of the fourth scale decomposition after feature extraction by the feature extraction module.

9. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 8 is characterized in that: The step 2.7 is specifically implemented according to the following steps: The features extracted from the feature extraction after restoring the channel in step 2.5 and the features of the four scale decompositions obtained by the feature extraction module in step 2.6 are combined into features and Upsampling, splicing and feature extraction operations are performed step by step, i = 1, 2, 3, 4, and finally the enhanced image, that is, the restored image, is obtained, which is as follows: Where: H input ,H Ref They represent the enhanced input image and reference image respectively, and Decoder(·) represents the decoder structure.

10. The non-matching image super-resolution reconstruction method based on multi-scale hash adaptation according to claim 9, characterized in that: The step 3 is specifically implemented according to the following steps: In the image super-resolution network, reconstruction loss Lrec, perceptual loss Lper and adversarial loss Ladv are used; Step 3.1, reconstruction loss: The L1 norm is used to maintain the spatial structure of the LR image, which is defined as follows: L rec =||I HR -I SR ||1 (19) Step 3.2, perceptual loss: Use perceptual loss to improve visual quality, defined as follows: Where V and C are the volume and number of channels of the feature map, φ represents the relu5_1 feature of the VGG19 model, ||·|| F is the Frobenius norm; Step 3.3, Adversarial loss: Adversarial loss is defined as follows: L adv =-D(I SR ) (21) The loss for training the discriminator D is defined as follows: in isI SR and I HR Random convex combinations of ; Step 3.4, the overall loss of the model is finally: L=λ rec L rec +λ per L per +λ adv L adv (23).

Images