A super-resolution reconstruction method of infrared images based on reference images

Abstract

The invention belongs to the technical field of infrared image super-resolution in computer vision, and in particular is a method for super-resolution reconstruction of infrared images based on reference images, comprising the following steps: step 1, data set preprocessing: adjusting the size of a prepared data set and performing rough registration; step 2, building a network model: building a network model including a mask marking module, a feature matching module, a feature alignment module, a feature fusion module and a residual module; step 3, training the network model: training a super-resolution reconstruction model of infrared images; the invention can accurately identify the complexity of each region of an input image, adopt different methods to perform super-resolution reconstruction on the region according to the complexity of the region, effectively reduce the calculation amount of the overall network, and improve the super-resolution reconstruction effect of complex regions.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared image super-resolution in computer vision, and in particular to a method for super-resolution reconstruction of infrared images based on a reference image. Background Art

[0002] Infrared images can be used for effective monitoring and identification in dark or harsh environments, but infrared images have the problems of low pixels and poor quality. Due to the imaging principle of infrared lenses, the use of super-resolution reconstruction technology can effectively improve the quality of infrared images, and relevant personnel can use high-quality infrared images for subsequent tasks such as segmentation and detection. However, the effect of infrared image super-resolution reconstruction in complex areas of the image is not satisfactory and will produce artifacts. Recent studies have provided high-frequency information of visible light images to low-resolution infrared images through self-attention networks, achieving better super-resolution reconstruction effects and eliminating artifacts. However, this method uses the same and large amount of computing resources for all features in the reference image feature matching stage, resulting in excessive computational complexity, and the reconstruction effect of complex areas is still insufficient. Therefore, it is of great research significance to design a method and system for infrared image super-resolution reconstruction based on reference images.

[0003] The Chinese patent publication number is "CN117495680A", and its name is "Multi-contrast MRI image super-resolution method based on feature fusion Transformer". This method extracts low-resolution image features through a dual-domain feature extraction module, and uses Transformer to deeply fuse the low-resolution image features with the multi-contrast MRI image and output the reconstructed high-resolution image; the image obtained by this method has poor reconstruction effect in complex areas, and the calculation amount is large, and the reconstruction of low-frequency information also occupies a large amount of computing resources; therefore, we propose a reference image-based infrared image super-resolution reconstruction method and system to solve the above problems. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides a method and system for super-resolution reconstruction of infrared images based on reference images, which divides the low-resolution image into regions, marks the locations of complex regions and simple regions with binary masks, supplements the high-frequency information of the reference image to the low-resolution image in the complex region, and uses a residual network super-resolution reconstruction for the simple region, thereby solving the problems of waste of computing resources and insufficient computational effort in complex regions.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] A method for super-resolution reconstruction of infrared images based on a reference image comprises the following steps:

[0009] Step 1, data set preprocessing: prepare infrared and visible light datasets of the same scene, enhance the infrared image DDE to improve the contrast; adjust the image size after pairing the infrared and visible light images and augment the image data;

[0010] Step 2, build the network model: the entire network includes a mask marking module, a feature matching module, a feature alignment module, a feature fusion module and a residual module;

[0011] Step 3, training the network model: input the data set prepared in step 1 into the network model constructed in step 2 for training;

[0012] Step 4, design the loss function: obtain the best reconstructed image effect by minimizing the loss function and save the model parameters;

[0013] Step 5, save the model: solidify the model parameters. When the infrared image super-resolution reconstruction is required later, the low-resolution infrared image and the high-resolution visible light image can be input into the network at the same time to obtain the high-resolution infrared image.

[0014] Furthermore, the data set in step 1 uses FLIRADAS and LLVIP data sets; some image pairs are selected therefrom for rough registration;

[0015] Furthermore, the data augmentation methods in step 1 include image translation, image rotation, and image scaling, and all transformation centers are the image center;

[0016] Furthermore, in step 2, the mask marking module divides the image and performs downsampling, calculates the offset of the image, calculates the binary mask of each region of the divided image by gumble softmax, inserts the label value of the binary mask into the features of each region of the image, repeats the previous operation after upsampling the image, determines the boundary between the simple region and the complex region by comparing the mask difference between the two regions, assigns the generated mask value to the feature vector of each region, thereby dividing the image into two parts, the simple region and the complex region, trains the simple region with the residual network, and reconstructs the complex region finely by fusion of reference image features;

[0017] Furthermore, the feature matching module has a built-in new Transformer reference image matching method, which detects the 1×, 2× and 4× feature vectors corresponding to the infrared image with mask values ​​and the reference image through mask detection, where Q is the infrared image feature vector of the corresponding multiple, K is the feature vector after downsampling and upsampling of the reference image, and V is the feature vector of the reference image; Q, K, V correspond to Q after mask detectionT , K T 、V T are the feature vectors re-screened and arranged by mask marking, is the feature vector extracted by dilated convolution after downsampling and upsampling of the reference image; With Q T and K T With Q T The correlation of the feature vectors is calculated respectively and the calculation results are weighted by different weights a and b, and the V corresponding to the area with the maximum correlation value is taken. T As the reference image of this area of ​​the infrared image, V T The eigenvector and Q T The feature vector channels are concatenated and input into the feature alignment module;

[0018] The feature alignment module matches the reference image features of different scaling factors with the low-resolution image positions through multi-scale information extraction and dilated convolution, reduces the channels of feature mapping, uses deformable convolution to avoid artifacts in the reconstructed image caused by feature position offset, and selects the most appropriate reference image feature variable pair to output to the feature fusion module;

[0019] The feature fusion module fuses the 1× reference image texture features into the 1× infrared image through weight parameters β and γ, upsamples the fused image and fuses it with the 2× reference image texture features, and then upsamples the fused image again and fuses it with the 4× reference image texture features to reconstruct a high-resolution image of a complex area;

[0020] Furthermore, the mask detection in step 2 needs to be pre-trained, the registered infrared visible light image data set is masked, and after the mask value of the visible light image is subtracted from the mask value of the infrared image in the same area, these data are combined and fitted into a normal distribution function. The mask detection calls this function to rearrange the regions of the visible light image in subsequent modules, give priority to the retrieval of visible light images with higher normal distribution values, and set an adaptive adjustable threshold to abandon the subsequent Transformer texture matching retrieval of the regions at both ends of the normal distribution, thereby saving computing resources;

[0021] Furthermore, the step 4 adopts L1 loss, perceptual loss, and adversarial loss.

[0022] A terminal device for super-resolution reconstruction of infrared images based on reference images comprises a processor, a memory, a communication port and a computer program stored in the memory and executed by the processor. When the computer program is executed, the image super-resolution reconstruction method is implemented.

[0023] A computer-readable storage medium for super-resolution reconstruction of an infrared image based on a reference image, wherein the computer-readable medium includes a stored computer program, wherein when the computer program is run, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for super-resolution reconstruction of an infrared image based on a reference image.

[0024] (III) Beneficial effects

[0025] Compared with the prior art, the present invention provides a method and system for super-resolution reconstruction of infrared images based on reference images, which has the following beneficial effects:

[0026] The present invention proposes a super-resolution reconstruction method for infrared images based on reference images, classifies regions of different complexity by mask marking, and divides the overall image into complex regions and simple regions. The complex regions use the Transformer self-attention calculation method to fuse the high-resolution visible light image texture information into the low-resolution infrared image, thereby improving the super-resolution reconstruction effect of the complex regions. Different from the traditional Transformer, the network Transformer only selects the complex regions of the infrared image to perform texture matching on the complex regions of the reference image, rearranges the calculation regions by embedding a pre-trained retrieval function, and adopts a local retrieval matching method to reduce the matching calculation amount of the Transformer. For simple regions, a residual network is used to ensure the reconstruction effect while reducing the ultra-high calculation amount caused by the redundant calculation of the self-attention calculation mechanism.

[0027] The present invention, by designing a feature alignment module, senses the pixel motion changes between the reference image features and the low-resolution image features, increases the receptive field, and retains the boundary details of the features. The deformable convolution uses dynamic weights to adaptively sample the features, so that features of different magnifications can be automatically aligned, avoiding artifacts in the reconstructed image due to feature misalignment when the visible light image supplements the infrared image details.

[0028] The present invention designs a new feature fusion module to fuse multiple segments of reference features of different magnifications that are temporarily aligned at the same position, prevent the corresponding complementary information from being gradually weakened during training, use high-magnification features to refine low-magnification features, and learn the complex nonlinear mapping from LR to HR image space without the need for additional parameters, thereby improving the reconstruction effect of complex image areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a method for super-resolution reconstruction of infrared images based on a reference image according to the present invention;

[0030] Figure 2A network structure diagram of a method for super-resolution reconstruction of infrared images based on a reference image according to the present invention;

[0031] Figure 3 This is a network structure diagram of the mask marking module of the present invention;

[0032] Figure 4 This is a structural diagram of the feature matching module of the present invention;

[0033] Figure 5 This is a structural diagram of the feature alignment module of the present invention;

[0034] Figure 6 This is a structural diagram of the feature fusion module of the present invention;

[0035] Figure 7 It is the residual module diagram of the present invention;

[0036] Figure 8 This is a comparison table of the complexity of feature matching of the present invention and traditional Transformer;

[0037] Fig. 9 A comparison table of related indicators between the prior art and the method proposed in the present invention;

[0038] Fig.10 This is a diagram of the terminal equipment of the system of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example

[0041] like Figure 1 , 2 As shown, an infrared image super-resolution reconstruction method based on a reference image is proposed in one embodiment of the present invention:

[0042] Step 1, data set preprocessing: Use the high-contrast daytime images of the LLVIP and FLIRADAS data sets as the training and testing data sets for the overall network. The training set uses 6700 pairs of daytime images and the test set uses 1500 pairs of daytime images. After the infrared images are enhanced using DDE, the images are processed to the same size and then data augmentation is performed including translation, rotation and scaling. The transformation center is the image center. Finally, MATLAB is used to perform coarse alignment of the images through the mapping matrix. The input low-resolution infrared image differs by 4× from the visible light reference image in size. Use the 2023 PBVS reference image data set as the pre-trained infrared-visible light full alignment data set for the mask labeling module.

[0043] Step 2: Build a network model: Figure 3 As shown, the reference image and infrared image are input into the mask marking module respectively. The mask marking module first divides the image into multiple regions of the same size, downsamples the image and calculates the offset. The binary mask marks the complexity of each region. The calculation formula is as follows

[0044] F=f head (C l ,C g ,C w )

[0045]

[0046] ΔP=r·f offstes (F)∈R 2×H×W

[0047]

[0048] ΔP is a content-dependent offset matrix used to distort regions with complex structures, and r is a scalar that controls the offset range. is the feature to be rearranged according to the size of the divided region M, ω is the mask used to mark the region, ω d is the mask threshold, which is used to control whether the area is calculated by the residual module or reconstructed using the reference image super-resolution method; ω of the low-resolution image d Set to 0.3, the ω of the reference image d Set to 0.2; after upsampling, the marked area is subjected to high-precision offset calculation and the complex area mask is finely marked, the whole image is divided into simple areas and complex areas, and the location information of each area is recorded; the simple area is input to the residual module for super-resolution reconstruction, and the complex area is input to the feature matching module. The feature matching module is as follows Figure 4As shown, it consists of layer normalization, multi-layer perceptron and self-attention layer, among which the self-attention layer detects the mask value of each regional feature of infrared and visible light images. The mask detection block is pre-trained with the PBVS2023 reference image dataset before the overall network training. After the dataset is masked and marked, the features are extracted and sent to the mask detection module. After the mask value of the visible light image is subtracted from the mask value of the infrared image in the same area, a data group is generated. These data are fitted into a function that approximates a normal distribution; the function is solidified into the mask detection module to complete the pre-training of the mask detection module;

[0049] The mask detection block rearranges the features according to the fitted function, prioritizes the feature areas with higher function values, and abandons the search for the areas in the last quarter of the reference image mask arrangement priority, thereby greatly improving the algorithm's calculation speed and reducing the calculation complexity. If the correlation calculation values ​​of the first three quarters are all low, then the entire reference image area is retrieved; in the feature matching module, Q is the infrared image feature vector of the corresponding multiple, K is the feature vector after downsampling and upsampling of the reference image, and V is the feature vector of the reference image; Q, K, and V are detected by the mask and correspond to Q T , K T 、V T are the feature vectors re-screened and arranged by mask marking, is the feature vector extracted by dilated convolution after downsampling and upsampling of the reference image; With Q T and K T With Q T The correlation of the feature vectors is calculated respectively and the calculation results are weighted by different weights a and b, and the V corresponding to the area with the maximum correlation value is taken. T As the reference image of this area of ​​the infrared image, V T The eigenvector and Q T The feature vector channels of are concatenated and input into the feature alignment module, such as Figure 5 As shown in the figure, convolution block 1 extracts shallow feature information of the image block and inputs it into 3×3 and 5×5 convolution blocks respectively. After further extracting multi-scale information, channel fusion is performed and input into dilated convolution blocks with convolution kernels of 2 and 3 respectively to increase the receptive field. After the information and shallow information extracted by convolution block 1 are input into the deformable convolution, the reference feature with the most appropriate current scaling factor is output. to the feature fusion module, the feature fusion module is as follows Figure 6 As shown in the figure, the feature fusion module inputs the 1× reference feature into convolution blocks 1 and 2, multiplies the feature map by γ and then adds it to β to obtain the new feature The 2× and 4× reference features are also merged with the 4× reference features after the same operation to output the final fused feature. The affine transformation principle is:

[0050]

[0051] Where β and γ are the parameters of scaling and shifting, ⊙ represents pixel-by-pixel multiplication, and ⊕ represents feature fusion. All fused features are concatenated according to the original position encoding information to obtain the super-resolution reconstructed image of the complex area. The simple area is super-reconstructed by the residual module. The residual module is as follows: Figure 7 As shown in , the super-resolution reconstructed images of simple and complex areas are finally merged into a complete high-resolution infrared image. The computational complexity of the feature matching of this network and the traditional Transformer feature matching is as follows Figure 8 As shown, M is the Transformer window size, h and w represent the height and width of the image respectively, h1 and w1 represent the height and width of the complex part of the infrared image, h2 and w2 represent the height and width of the complex part of the visible light image, m is the number of convolutions used in the network, k is the convolution kernel size, C is the input and output channels, and the computational complexity of this method varies with the size of the complex area marked;

[0052] Step 3, training the network model: input the data set in step 1 into the network model constructed in step 2 for training;

[0053] Step 4, design loss function: L1 loss, perceptual loss and adversarial loss are used in complex areas of this network, and L1 loss is used in simple areas. The definitions of L1 loss, perceptual loss and adversarial loss are:

[0054] L rec =||I HR -I SR ||1

[0055]

[0056] The overall loss of complex regions can be expressed as:

[0057] L(θ)=λ rec L rec +λ per L per +λ adv L adv

[0058] The weight coefficient λ of the loss function rec , per , adv are set to 1, 1, and 0.005 respectively, where I SR ,I HRare low-resolution images and high-resolution images in the dataset respectively; by minimizing the loss function generated by the network through reverse gradient conduction, better weight parameters can be obtained.

[0059] Step 5, save the model: After the network training is completed, the weight parameters of the network training need to be saved and solidified. The low-resolution infrared image and the visible light reference image can be input into the network to reconstruct the high-resolution infrared image.

[0060] Among them, the implementation of convolution, activation function, splicing operation, batch normalization, multiple perceptrons, self-attention layer and residual network are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0061] The present invention constructs a super-resolution reconstruction method for infrared images based on reference images. The method can mark simple and complex areas of the image by masking the image, and perform super-resolution reconstruction using a residual network and a reference image super-resolution network respectively. The pre-trained embedded function is used in the reference image super-resolution area to abandon the retrieval of some image areas, thereby reducing the number of parameters of the overall network and improving the reconstruction accuracy of complex areas. Under the same conditions, the feasibility and superiority of the method are further verified by calculating the relevant indicators of the image obtained by the existing method. The relevant indicators of the existing technology and the method proposed by the present invention are compared. Fig. 9 As shown, experimental data show that the method proposed in the present invention is superior to other super-resolution reconstruction methods, has higher peak signal-to-noise ratio and structural similarity than existing methods, and performs well in complex image areas.

[0062] Embodiments of the present invention may be presented as methods, systems, or computer program products. Therefore, the present invention may be in a completely hardware-based form, a completely software-based form, or a combination of hardware and software. In addition, the present invention may also be implemented by a computer program product on a computer-usable storage medium, which may include one or more portions storing computer-usable program code.

[0063] The present invention refers to flowcharts and / or block diagrams of methods, systems, and computer program products related to embodiments of the present invention. Each step or module in these flowcharts and / or block diagrams can be implemented by computer program instructions, and these computer program instructions can be run on a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, so that when the processor of the computer or other programmable data processing device executes these instructions, the functions described in the flowchart or block diagram can be implemented.

[0064] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner. The computer-readable memory may include, but is not limited to, RAM, ROM, EEPROM, or other memory technologies. These instructions enable the program stored in the computer-readable memory to perform the functions specified in the flowchart or block diagram.

[0065] It should be noted that although Fig.10 The system terminal device diagram shown only shows the processor, communication interface and storage, but in the process of system implementation, those skilled in the art should understand that the system may include but is not limited to the above devices; according to specific needs, the system may not need to use Fig.10 All devices in.

[0066] Finally, it should be pointed out that the above description is only a preferred embodiment of the present invention and does not constitute a limitation of the present invention. Although the various embodiments of the present invention have been described in detail, those skilled in the art can still modify the technical solutions described therein or replace some of the technical features therein with equivalents. Any modification, replacement and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for super-resolution reconstruction of infrared images based on a reference image, characterized in that: The following steps are included: Step 1, data set preprocessing: prepare infrared and visible light scene data sets, enhance the infrared image DDE to improve the contrast; adjust the image size after pairing the infrared and visible light images and augment the image data; Step 2, build the network model: the entire network includes a mask marking module, a feature matching module, a feature alignment module, a feature fusion module and a residual module; Mask marking module: divides the image into two parts: simple area and complex area; The reference image feature fusion part includes a feature matching module, a feature alignment module and a feature fusion module; Feature matching module: After arranging the feature vectors of the complex area of ​​the infrared image and the reference image, they are calculated and the feature vector with the maximum value is input into the feature alignment module; Feature alignment module: matches the reference image features of different scaling factors with the low-resolution image positions, and selects the reference image feature variable pairs to output to the feature fusion module; Feature fusion module: fuses the texture features of the reference image into the infrared image to reconstruct a high-resolution image of a complex area; Residual module: reconstructs simple areas with super-resolution; Step 3, training the network model: input the data set prepared in step 1 into the network model constructed in step 2 for training; Step 4, design the loss function: obtain the best reconstructed image effect by minimizing the loss function and save the model parameters; Step 5, save the model: solidify the model parameters. When super-resolution reconstruction of the infrared image is required later, the low-resolution infrared image and the high-resolution visible light image can be input into the network at the same time to obtain a high-resolution infrared image.

2. The method for super-resolution reconstruction of infrared images based on a reference image according to claim 1, characterized in that: In step 1, the data set uses FLIRADAS and LLVIP data sets; some image pairs are selected from them for rough registration.

3. The method for super-resolution reconstruction of infrared images based on reference images according to claim 1, characterized in that: The data augmentation methods in step 1 include image translation, image rotation, and image scaling, and all transformation centers are the image center.

4. The method for super-resolution reconstruction of infrared images based on reference images according to claim 1, characterized in that: In step 2, the mask marking module divides the image and then downsamples it, calculates the offset of the image, calculates the binary mask of each area of ​​the divided image through gumble softmax, inserts the label value of the binary mask into the features of each area of ​​the image, and repeats the previous operation after upsampling the image. The boundary between the simple area and the complex area is determined by comparing the mask difference between the two areas, and the generated mask value is assigned to the feature vector of each area, thereby dividing the image into two parts, the simple area and the complex area. The simple area is trained with a residual network, and the complex area is finely reconstructed by fusion of reference image features.

5. The method for super-resolution reconstruction of infrared images based on reference images according to claim 1, characterized in that: In step 2, the feature matching module has a built-in Transformer reference image matching method, which detects the 1×, 2× and 4× feature vectors corresponding to the infrared image with mask values ​​and the reference image through mask detection, where Q is the infrared image feature vector of the corresponding multiple, K is the feature vector after downsampling and upsampling of the reference image, and V is the feature vector of the reference image; Q, K, V correspond to Q after mask detection T , K T 、V T are the feature vectors re-screened and arranged by mask marking, is the feature vector extracted by dilated convolution after downsampling and upsampling of the reference image; With Q T and K T With Q T The correlation of the feature vectors is calculated respectively and the calculation results are weighted by different weights a and b, and the V corresponding to the area with the maximum correlation value is taken. T As the reference image of the infrared image of this area, V T The eigenvector and Q T The feature vectors are concatenated channel by channel and input into the feature alignment module; The feature alignment module matches the reference image features of different scaling factors with the low-resolution image positions through multi-scale information extraction and dilated convolution, reduces the channels of feature mapping, uses deformable convolution to avoid artifacts in the reconstructed image caused by feature position offset, and selects the most appropriate reference image feature variable pair to output to the feature fusion module; The feature fusion module fuses the 1× reference image texture features into the 1× infrared image through weight parameters β and γ, upsamples the fused image and fuses it with the 2× reference image texture features, then upsamples the fused image again and fuses it with the 4× reference image texture features to reconstruct a high-resolution image of a complex area.

6. The method for super-resolution reconstruction of infrared images based on reference images according to claim 5, characterized in that: The mask detection in step 2 needs to be pre-trained, and the registered infrared visible light image data set is masked and marked. After subtracting the mask value of the visible light image from the mask value of the infrared image in the same area, these data are combined and fitted into a function that approximates a normal distribution. The mask detection calls this function to rearrange the regions of the visible light image in subsequent modules, give priority to the retrieval of visible light images with higher normal distribution values, and set an adaptive adjustable threshold to abandon the subsequent Transformer texture matching retrieval of the regions at both ends of the normal distribution, thereby saving computing resources.

7. The method for super-resolution reconstruction of infrared images based on reference images according to claim 1, characterized in that: The step 4 adopts L1 loss, perceptual loss, and adversarial loss.

8. A terminal device for super-resolution reconstruction of infrared images based on reference images, characterized in that: The invention comprises a processor, a memory, a communication port and a computer program stored in the memory and executed by the processor, wherein when the computer program is executed, the image super-resolution reconstruction method according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium for super-resolution reconstruction of infrared images based on reference images, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute an image super-resolution reconstruction method as claimed in any one of claims 1 to 7.

Images