Infrared image super-resolution reconstruction method based on thermal imager fuzzy kernel calibration

Abstract

The invention provides an infrared image super-resolution reconstruction method based on thermal imager blurring kernel calibration, and the method comprises the following steps: building a multi-circular-hole calibration target image collection system, and collecting a group of target images through a to-be-calibrated thermal infrared imager; for each target image, the position of each circle center is detected, the posture of the thermal imager is solved, and a corresponding high-resolution target image is generated; inputting the actual target image and the generated high-resolution target image into a blurring kernel solving network at the same time to obtain a blurring kernel calibration result of a corresponding region; dividing a to-be-reconstructed low-resolution image into a plurality of image blocks, and inputting the image blocks and the blurring kernels of the corresponding regions into the non-blind super-resolution reconstruction network at the same time to obtain reconstructed image blocks; and carrying out image merging on the reconstructed image blocks to obtain a final super-resolution reconstructed image. The invention provides an infrared image super-resolution reconstruction method based on thermal imager blurring kernel calibration, which can accurately calibrate a spatial inconsistent blurring kernel caused by an optical system so as to improve an infrared image super-resolution reconstruction effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrared image processing, and particularly relates to an infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration. Background Art

[0002] Infrared imaging has the advantages of quasi-all-day and quasi-all-weather operation, and has been increasingly widely used in the fields of security, industry, remote sensing, etc. Due to the complex manufacturing process of infrared detectors and the difficulty in preparing large-size and good-uniformity infrared crystal materials, the yield of large-area and high-resolution focal plane devices is low and the price is expensive; on the other hand, the wavelength of infrared is longer and the diffraction spot is also larger, which restricts the improvement of the resolution of infrared sensors. Therefore, the infrared image super-resolution reconstruction method has important application value.

[0003] At present, there are a large number of infrared image super-resolution reconstruction methods. Although these methods have achieved relatively ideal results on simulated images, when testing actual images, problems such as artifacts or over-blurred images are likely to occur. The main reason is that it is assumed that the blur kernel is globally invariant, and only one blur kernel is estimated for the entire image. However, due to the existence of aberrations, thermal defocusing and other phenomena in the actual infrared optical system, it is not a strictly linear space-invariant system. Therefore, the above methods are still difficult to achieve ideal results for actual scene images. Most of the research on spatially non-uniform blur kernels focuses on motion blur caused by camera shake or object movement, and less attention is paid to the blur caused by the optical system itself. Some deep learning-based methods directly estimate the spatially non-uniform blur kernel from the low-resolution image, but since the blur kernel estimation is an ill-posed problem, it is difficult to achieve ideal results, and it is difficult to obtain accurate blur kernel estimation for the weak texture area and single-edge area of the image. Summary of the Invention

[0004] Aiming at the deficiencies of the above-mentioned prior art, the present invention proposes an infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration, which can accurately estimate the blur kernels at each spatial position of the thermal imager, and thus improve the super-resolution reconstruction effect. In addition, for any infrared imaging device, by using the technology of the present invention, only a group of multi-circular hole target images covering each area of the image needs to be collected to achieve spatially non-uniform blur kernel calibration, with less workload and easy to implement.

[0005] The technical solution adopted by the present invention to achieve the above object is as follows:

[0006] An infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration, comprising the following steps:

[0007] 1) Collect a group of target images, detect the positions of the centers of each circle in each target image, obtain the attitude of the thermal imager, and generate the corresponding high-resolution target image;

[0008] 2) The collected original target image and the generated corresponding high-resolution target image are simultaneously input into the fuzzy kernel solution network to obtain the fuzzy kernel calibration result of the corresponding area;

[0009] 3) Dividing the low-resolution image to be reconstructed into multiple image blocks, inputting the image blocks and the blur kernels of the corresponding regions into a non-blind super-resolution reconstruction network at the same time to obtain reconstructed image blocks;

[0010] 4) Merge the reconstructed image blocks to obtain the final super-resolution reconstructed image.

[0011] The step 1) comprises the following steps:

[0012] 1.1) Establish the image coordinate system and the target coordinate system respectively, and use the OpenCV circle center detection algorithm to calculate the center p of each circle in the image 1 ~p n The pixel coordinates (u 1 ,v 1 )~(u n , v n );

[0013] 1.2) According to the physical size of the target, the corresponding center p is obtained 1 1 ~p ′ n The physical coordinates (x 1 ,y 1 )~(x n ,y n );

[0014] 1.3) Calculate the homography matrix H between pixel coordinates and physical coordinates

[0015]

[0016] 1.4) Traverse the pixel coordinates (u) in the high-resolution image to be obtained HR ,v HR ), calculate the corresponding pixel coordinates (u LR ,v LR ), and calculate (u according to the homography matrix H LR ,v LR ) corresponds to the physical coordinates (x,y)

[0017]

[0018]

[0019] Where, s is the super-resolution ratio;

[0020] 1.5) Determine whether the distances from the physical coordinates (x, y) of each pixel point to the centers p ′ 1 ~p ′ n coordinates are all greater than the radius of the circle. If they are all greater than the radius of the circle, assign the average gray value of the background outside the circle to the corresponding pixel; otherwise, assign the average gray value inside the circle to the corresponding pixel to obtain a high-resolution target image.

[0021] Step 1) further includes the following steps:

[0022] 1.6) Divide the obtained high-resolution image into multiple regions for solving the blur kernel of each region, specifically:

[0023] Select the center of any circle not on the boundary, and select 4 circles with the shortest distance between the centers from the adjacent circles. Use 1.5 times the distance between any two adjacent centers of these 4 circles as the side length to intercept a square region, and each region contains 5 complete circles.

[0024] Step 2) includes the following steps:

[0025] 2.1) Simulate and generate training data for the blur kernel solving network:

[0026] 2.2) Use the training data to train the blur kernel solving network:

[0027] Take the L i loss between the blur kernel k gt output by the blur kernel solving network and the blur kernel k 1 simulated and generated as the loss function for supervised training;

[0028] 2.3) Input the actual target image and the generated high-resolution target image into the trained blur kernel solving network at the same time for network inference to obtain the blur kernel solving results of the corresponding regions.

[0029] Step 2.1) includes the following steps:

[0030] 2.1.1) Randomly generate the attitude angle and focal length of the virtual thermal imager and calculate the homography matrix H t

[0031] H t =K v R rot R el R az

[0032] where R az 、R el and R rotThey are rotation matrices corresponding to the heading, pitch, and roll of the virtual thermal imager, respectively. K v is the internal parameter matrix of the virtual camera;

[0033]

[0034] Among them, f x and f y are the focal ratios, which are obtained by adding a 10% random error to the focal ratio of the thermal imager to be measured. W t ×H t is the resolution of the generated image;

[0035] 2.1.2) Use the homography matrix H t to generate a simulated high-resolution image;

[0036] 2.1.3) Randomly generate an anisotropic Gaussian blur kernel k gt (i,j)

[0037]

[0038] Among them, N is the normalization parameter, C is the spatial coordinate, and Σ is the covariance matrix

[0039]

[0040] Among them, θ is the rotation angle, λ 1 and λ 2 are both eigenvalues;

[0041] 2.1.4) Use the generated blur kernel to blur and downsample the simulated high-resolution image, and add Gaussian random noise to obtain a low-resolution image as training data.

[0042] The blur kernel solving network consists of a shallow feature extraction module, a deep feature extraction module, and a blur kernel reconstruction module connected in sequence, where:

[0043] The shallow feature extraction module extracts low-resolution image features and high-resolution image features respectively, and obtains shallow features through dimension concatenation;

[0044] The deep feature extraction module passes the shallow features through a residual module, a downsampling module, and an upsampling module in sequence to obtain extracted deep features;

[0045] The blur kernel reconstruction module passes the deep features through a first convolutional layer, a global average pooling layer, a second convolutional layer, and a soft threshold activation layer in sequence to obtain a reconstructed blur kernel.

[0046] Step 3) includes the following steps:

[0047] 3.1) Divide the low-resolution image to be reconstructed into multiple overlapping image patches;

[0048] 3.2) Input the image patch and the corresponding blur kernel of the image patch calculated by the blur kernel solving network into the trained non-blind super-resolution reconstruction network simultaneously to obtain the reconstructed image patch.

[0049] Step 4) includes the following steps:

[0050] 4.1) Remove the boundaries of each reconstructed image patch;

[0051] 4.2) Calculate the Euclidean distance from the pixels in the overlapping area to the center of the image patch;

[0052] 4.3) Perform weighted average fusion according to the Euclidean distance.

[0053] An infrared image super-resolution reconstruction system based on thermal imager blur kernel calibration includes:

[0054] A high-resolution target image generation module, which is used to collect a set of target images, detect the positions of the centers of each circle in each target image, obtain the attitude of the thermal imager, and generate the corresponding high-resolution target image;

[0055] A blur kernel calibration module, which is used to input the collected original target image and the generated corresponding high-resolution target image into the blur kernel solving network simultaneously to obtain the blur kernel calibration result of the corresponding area;

[0056] An image reconstruction module, which is used to divide the low-resolution image to be reconstructed into multiple image patches, input the image patches and the blur kernels of the corresponding areas into the non-blind super-resolution reconstruction network simultaneously to obtain the reconstructed image patches;

[0057] An image stitching module, which is used to merge the reconstructed image patches to obtain the final super-resolution reconstructed image.

[0058] The present invention has the following beneficial effects and advantages:

[0059] The infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration of the present invention can calibrate the spatially non-uniform blur caused by the optical system, and the super-resolution reconstruction effect can be significantly improved by using the calibrated blur kernel and the super-resolution reconstruction method of image block division. Description of the Drawings

[0060] Figure 1 It is a schematic diagram of a multi-circular hole calibration target pattern in an embodiment of the present invention;

[0061] Figure 2 It is a schematic diagram of the target image actually captured by the thermal imager in an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of the image coordinate system and physical coordinate system constructed for solving the attitude of the thermal imager in the embodiment of the present invention;

[0063] Figure 4 Schematic diagram of the high / low resolution target image pair intercepted in the embodiment of the present invention;

[0064] Figure 5 Structural diagram of the blur kernel solving network in the embodiment of the present invention;

[0065] Figure 6 Schematic diagram of the blur kernel calibration result of a thermal imager in the embodiment of the present invention;

[0066] Figure 7 Principle block diagram of super-resolution reconstruction in the embodiment of the present invention;

[0067] Figure 8 Comparison chart of the reconstruction results of the method of the present invention and two other methods in the embodiment of the present invention;

[0068] Figure 9 Flow chart of the present invention. Detailed implementation manners

[0069] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0070] The present invention provides an infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration, as Figure 9 shown, which includes the following steps:

[0071] Step S1: Build a target image acquisition system, and use the thermal imager to be calibrated to collect a group of images of a multi-circular hole calibration target.

[0072] Specifically, as Figure 1 shown, the calibration target pattern in this embodiment consists of 13 circular holes arranged regularly. Its main advantages are that it contains gradient information in all directions and can be used for anisotropic blur kernel estimation; the center coordinates can be accurately extracted through the center detection algorithm, and then the camera attitude can be obtained without auxiliary features such as corner points; according to the obtained camera attitude, an ideal high-resolution image can be quickly generated.

[0073] Specifically, the target image acquisition system in this embodiment consists of a black body, a multi-circular hole calibration target, a collimator, a rotating platform, the thermal imager to be calibrated, and a computer. The infrared thermal imager is fixed on the two-axis rotating platform, and the optical axis direction is aligned with the collimator. As Figure 2 shown is the target image actually captured by the thermal imager in this embodiment.

[0074] Further, since the target image cannot cover the entire field of view, the rotation platform is used to drive the thermal imager to perform scanning movements in the horizontal and pitching directions, and a set of target images are collected to achieve full coverage of the field of view area.

[0075] Step S2: For each target image, detect the positions of the centers of the circles in the image, calculate the attitude of the thermal imager, and generate the corresponding high-resolution target image;

[0076] Specifically, the generation of the high-resolution target includes the following steps:

[0077] S201: As Figure 3 shown, in this embodiment, an image coordinate system and a target coordinate system are established, and the center points p 1 ~p 13 in the image are calculated using the OpenCV center point detection algorithm, and the pixel coordinates (u 1 , v 1 )~(u 13 , v 13 ) are obtained;

[0078] S202: According to the physical size of the target, the physical coordinates (x ′ 1 ~p ′ 13 , y 1 , y 1 )~(x 13 , y 13 ) of the corresponding center points p are obtained.

[0079] S203: Calculate the transformation matrix H between the image coordinates and the physical coordinates through the following formula

[0080]

[0081] S204: Traverse the pixel coordinates (u HR , v HR ) in the high-resolution image to be obtained, and calculate its pixel coordinates (u LR , v LR ) in the low-resolution image

[0082]

[0083] Calculate the corresponding physical coordinates (x, y) according to the homography matrix H for (u LR , v LR )

[0084]

[0085] S205: Judge whether the physical coordinates (x, y) to each center point p ′ 1 ~p′ 13 Whether the distances of the coordinates are all greater than the radius of the circle. If they are all greater than the radius of the circle, the corresponding pixel is assigned the average gray value of the background outside the circle; otherwise, the corresponding pixel is assigned the average gray value inside the circle to obtain a high-resolution target image.

[0086] Further, with the center points p 4 , p 5 , p 9 , p 10 as the centers respectively, and 1.5 times the distance between p 4 and p 5 as the side length, square image regions are intercepted. Each region basically contains 5 complete circles to obtain an image pair as shown in Figure 4 for solving the blur kernel of this region.

[0087] Step S3: Input the collected target image and the generated high-resolution target image into the blur kernel solving network simultaneously to obtain the blur kernel calibration result of the target region;

[0088] Specifically, as shown in Figure 5 is the structure diagram of the blur kernel solving network provided in this embodiment. The blur kernel estimation network consists of a shallow feature extraction module, a deep feature extraction module, and a blur kernel reconstruction module. The shallow feature extraction module extracts low-resolution image features and high-resolution image features, and through dimension concatenation, obtains a shallow feature representation. The deep feature extraction module adopts a Unet structure and consists of a residual module, a downsampling module, and an upsampling module to extract deep features. The blur kernel reconstruction module consists of a first convolutional layer, a global average pooling layer, a second convolutional layer, and a soft threshold (softmax) activation layer to reconstruct the blur kernel using deep features.

[0089] Further, the training process of the shown blur kernel solving network includes the following steps:

[0090] S301: Simulate and generate the training data of the blur kernel solving network.

[0091] Specifically, randomly generate the attitude angles and focal lengths of the virtual thermal imager, and calculate the homography matrix according to the formula

[0092] H t = K v R rot R el R az

[0093] where R az , R el and R rotThey are rotation matrices corresponding to the heading, pitch, and roll of the virtual thermal imager respectively. The heading angle and pitch angle are uniformly distributed within (-10°, 10°), and the roll angle is uniformly distributed within (-5°, 5°); K v is the internal parameter matrix of the virtual camera

[0094]

[0095] where f x and f y are the focal ratios, obtained by adding a 10% random error to the focal ratio of the thermal imager to be measured. W t ×H t is the resolution of the generated image.

[0096] Furthermore, using the homography matrix H t , according to steps S204 to S205, a simulated high-resolution image is generated;

[0097] Furthermore, an anisotropic Gaussian blur kernel is randomly generated

[0098]

[0099] where N is the normalization parameter, C is the spatial coordinate, i, j ∈ [-r, r], the size of the blur kernel is (2r + 1) × (2r + 1), and Σ is the covariance matrix

[0100]

[0101] where θ is the rotation angle, λ 1 and λ 2 are both eigenvalues.

[0102] Furthermore, the generated blur kernel is used to blur and downsample the high-resolution image, and Gaussian random noise is added to obtain a low-resolution image.

[0103] S302: Using the simulated generated data, train the network parameters for solving the blur kernel.

[0104] Specifically, taking the L i loss between the blur kernel j gt output by the network and the blur kernel k 1 simulated and generated as the loss function, perform supervised training.

[0105] S303: Input the actual target image output in step 2 and the generated high-resolution target image into the blur kernel solving network simultaneously, perform network inference, and obtain the blur kernel solving result for the corresponding region.

[0106] As Figure 6 shown is the blur kernel of a thermal imager calibrated in this embodiment.

[0107] As shown Figure 7 in the principle block diagram of image super-resolution reconstruction provided in this embodiment. Specifically, it includes two steps S4 and S5.

[0108] Step S4: Divide the low-resolution image to be reconstructed into n image patches, and input the image patches and the corresponding blur kernels of the image patches into the non-blind super-resolution reconstruction network simultaneously to obtain the reconstructed image patches;

[0109] Specifically, step S4 includes the following steps:

[0110] S401: Divide the low-resolution image to be reconstructed by overlapping image patches into n image patches. The size of the image patches is 128×128, with 40 columns overlapping between adjacent left and right image patches and 40 rows overlapping between adjacent upper and lower image patches.

[0111] S402: Input the image patches and the corresponding blur kernels of the image patches into the trained UsrNet super-resolution reconstruction network simultaneously to obtain the reconstructed image patches. Among them, UsrNet can also be replaced by other non-blind super-resolution reconstruction networks.

[0112] Step S5: Obtain the final super-resolution reconstructed image by merging the n reconstructed image patches.

[0113] Specifically, step S5 includes the following steps:

[0114] S501: For each reconstructed 128×128 image patch, remove the boundary and retain the central 120×120 pixels;

[0115] S502: Calculate the Euclidean distance from the pixels in the overlapping area to the center of the image patch;

[0116] S503: Perform weighted average fusion according to the Euclidean distance.

[0117] As shown Figure 8 in the comparison diagram of the super-resolution reconstruction effects of the method in this embodiment and two other methods. Among them, (a) is the Bicubic method, (b) is the MANet+RRDB-SFT method, and (C) is the method provided by the present invention. The edge sharpness of the reconstructed image of the method of the present invention is higher, and the weak texture is also clearer, achieving better results.

Claims

1. An infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration, characterized in that, it includes the following steps: 1) Collect a set of target images, detect the positions of the centers of each circle in each target image, obtain the attitude of the thermal imager, and generate corresponding high-resolution target images; 2) Input the collected original target images and the generated corresponding high-resolution target images into the blur kernel solving network at the same time to obtain the blur kernel calibration results of the corresponding regions; 3) Divide the low-resolution image to be reconstructed into multiple image blocks, input the image blocks and the blur kernels of the corresponding regions into the non-blind super-resolution reconstruction network at the same time to obtain reconstructed image blocks; 4) Merge the reconstructed image blocks to obtain the final super-resolution reconstructed image.

2. The infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration according to claim 1, characterized in that, the step 1) includes the following steps: 1.1) Establish an image coordinate system and a target coordinate system respectively, and use the OpenCV circle center detection algorithm to calculate the pixel coordinates (u 1 ~p n ), v 1 )~(u 1 ), v n ) of each circle center p n ) in the image; 1.2) Obtain the corresponding center point p′ according to the physical size of the target 1 ~p′ n The physical coordinates of (x 1 , y 1 )~(x n , y n ); 1.3) Calculate the homography matrix H between pixel coordinates and physical coordinates 1.4) Traverse the pixel coordinates (u HR , v HR ) in the high-resolution image to be obtained, calculate the corresponding pixel coordinates (u LR , v LR ) in the low-resolution image, and calculate the physical coordinates (x, y) corresponding to (u LR , v LR ) according to the homography matrix H where s is the super-resolution magnification; 1.5) Determine whether the distances from the physical coordinates (x, y) of each pixel point to the centers of the circles p ′ 1 ~p ′ n coordinates are all greater than the radius of the circle. If they are all greater than the radius of the circle, the corresponding pixel is assigned the average gray value of the background outside the circle; otherwise, the corresponding pixel is assigned the average gray value inside the circle, and a high-resolution target image is obtained.

3. The infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration according to claim 2, characterized in that, the step 1) further includes the following steps: 1.6) Divide the obtained high-resolution image into multiple regions for solving the blur kernel of each region, specifically: Select the center of any circle not on the boundary as the center, select 4 circles with the shortest distance between the center and the center among the adjacent circles, and use 1.5 times the distance between any two adjacent centers of the 4 circles as the side length to intercept a square region, and each region contains 5 complete circles.

4. The infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration according to claim 1, characterized in that, the step 2) includes the following steps: 2.1) Simulate and generate the training data of the blur kernel solving network: 2.2) Use the training data to train the blur kernel solving network: The blurred kernel k output by the blurred kernel solving network i and the blurred kernel k generated by simulation gt The L 1 loss is used as the loss function for supervised training; 2.3) Input the actual target image and the generated high-resolution target image into the trained blur kernel solving network at the same time for network inference to obtain the blur kernel solving results of the corresponding regions.

5. The infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration according to claim 4, characterized in that, the step 2.1) includes the following steps: 2.1.1) Randomly generate the attitude angle and focal length of the virtual thermal imager, and calculate the homography matrix H t H t = K v R rot R el R az Among them, R az , R el and R rot are the rotation matrices corresponding to the three attitude angles of heading, pitch, and roll of the virtual thermal imager, and K v is the internal parameter matrix of the virtual camera; Among them, f x , f y is the focal ratio, obtained by adding a 10% random error to the focal ratio of the thermal imager under test, and W t ×H t is the resolution of the generated image; 2.1.2) Using the homography matrix H t , generate a simulated high-resolution image; 2.1.3) Randomly generate an anisotropic Gaussian blur kernel k gt (i,j) where N is the normalization parameter, C is the spatial coordinate, and Σ is the covariance matrix where θ is the rotation angle, λ 1 and λ 2 are both eigenvalues; 2.1.4) Use the generated blur kernel to blur and downsample the simulated high-resolution image, and add Gaussian random noise to obtain the low-resolution image as the training data.

6. The infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration according to claim 1, characterized in that, the blur kernel solving network is composed of a shallow feature extraction module, a deep feature extraction module and a blur kernel reconstruction module connected in sequence, where: The shallow feature extraction module extracts the low-resolution image features and high-resolution image features respectively, and obtains shallow features through dimension splicing; The deep feature extraction module passes the shallow features through the residual module, the downsampling module and the upsampling module in sequence to obtain the extracted deep features; The blur kernel reconstruction module sequentially passes the deep features through a first convolutional layer, a global average pooling layer, a second convolutional layer, and a soft threshold activation layer to obtain a reconstructed blur kernel.

7. An infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration according to claim 1, wherein, step 3) includes the following steps: 3.1) Divide the low-resolution image to be reconstructed into multiple image patches by using overlapping image patch division; 3.2) Input the image patch and the blur kernel corresponding to the image patch calculated by the blur kernel solving network into the trained non-blind super-resolution reconstruction network at the same time to obtain a reconstructed image patch.

8. An infrared image super-resolution reconstruction method based on thermal imager blur kernel calibration according to claim 1, wherein, step 4) includes the following steps: 4.1) Remove the boundaries of each reconstructed image patch; 4.2) Calculate the Euclidean distance from the pixels in the overlapping area to the center of the image patch; 4.3) Perform weighted average fusion according to the Euclidean distance.

9. An infrared image super-resolution reconstruction system based on thermal imager blur kernel calibration, wherein, it includes: A high-resolution target image generation module, configured to collect a set of target images, detect the positions of the centers of each circle in each target image, obtain the attitude of the thermal imager, and generate a corresponding high-resolution target image; A blur kernel calibration module, configured to input the collected original target image and the generated corresponding high-resolution target image into the blur kernel solving network at the same time to obtain the blur kernel calibration result of the corresponding area; An image reconstruction module, configured to divide the low-resolution image to be reconstructed into multiple image patches, and input the image patches and the blur kernels of the corresponding areas into the non-blind super-resolution reconstruction network to obtain reconstructed image patches; An image stitching module, configured to merge the reconstructed image patches to obtain the final super-resolution reconstructed image.

Images