A method for registering and fusing visible and mid-wave infrared images based on feature coupling

Abstract

The application provides a feature coupling-based visible and medium-wave infrared image registration and fusion method, which comprises the following steps: sub-block division is performed on visible light images and medium-wave infrared images, a correlation correlation matching algorithm is used to extract homonymic points, affine transformation is used to correct the relative offset of the homonymic points, and coarse registration of the images is completed; an improved SIFT feature point matching method is used to extract SIFT feature points of the images after coarse registration, the extracted SIFT feature points are used as registration control points to construct an image triangular net, and a small facet registration model is used to complete accurate registration of the images; a saturation dispersion suppression model is further constructed, the visible light images and the medium-wave infrared images after accurate registration are fused, finally, a color image enhancement method is used to perform enhancement processing on the fused images, and the final images are obtained. The application improves the registration accuracy of the visible light images and the medium-wave infrared images, improves the registration speed, and realizes high-fidelity remote sensing image fusion.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, and specifically relates to a method for registration and fusion of visible and mid-wave infrared images based on feature coupling. Background Technology

[0002] Infrared images are characterized by blurred details, concentrated grayscale distribution, low signal-to-noise ratio, and low contrast, while visible images have high contrast and better reflect the texture details of ground features. In applications, it is often necessary to register, fuse, and compare infrared and visible images of the same area to achieve accurate interpretation of ground features and targets. However, current registration of visible and mid-wave infrared images still faces the following problems: despite using data from the same time phase, there is still significant relative distortion between visible and mid-wave infrared images. For example, there is a large offset between ground features, and the offset direction and distance of ground features vary in different areas, making feature point matching extremely difficult; there is significant deformation between ground features, especially in mountainous areas with uneven terrain, where the deformation is even more severe, making image distortion correction very difficult; visible and mid-wave infrared images are very large, resulting in a large computational burden for image registration, but practical applications have very high requirements for the computational efficiency of image registration. Strict timeliness requirements pose a significant challenge to the design of wide-swath remote sensing image registration methods, requiring a combination of the characteristics of remote sensing images to significantly reduce the computational burden of registration methods. To meet the needs of actual production, it is urgent to develop a new image registration and fusion method to overcome the above problems. Summary of the Invention

[0003] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a visible and mid-wave infrared image registration and fusion method based on feature coupling, which improves the registration accuracy of visible and mid-wave infrared images, while also increasing the registration speed and realizing high-fidelity remote sensing image fusion.

[0004] The technical solution of this invention is:

[0005] A method for registration and fusion of visible and mid-infrared images based on feature coupling includes the following steps:

[0006] (1) Acquire a visible light image and a mid-wave infrared image of the same area, wherein the resolution of the visible light image is higher than that of the mid-wave infrared image;

[0007] (2) Divide the mid-wave infrared image and the visible light image into sub-blocks, obtain the corresponding points of the mid-wave infrared image and the visible light image sub-blocks based on the phase correlation matching algorithm, and use affine transformation to correct the relative distortion of the corresponding points of the mid-wave infrared image and the corresponding points of the visible light image sub-blocks to obtain the mid-wave infrared image after coarse registration.

[0008] (3) SIFT feature point extraction and matching are performed on the coarsely registered mid-wave infrared image and visible light image to obtain the SIFT feature points matched between the coarsely registered mid-wave infrared image and visible light image.

[0009] (4) Using the matched SIFT feature points as registration control points, construct the image triangulation network of the coarsely registered mid-wave infrared image and the visible light image respectively, correct the relative distortion of the triangular surface elements of the coarsely registered mid-wave infrared image and the corresponding triangular surface elements of the visible light image, and obtain the accurately registered mid-wave infrared image.

[0010] (5) After upsampling the precisely registered mid-wave infrared image to the same resolution as the visible light image, the mid-wave infrared image and the visible light image are fused to obtain a fused image;

[0011] (6) The fused image is enhanced using a color image enhancement method to obtain the final fused image.

[0012] Preferably, in step (3), SIFT feature point extraction and matching are performed on the coarsely registered mid-wave infrared image and visible light image, specifically as follows:

[0013] SIFT feature points are extracted from mid-infrared and visible light images based on the SIFT algorithm. SIFT feature points extracted from mid-infrared and visible light images of the same resolution are matched. Each SIFT feature point searches for matching SIFT feature points within a circular region with a radius of R pixels based on the similarity of its SIFT feature vectors.

[0014] Preferably, the value of R is in the range of 3 to 5.

[0015] Preferably, in step (5), the fusion of the mid-wave infrared image and the visible light image specifically includes:

[0016] (51) Obtain the DN value distribution of the mid-wave infrared image and the visible light image respectively between the kth percentile and the (100-k)th percentile;

[0017] (52) Based on the DN value distribution of the visible light image, histogram matching is performed on the mid-wave infrared image to obtain the matched mid-wave infrared image;

[0018] (53) Calculate the ratio of the DN value of the visible light image to the matched mid-wave infrared image for each pixel. If the ratio of the same pixel region is greater than or equal to 1, multiply the DN value of the visible light image by (1+e) to obtain the DN value of the fused pixel region. If the ratio of the same pixel region is less than 1, multiply the DN value of the matched mid-wave infrared image by (1+e) to obtain the DN value of the fused pixel region.

[0019] Preferably, the value of k is in the range of 2 to 5.

[0020] Preferably, in step (6), a color image enhancement method is used to enhance the fused image, specifically including:

[0021] (61) Convert the fused image data from the RGB color space to the IHS color space using the following expression:

[0022]

[0023]

[0024] Where I, H, and S represent brightness, chroma, and saturation, respectively, and R, G, and B represent the image DN value;

[0025] (62) Construct the scaling parameter and transformation parameter, where the expression for the scaling parameter α is:

[0026]

[0027] The expression for the transformation parameter β is:

[0028] β=I-B1*I

[0029] Where B1 represents the convolution kernel, and is set as a B3 spline curve;

[0030] (63) The fused image converted to the IHS color space is transformed based on the constructed scale parameters and transformation parameters, and the expression is:

[0031] I'=α*I+β

[0032] Where I' represents the I value of the transformed fused image.

[0033] Preferably, in step (62), the expression for convolution kernel B1 is:

[0034]

[0035] Preferably, in step (2), the mid-wave infrared image and the visible light image are divided into sub-blocks, specifically as follows:

[0036] After upsampling the mid-wave infrared image of the same area to the same resolution as the visible light image, the image is divided into grids, each grid being M×M pixels in size. Each grid is then expanded outwards by m pixels to ensure seamless stitching between image sub-blocks, resulting in image sub-blocks of size (M+2m)×(M+2m) pixels.

[0037] Preferably, the value of M is in the range of 128 to 1024, and the value of m is 1 / 6 to 1 / 5 of M.

[0038] Preferably, the step of obtaining corresponding points of mid-wave infrared and visible light image sub-blocks based on the phase correlation matching algorithm specifically involves:

[0039] The cross-power spectrum E of the mid-infrared image sub-block and the visible light image sub-block is calculated using the following expression:

[0040]

[0041] Where (u,v) represents the center coordinates of the image sub-block, and F1(u,v) represents the Fourier transform of the visible light image sub-block. The complex conjugate of the Fourier transform of a mid-wave infrared image sub-block. represent The magnitude of the cross-power spectrum; then calculate the inverse Fourier transform F of the cross-power spectrum. -1 (E), Search | F -1 (E)| The center coordinates of the image sub-block corresponding to the maximum value point are the relative offsets between the visible light image sub-block and the mid-wave infrared image sub-block;

[0042] Finally, based on the relative offset between the visible light image sub-block and the mid-wave infrared image sub-block, the corresponding points of the visible light image and the mid-wave infrared image sub-block are obtained.

[0043] The advantages of this invention compared to the prior art are:

[0044] (1) The present invention calculates the overall displacement by phase correlation matching to obtain visible and mid-wave infrared image sub-blocks without relative offset, which is beneficial to the accurate matching and fast search of feature points;

[0045] (2) This invention improves the registration accuracy of visible and mid-wave infrared images by improving the SIFT feature operator, reducing the calculation time and increasing the calculation speed by more than 3 times;

[0046] (3) The present invention adopts a registration model based on small facets, which improves the overall registration accuracy of the image;

[0047] (4) The color image enhancement method proposed in this invention improves the processing effect and processing stability of the linear model, so that the enhanced color image effect reaches the optimal level. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the visible and mid-wave infrared image registration and fusion method based on feature coupling according to the present invention. Detailed Implementation

[0049] The features and advantages of the present invention will become clearer and more explicit through the following detailed description.

[0050] This invention provides a visible and mid-infrared image registration and fusion method based on feature coupling, including a region-division-based image accurate registration method and a high-fidelity remote sensing image fusion method. This method improves the registration accuracy of visible and mid-infrared images, reduces computation time, and achieves optimal color image quality after registration and fusion.

[0051] The flowchart of the visible mid-wave infrared image registration and fusion method of the present invention is as follows: Figure 1 As shown, it includes the following steps:

[0052] (1) Divide the visible and mid-wave infrared images into multiple smaller sub-blocks, transform the complex distortion of the large image into the simple distortion of the small image, and then use affine transformation to correct the simple distortion of the sub-blocks. The corresponding points required for the affine transformation are the corresponding feature points of the mid-wave infrared image and the visible image obtained by the phase correlation matching algorithm.

[0053] Specifically, before image segmentation, the following preprocessing is required: First, the mid-wave infrared image is upsampled to the same resolution as the panchromatic image using bilinear interpolation. Then, the overlapping areas of the visible and mid-wave infrared images are determined using the latitude and longitude positioning information of the remote sensing image.

[0054] Furthermore, the overlapping area of ​​the visible and mid-infrared images is divided into a series of sub-blocks, where each grid is M×M pixels in size. Considering the seamless stitching between the sub-blocks, each image sub-block is extended outwards by m pixels, so the actual size of the sub-block is (M+2m)×(M+2m) pixels.

[0055] The value of M ranges from 128 to 1024, and m is 1 / 6 to 1 / 5 of M.

[0056] Specifically, a phase correlation matching algorithm is used to extract corresponding points from visible and mid-infrared images, performing coarse matching to reduce the time required for subsequent fine matching while improving its accuracy. First, the cross-power spectrum E of the visible and mid-infrared image sub-blocks is calculated, expressed as:

[0057]

[0058] Where F1 is the Fourier transform of the fully visible image, The complex conjugate of the Fourier transform of the upsampled mid-wave infrared image is obtained; then, the inverse Fourier transform F of the cross-power spectrum E is calculated. -1 (E), Search | F -1(E)| Coordinates of the peak point. These coordinates represent the relative offset between the visible and mid-infrared image sub-blocks. Based on the calculated offset, corresponding points are extracted from the sub-blocks.

[0059] Furthermore, affine transformations are performed using the following expression:

[0060]

[0061] Where a1, a2, a3, a4, a5, and a6 are the six parameters of the affine transformation, with a1, a2, a3, and a4 being rotation parameters and a5 and a6 being translation parameters. The x and y coordinates before the affine transformation. These are the x and y coordinates after the affine transformation.

[0062] (2) The SIFT feature point matching method is improved by combining the matching strategy of feature points with the traditional distance constraint strategy. The visible and mid-wave infrared images are further subdivided into smaller sub-blocks. The relative distortion of the small sub-blocks is corrected based on the registration model of small sub-blocks, thereby improving the overall registration accuracy of the image.

[0063] Specifically, by improving the SIFT feature point detection method to extract feature points from visible and mid-infrared images, existing SIFT feature operator matching strategies are mainly aimed at matching large numbers of feature points, such as using the Kd-tree-based BBF matching method to reduce the computational cost of feature point matching. However, these strategies are not suitable for feature point matching in visible and mid-infrared images: First, the imaging mechanisms of visible and mid-infrared images are different, and the number of feature points that can be extracted from each sub-block is relatively small. When the number of feature points is small, the Kd-tree-based BBF matching method is actually more time-consuming than the exhaustive search-based matching method. Considering that the present invention eliminates approximate offsets in image sub-blocks through coarse matching in the initial stage, the search for feature points is limited to a small range. Based on this characteristic, SIFT feature points extracted from visible and mid-wave infrared image sub-blocks are classified according to resolution. For example, for the registration of a 1-meter resolution visible image and a 5-meter resolution mid-wave infrared image, features extracted at resolutions of 5 meters, 10 meters, 20 meters, and 40 meters are divided into different categories. Then, for similar feature points in both visible and mid-wave infrared images, matching feature points are searched within a circular region with a radius of R pixels based on the similarity of the SIFT feature vectors. Preferably, the value of R ranges from 3 to 5.

[0064] Specifically, the extracted matching feature points are used as registration control points (RCPs) to construct a dense image triangular network. Each triangular element is corrected separately, and a polynomial is established to achieve accurate registration between the target image and the reference image.

[0065] Furthermore, by adjusting the feature point extraction threshold and locally adding or deleting feature points, the distribution of RCP is controlled, and a triangular mesh is built on the image. A dense triangular mesh is built for regions with large undulations, while a sparse triangular mesh is built for flat regions. Then, each triangular element is corrected individually. A triangular mesh is built using the large number of dense RCPs after extraction and matching, and then a first-order polynomial is built for each small triangular element:

[0066]

[0067] The six coefficients in the above formula are calculated based on the three vertices of the triangle. Then, the triangle on the target image is corrected to the triangle on the reference image according to this polynomial, thus completing the accurate registration between the target image and the reference image.

[0068] (3) A saturation diffusion suppression model is constructed for the registered visible and mid-wave infrared images. The ratio of the mapped visible light image to the mid-wave infrared image is calculated pixel by pixel. The final fused image is obtained by calculating the ratio.

[0069] Specifically, it includes the following steps:

[0070] (31) Upsample the original mid-wave infrared image to the same spatial resolution as the original visible light image to obtain the upsampled mid-wave infrared image;

[0071] (32) Calculate the kth percentile and (100-k)th percentile of the pixel values ​​of the upsampled mid-wave infrared image and the original visible light image, respectively, as well as the mean and variance of the upsampled mid-wave infrared image and the original visible light image. Let the average of the minimum kth percentile and the maximum (100-k)th percentile be the common mean, and let the maximum variance be the common variance. The value of k ranges from 2 to 5, and is generally taken as 5.

[0072] (33) Linearly map the original visible light image and the upsampled mid-wave infrared image according to the common mean and common variance to obtain the mapped mid-wave infrared image;

[0073] (34) Using visible light images as the dependent variable and mapped mid-wave infrared images as the independent variable, the weighting coefficients of synthesized low-resolution visible light images are calculated using multiple linear regression, and low-resolution visible light images are synthesized by linear weighting of mapped mid-wave infrared images.

[0074] (35) Calculate the ratio of the low-resolution visible light image to the mid-wave infrared image for each pixel, compare the size of each ratio, if the ratio of the low-resolution visible light image to the mid-wave infrared image in the same area is greater than 1, take the low-resolution visible light image component as the main component, if it is less than 1, take the mid-wave infrared image component as the main component, then multiply the DN value of the pixel at that position in the upsampled mid-wave infrared image with the above ratio and superimpose it on the original visible light image to obtain the final fused image.

[0075] (4) Enhancement processing is performed using the color image enhancement method (IEM) to optimize the quality of the fused image and obtain the final image. The image data is converted from the RGB color space to the IHS color space. The scale parameters and transformation parameters of the linear model are modeled. The scale parameters are convolved with the original image. Then the transformation parameters are added to the brightness component of the image to compensate for the loss of image details caused during the enhancement process.

[0076] Specifically, it includes the following steps:

[0077] (41) Perform IHS transformation on the fused image. The original fused image is a single-band image. The original fused image is copied and filled with the three spatial variables R, G, and B to obtain a three-channel grayscale image. Then, perform IHS transformation on the three-channel grayscale image. The specific formula is as follows:

[0078]

[0079]

[0080] Where I, H, and S represent brightness, chroma, and saturation, and R, G, and B represent the image DN value.

[0081] (42) The construction of scale parameters and transformation parameters, and the specific formula derivation are as follows:

[0082]

[0083] From the above formula, it can be seen that the smaller the luminance component, the larger the slope of the corresponding function. This means that the scaling parameter can not only increase the luminance information of the color image, but also widen the luminance difference between pixels in dark areas. At the same time, the maximum value of the image before and after the transformation is the same, which effectively ensures that the image before and after the transformation will not exhibit oversaturation. However, the above function will reduce the luminance difference between pixels in bright areas, i.e., reduce the texture information in bright areas. Therefore, the transformation parameter needs to be added to the image after scaling parameter processing. The transformation parameter β is defined as:

[0084] β=I-B1*I

[0085] In the above formula, I represents the brightness information in the original color image, and B1 represents the convolution kernel. In this invention, B1 is set as a B3 spline curve, and an optimal set of B3 spline curves was selected through experiments to generate transform parameters. By comparing the effects of three B3 spline curves with different smoothing rates, the final selected B3 spline curve is:

[0086]

[0087] (43) Perform the following transformation on the image:

[0088] y = α*x + β

[0089] Where α is the scale parameter, β is the transformation parameter, x represents the I value in the original color image, and y represents the I value after transformation.

[0090] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A method for registration and fusion of visible and mid-infrared images based on feature coupling, characterized in that, Includes the following steps: (1) Acquire a visible light image and a mid-wave infrared image of the same area, wherein the resolution of the visible light image is higher than that of the mid-wave infrared image; (2) Divide the mid-wave infrared image and the visible light image into sub-blocks, obtain the corresponding points of the mid-wave infrared image and the visible light image sub-blocks based on the phase correlation matching algorithm, and use affine transformation to correct the relative distortion of the corresponding points of the mid-wave infrared image and the corresponding points of the visible light image sub-blocks to obtain the mid-wave infrared image after coarse registration. (3) SIFT feature point extraction and matching are performed on the coarsely registered mid-wave infrared image and visible light image to obtain the SIFT feature points matched between the coarsely registered mid-wave infrared image and visible light image. (4) Using the matched SIFT feature points as registration control points, construct the image triangulation network of the coarsely registered mid-wave infrared image and the visible light image respectively, correct the relative distortion of the triangular surface elements of the coarsely registered mid-wave infrared image and the corresponding triangular surface elements of the visible light image, and obtain the accurately registered mid-wave infrared image. (5) After upsampling the precisely registered mid-wave infrared image to the same resolution as the visible light image, the mid-wave infrared image and the visible light image are fused to obtain a fused image; specifically including: (51) Obtain the DN value distribution of the mid-wave infrared image and the visible light image respectively between the kth percentile and the (100-k)th percentile; (52) Based on the DN value distribution of the visible light image, histogram matching is performed on the mid-wave infrared image to obtain the matched mid-wave infrared image; (53) Calculate the ratio of the DN value of the visible light image to the matched mid-wave infrared image for each pixel. If the ratio of the same pixel region is greater than or equal to 1, multiply the DN value of the visible light image by (1+e) to obtain the DN value of the fused pixel region. If the ratio of the same pixel region is less than 1, multiply the DN value of the matched mid-wave infrared image by (1+e) to obtain the DN value of the fused pixel region. (6) The fused image is enhanced using a color image enhancement method to obtain the final fused image.

2. The visible and mid-infrared image registration and fusion method based on feature coupling according to claim 1, characterized in that, In step (3), SIFT feature point extraction and matching are performed on the coarsely registered mid-wave infrared image and visible light image, specifically as follows: SIFT feature points are extracted from mid-infrared and visible light images based on the SIFT algorithm. SIFT feature points extracted from mid-infrared and visible light images of the same resolution are matched. Each SIFT feature point searches for matching SIFT feature points within a circular region with a radius of R pixels based on the similarity of its SIFT feature vectors.

3. The visible and mid-infrared image registration and fusion method based on feature coupling according to claim 2, characterized in that, The value of R ranges from 3 to 5.

4. The visible and mid-infrared image registration and fusion method based on feature coupling according to claim 1, characterized in that, The value of k ranges from 2 to 5.

5. The visible and mid-infrared image registration and fusion method based on feature coupling according to claim 1, characterized in that, In step (6), a color image enhancement method is used to enhance the fused image, specifically including: (61) Convert the fused image data from the RGB color space to the IHS color space using the following expression: Where I, H, and S represent brightness, chroma, and saturation, respectively, and R, G, and B represent the image DN value; (62) Construct the scaling parameter and transformation parameter, where the expression for the scaling parameter α is: The expression for the transformation parameter β is: β=I-B1*I Where B1 represents the convolution kernel, and is set as a B3 spline curve; (63) The fused image converted to the IHS color space is transformed based on the constructed scale parameters and transformation parameters, and the expression is: I'=α*I+β Where I' represents the I value of the transformed fused image.

6. The visible and mid-infrared image registration and fusion method based on feature coupling according to claim 5, characterized in that, In step (62), the expression for convolution kernel B1 is:

7. The visible and mid-infrared image registration and fusion method based on feature coupling according to claim 1, characterized in that, In step (2), the mid-wave infrared image and the visible light image are divided into sub-blocks, specifically as follows: After upsampling the mid-wave infrared image of the same area to the same resolution as the visible light image, the image is divided into grids, each grid being M×M pixels in size. Each grid is then expanded outwards by m pixels to ensure seamless stitching between image sub-blocks, resulting in image sub-blocks of size (M+2m)×(M+2m) pixels.

8. The visible and mid-infrared image registration and fusion method based on feature coupling according to claim 7, characterized in that, The value of M ranges from 128 to 1024, and the value of m ranges from 1 / 6 to 1 / 5 of M.

9. The visible and mid-infrared image registration and fusion method based on feature coupling according to claim 1, characterized in that, The method for obtaining corresponding points of sub-blocks in mid-wave infrared and visible light images based on the phase correlation matching algorithm is as follows: The cross-power spectrum E of the mid-infrared image sub-block and the visible light image sub-block is calculated using the following expression: Where (u,v) represents the center coordinates of the image sub-block, and F1(u,v) represents the Fourier transform of the visible light image sub-block. The complex conjugate of the Fourier transform of a mid-wave infrared image sub-block. represent The magnitude of the cross-power spectrum; then calculate the inverse Fourier transform F of the cross-power spectrum. -1 (E), Search | F -1 (E)| The center coordinates of the image sub-block corresponding to the maximum value point are the relative offsets between the visible light image sub-block and the mid-wave infrared image sub-block; Finally, based on the relative offset between the visible light image sub-block and the mid-wave infrared image sub-block, the corresponding points of the visible light image and the mid-wave infrared image sub-block are obtained.

Images