Multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning

Abstract

The invention discloses a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning, and belongs to the field of remote sensing image processing. The method comprises the following steps of: extracting feature correlation by using a multi-stage network based on an image pyramid so as to realize multi-scale prediction on a feature level; a method of respectively extracting and fusing global features and local features is used to guarantee effective improvement of registration precision; a Transform module is used for carrying out correlation analysis on the features of the target image and the reference image so as to realize more remote and more accurate feature correlation capture; and a detail feature extraction mechanism based on discrete detection of a feature region is used to remove redundant information, so that the registration precision can be improved. According to the method, a multi-stage feature extraction network and a Transform module are adopted to extract image correlation features, and on the basis of discrete feature detection, effective local feature regions are selected and spliced while abnormal feature regions are removed, so that fusion of local high-resolution features and global low-resolution features is realized.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing, and relates to a remote sensing image matching method, in particular to a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning. Background Art

[0002] Infrared remote sensing images have extremely high application value in all-weather remote sensing observations. In order to achieve change detection in the overlapping area or obtain a larger field-of-view imaging result, it is necessary to register multi-source infrared remote sensing images from different perspectives. However, currently, traditional image stitching methods mainly rely on geometric features, and feature detection is the key factor affecting the stitching performance. In recent years, such methods have become increasingly dependent on complex geometric features, resulting in a significant increase in algorithm complexity. At the same time, these methods are difficult to adapt to usage scenarios lacking texture features such as infrared remote sensing imaging.

[0003] In recent years, deep learning methods have developed rapidly. Based on convolutional neural network (CNN) technology, they directly extract high-level semantic features of images instead of extracting existing geometric features, thus solving the problem of difficult feature extraction in infrared remote sensing images to a certain extent. This method avoids some problems existing in traditional stitching and fusion methods and is robust to multi-source heterogeneous remote sensing images. CNN realizes unsupervised alignment of images by extracting image features and regressing to obtain a homography matrix. Currently, deep learning methods adaptively learn in a supervised, weakly supervised, or unsupervised manner in a data-driven mode and show great potential in visual tasks such as optical flow estimation and homography estimation.

[0004] Infrared remote sensing images often have low resolution, making it difficult to extract effective feature points, and there are a large number of low-texture areas. Moreover, there is an imaging parallax between clouds and surface scenery in the images, and changes in illumination angle and observation angle often also cause subtle changes in the shadows and shapes of surface scenery. Therefore, it is necessary to weaken the influence of parallax as much as possible to improve the feature alignment and registration accuracy of multi-level infrared remote sensing images. Summary of the Invention

[0005] The object of the present invention is to meet the registration requirements of infrared remote sensing images, apply unsupervised deep learning to the registration of infrared remote sensing images, and propose a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning. This method uses a multi-stage progressive homography estimation network model. In each stage, using the global and local features distorted by the previous stage, it estimates the vertex motion residuals of the previous stage, and improves the global homography estimation accuracy from coarse to fine, thereby achieving an improvement in registration accuracy. Finally, based on discrete feature detection, redundant information is removed to realize the fusion of local high-resolution features and global low-resolution features.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning, the method comprising the following steps:

[0008] Step 1: Use a multi-stage network based on an image pyramid to extract feature correlations to achieve multi-scale prediction at the feature level;

[0009] Step 2: On the basis of Step 1, use a method of separately extracting and fusing global features and local features to effectively improve the registration accuracy;

[0010] Step 3: After processing the features in Step 2, use a Transformer module to perform correlation analysis on the features of the target image and the reference image to achieve more distant and more accurate capture of feature correlations;

[0011] Step 4: Use a detailed feature extraction mechanism based on discrete detection of feature regions to remove redundant information and improve the registration accuracy.

[0012] Further, in Step 1, the use of a multi-stage network based on an image pyramid to extract feature correlations is specifically as follows:

[0013] Step 1-1: For two or more input infrared remote sensing images, use a convolutional neural network layer with shared weights to extract features and downsample. The network structure uses the YOLO-V8 backbone network, which is composed of a basic convolutional module and a c2f module, and uses strided convolution to achieve multi-scale downsampling of the image;

[0014] Step 1-2: The multi-stage network extracts image features at three levels of the original resolution, 1 / 2 resolution, and 1 / 8 resolution respectively;

[0015] Step 1-3: Use the image features extracted at each level to estimate the homography step by step, and transfer the estimated upper-level homography to the lower level to gradually improve the accuracy of homography estimation, so as to achieve coarse-to-fine homography prediction.

[0016] Further, in Step 2, the global features are obtained through the top layer of the image pyramid, specifically as follows:

[0017] The image features at each level are reprocessed through the residual network after being distorted based on the upper-level homography matrix, and finally the top-level features are obtained, and the resolution and scale of the global features at each level are the same; the local features are obtained by splicing after performing a block extraction operation on the image features at each level, and the scales of the local features at each level are the same, but the resolution increases gradually.

[0018] Further, in step three, the Transformer module is used to perform correlation analysis on the features of the target image and the reference image to achieve more accurate capture of feature correlations at a greater distance; specifically:

[0019] Step three-one: For global features, overall feature map self-attention analysis is adopted, and the Transformer module is used to capture global correlations under large baselines to ensure the correlation extraction performance under large baselines; for local features, the Swim-Transformer module is used, with the window size set to the block size and the depth being 1;

[0020] Step three-two: After using the multi-stage network to extract features at all levels and calculate correlations, a regression network is designed, which is composed of a convolutional layer and a fully connected layer, to predict the displacements of four vertices for determining the homography; except for the top-level features, correlations are extracted for the warped target features and reference features at each level of the multi-stage network, that is, only the residual offset Δ k is predicted, rather than the complete offset;

[0021] Step three-three: The target image is warped by solving the homography calculated from the final offset to achieve rough alignment of the image; the homography estimation ability at each stage is unsupervised trained using the warped image and the reference image obtained at each level.

[0022] Further, in step three-two, the formula for the residual offset Δ k is as follows:

[0023]

[0024] In the formula, A is the target image, B is the reference image, DLT is the homography operation, W is the warping of the image using the homography, is the operation of the vertex residual offset between the reference image feature and the warped target image feature, k is the current stage number, Δ k is the residual offset corresponding to the current stage, is the sum of all residual offsets before the k-th stage.

[0025] Further, in step three-three, the final offset is calculated as follows:

[0026] Δ Fin = Δ 1 + Δ 2 + Δ 3

[0027] In the formula, Δ Fin is the final offset.

[0028] Further, in step four, a detailed feature extraction mechanism based on discrete detection of feature regions is used to remove redundant information and improve the registration accuracy. Specifically:

[0029] Step four-one: Provide a feature detection head, which is composed of N + 3 convolutional layers. Among them, the first N layers perform N downsamplings on the high-resolution feature map of the current stage, the middle two layers extract features, and the last layer is a 1×1 convolutional layer for confidence regression calculation of strong features to obtain a confidence map of B×1×H / 2 N ×W / 2 N The value at each position in the confidence map represents the credibility of the corresponding strong feature region at that position;

[0030] Step four-two: In the confidence map, output the coordinates of the top m×m points with the highest confidence, and restore them to the original resolution to obtain the center point coordinates of the strong feature region. The formula is as follows:

[0031]

[0032] In the formula, x i , y i are the position coordinates of the feature points at the downsampled resolution, x i ′, y i ′ are the position coordinates of the feature points at the original resolution, b lth is the width of the cropped block, and H and W are the length and width of the feature map respectively;

[0033] Step four-three: After obtaining the center point coordinates of the strong feature region based on feature detection, construct a position mask of the strong feature region with the center point coordinates of the strong feature region as the center and the width of the cropped block as the radius, and perform cropping extraction on the high-resolution feature map;

[0034] Step four-four: Perform position encoding on each pixel in the high-resolution feature map;

[0035] Step four-five: Generate an absolute position information map with a scale of B×2×H×W. Among them, 2 represents the information of two channels, and the row and column coordinates of the corresponding pixels are stored in the two channels respectively, and the coordinates are normalized;

[0036] Step four-six: After being processed by the convolutional layer, generate a position encoding with a scale of B×C×H×W and add it to the feature of the original resolution to achieve the addition of position information; where C is the number of feature channels;

[0037] Step four-seven: Use the position mask of the strong feature region to segment and crop the original resolution feature added with the position encoding to obtain m×m image blocks with a scale of B×C×H b ×W b of, where Hb and W b are the length and width of the cropped image block, respectively, with a size of b lth ×2;

[0038] Step Four-Eight: Recombine the image blocks through stitching to obtain a new local feature map with a scale of B×C×mH b ×mW b of the new local feature map.

[0039] The beneficial effects of the present invention compared with the prior art are as follows: Based on the feature alignment and registration of multi-source infrared remote sensing images, the present invention proposes a method for feature alignment and registration of multi-source infrared remote sensing images based on unsupervised learning. This method uses a multi-stage feature extraction network and a Transformer module to extract image correlation features, ensuring the registration performance under large baselines; secondly, based on discrete feature detection, while removing abnormal feature regions, it selects and stitches effective local feature regions to realize the fusion of local high-resolution features and global low-resolution features. The specific innovations are as follows:

[0040] (1) Designed a multi-stage feature extraction and progressive alignment network; utilized multi-scale features, compatible with cascade estimation and cyclic estimation, effectively improving the registration accuracy and realizing unsupervised image distortion.

[0041] (2) Proposed a detailed feature supplement mechanism; based on discrete feature region detection, while removing low-texture regions, it performs local block sampling and recombination on strong feature regions at each stage, realizing the supplement and fusion of high-resolution detailed features on the basis of global features, reducing the computational amount, and effectively improving the registration accuracy.

[0042] (3) Used a method of separately extracting and fusing global features and local features to ensure the effective improvement of registration accuracy.

[0043] (4) Used the Transformer module to perform correlation analysis on the features of the target image and the reference image, making full use of global features and local features to achieve more distant and more accurate feature correlation capture.

[0044] (5) Based on discrete feature detection, while removing abnormal feature regions, it selects and stitches effective local feature regions, thereby realizing the fusion of local high-resolution features and global low-resolution features. Description of the Drawings

[0045] Figure 1 is a schematic flow chart of the method for feature alignment and registration of multi-source infrared remote sensing images based on unsupervised learning of the present invention;

[0046] Figure 2 is a schematic diagram of a multi-stage progressive homography estimation network;

[0047] Figure 3 It is a schematic diagram of the structure of the feature correlation calculation module, where: Figure 3 (a) It is a schematic diagram of the global correlation obtained by processing the global features using the Transformer module; Figure 3 (b) It is a schematic diagram of the local correlation obtained by processing the local features using the Swim-Transformer module after obtaining the local features by discrete feature detection;

[0048] Figure 4 It is a schematic diagram of the high-resolution feature discrete detection process. Specific implementation manner

[0049] As Figure 1 shown, this implementation manner discloses a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning, and the method includes the following steps:

[0050] Step 1: Use a multi-stage network based on an image pyramid to extract feature correlations to achieve multi-scale prediction at the feature level;

[0051] The extraction of feature correlations using a multi-stage network based on an image pyramid is specifically as follows:

[0052] Step 1-1: For two or more input infrared remote sensing images, use a convolutional neural network (CNN) layer with shared weights for feature extraction and downsampling. The network structure uses the YOLO-V8 backbone network, which is composed of a basic convolutional module and a c2f (CSP Bottleneck with 2 Convolutions) module. Use strided convolution to achieve multi-scale downsampling of the image, so as to extract the feature information of the image at different scales and provide a high-quality feature representation for subsequent image registration;

[0053] Step 1-2: The multi-stage network extracts image features at three levels of the original resolution, 1 / 2 resolution, and 1 / 8 resolution respectively;

[0054] Step 1-3: Use the image features extracted at each layer to estimate the homography step by step, and transfer the estimated upper-layer homography to the lower layer to gradually improve the accuracy of homography estimation, so as to achieve coarse-to-fine homography prediction.

[0055] Step 2: On the basis of Step 1, use a method of separately extracting and fusing global features and local features to effectively improve the registration accuracy;

[0056] The global features are obtained from the top layer of the image pyramid, specifically as follows:

[0057] After the image features at all levels are distorted based on the upper - layer homography matrix, they are processed again through the residual network, and finally the top - layer features are obtained. In Figure 2 the red line and the green line are used to represent the feature flow, and the resolution and scale of the global features at each level are the same; the local features are obtained by splicing after performing a block extraction operation on the image features at each level. The scales of the local features at each level are the same, but the resolution increases gradually.

[0058] The present invention adopts a method of separately extracting and fusing global features and local features to combine the advantages of global low - resolution features and local high - resolution features.

[0059] Step 3: After processing the features in Step 2, use the Transformer module to perform correlation analysis on the features of the target image and the reference image to achieve more accurate feature correlation capture at a greater distance;

[0060] The use of the Transformer module to perform correlation analysis on the features of the target image and the reference image to achieve more accurate feature correlation capture at a greater distance; specifically:

[0061] Step 3 - 1: To meet the image stitching requirements under a large baseline and improve the estimation accuracy, and achieve more accurate feature correlation capture at a greater distance, use the Transformer module to perform correlation analysis on the features of the target image and the reference image. As Figure 3 shown, for the global features, use the self - attention analysis of the overall feature map, and capture the global correlation under a large baseline through the Transformer module to ensure the performance of correlation extraction under a large baseline; for the local features, because the method of block extraction and recombination is used, which destroys the spatial continuity of the features, so when processing the local features, use the Swim - Transformer module, set the window size to the block size, and the depth is 1, which is equivalent to calculating the attention only within each local block and not across blocks, thus realizing independent correlation analysis for each;

[0062] Step 3 - 2: After extracting the features at all levels using the multi - stage network and calculating the correlation, design a regression network, which is composed of a convolutional layer and a fully - connected layer, to predict the four vertex displacements for determining the homography; except for the top - layer features, at each level of the multi - stage network, perform correlation extraction on the distorted target features and reference features, that is, only predict the residual offset Δ k from the previous level, rather than predicting the complete offset;

[0063] The formula for the residual offset Δ k is as follows:

[0064]

[0065] Wherein, A is the target image, B is the reference image, DLT is the homography operation, W is the image distortion using the homography, is the operation of the vertex residual offset between the reference image feature and the feature of the distorted target image, k is the current stage number, Δ k is the residual offset corresponding to the current stage, is the sum of all residual offsets before the k-th stage.

[0066] Step Three: Distort the target image using the homography obtained by solving the final offset to achieve the rough alignment of the image; the homography estimation ability of each stage is unsupervised trained using the distorted image and the reference image obtained at each level;

[0067] The calculation of the final offset is as follows:

[0068] Δ Fin = Δ 1 + Δ 2 + Δ 3

[0069] Wherein, Δ Fin is the final offset.

[0070] Step Four: Use the detailed feature extraction mechanism based on discrete detection of feature regions to remove redundant information and improve the registration accuracy;

[0071] The use of the detailed feature extraction mechanism based on discrete detection of feature regions to remove redundant information and improve the registration accuracy; specifically:

[0072] Step Four One: Provide a feature detection head, which is composed of N + 3 convolutional layers, as Figure 4 shown, where the first N layers perform N downsamplings on the high-resolution feature map of the current stage, the middle two layers extract features, and the last layer is a 1×1 convolutional layer for confidence regression calculation of strong features to obtain a confidence map of B×1×H / 2 N ×W / 2 N The value at each position in the confidence map represents the credibility of the corresponding strong feature region at that position. The purpose of N downsamplings is to expand the receptive field while making the feature regions as discrete as possible at the original resolution without overlapping regions;

[0073] Step Four Two: In the confidence map, output the coordinates of the top m×m points with the highest confidence and restore them to the original resolution to obtain the center point coordinates of the strong feature regions. The formula is as follows:

[0074]

[0075] Wherein, x i , yi are the position coordinates of feature points at the downsampled resolution, where x i ′, y i ′ are the position coordinates of feature points at the original resolution, b lth is the width of the cropped block, and H and W are the length and width of the feature map respectively;

[0076] Step Four Three: After obtaining the center point coordinates of the strong feature region based on feature detection, construct a position mask for the strong feature region with the center point coordinates of the strong feature region as the center and the width of the cropped block as the radius, and perform cropped block extraction on the high-resolution feature map;

[0077] Step Four Four: Perform position encoding on each pixel in the high-resolution feature map. Since the operations of cropping blocks and re-stitching will destroy the coordinate coherence of the original feature map, in order for the features to still contain the coordinate information at the original resolution after recombination, it is necessary to perform position encoding on each pixel in the high-resolution feature map;

[0078] Step Four Five: Generate an absolute position information map with a scale of B×2×H×W, where 2 represents the information of two channels, and the row and column coordinates of the corresponding pixels are stored in the two channels respectively, and the coordinates are normalized;

[0079] Step Four Six: After being processed by the convolutional layer, generate a position encoding with a scale of B×C×H×W and add it to the feature at the original resolution to achieve the addition of position information; where C is the number of feature channels;

[0080] Step Four Seven: Use the position mask of the strong feature region to segment and crop the feature at the original resolution with the added position encoding to obtain m×m image blocks with a scale of B×C×H b ×W b where H b and W b are the length and width of the cropped image respectively, and the size is b lth ×2;

[0081] Step Four Eight: Recombine the image blocks through stitching to obtain a new local feature map with a scale of B×C×mH b ×mW b . Through the process of segmentation and cropping and stitching recombination, while maintaining the original resolution, the size of the feature map is greatly reduced, redundant information is removed and the influence of low-texture feature regions is rejected, reducing the subsequent calculation amount and contributing to the improvement of the registration accuracy.

[0082] The present invention proposes a detailed feature extraction mechanism based on discrete detection of feature regions, which detects and screens strong texture feature regions in high resolution, so as to crop and stitch effective features, remove redundant features and significantly reduce the size of the feature map, thereby better realizing the fusion with global features.

[0083] The above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes, shall be covered by the protection scope of the present invention.

Claims

1. A multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning, characterized by: The method comprises the following steps: Step 1: Extract feature correlations using a multi-stage network based on an image pyramid to achieve multi-scale predictions at the feature level; Step 2: Based on step 1, use the method of extracting and fusing global features and local features separately to ensure the effective improvement of registration accuracy; Step 3: After processing the features in step 2, use the Transformer module to perform correlation analysis on the features of the target image and the reference image to achieve longer-range and more accurate feature correlation capture; Step 4: Use a detail feature extraction mechanism based on discrete detection of feature regions to remove redundant information and improve registration accuracy.

2. The multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning according to claim 1, characterized in that: In step 1, the feature correlation is extracted using a multi-stage network based on an image pyramid; Specifically: Step 1: For two or more input infrared remote sensing images, a convolutional neural network layer with shared weights is used for feature extraction and downsampling. The network structure adopts the YOLO-V8 backbone network, which is composed of a basic convolution module and a c2f module, and uses strided convolution to achieve multi-scale downsampling of images; Step 1 and 2: The multi-stage network extracts image features at three levels: original resolution, 1 / 2 resolution, and 1 / 8 resolution; Step 1-3: Use the image features extracted from each layer to estimate the homography step by step, and pass the estimated upper-layer homography to the lower layer to gradually improve the accuracy of the homography estimation, thereby achieving homography prediction from coarse to fine.

3. The multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning according to claim 1, characterized in that: In step 2, the global features are obtained through the top layer of the image pyramid, specifically: After being distorted based on the upper homography matrix, the image features at each level are reprocessed through the residual network to finally obtain the top-level features, and the resolution and scale of the global features at each level are the same; the local features are obtained by performing block operations on the image features at each level and then splicing them. The scale of the local features at each level is the same, but the resolution increases step by step.

4. The multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning according to claim 1, characterized in that: In step 3, the Transformer module is used to perform correlation analysis on the features of the target image and the reference image to achieve longer-range and more accurate feature correlation capture; Specifically: Step 31: The global features use the self-attention analysis of the overall feature map, and the Transformer module is used to capture the global correlation under the large baseline to ensure the correlation extraction performance under the large baseline; the local features use the Swim-Transformer module, the window size is set to the block size, and the depth is 1; Step 32: After extracting features at each level using a multi-stage network and calculating correlations, a regression network consisting of a convolutional layer and a fully connected layer is designed to predict the four vertex displacements used to determine the homography; in addition to the top-level features, each level of the multi-stage network extracts correlations between the distorted target features and the reference features, that is, only predicts the residual offset Δ from the previous level k , without predicting the full offset; Step 33: Warp the target image by solving the homography obtained by the final offset calculation to achieve rough alignment of the image; The homography estimation capability of each stage is trained unsupervised using the warped and reference images obtained at each stage.

5. The multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning according to claim 4 is characterized in that: In step 32, the residual offset Δ k The calculation formula is as follows: In the formula, A is the target image, B is the reference image, DLT is the homography operation, To warp the image using homography, is the calculation of vertex residual offset between reference image features and distorted target image features, k is the current stage number, Δ k is the residual offset corresponding to the current stage, is the sum of all residual offsets before the kth stage.

6. The multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning according to claim 4 or 5, characterized in that: In step 33, the final offset is calculated as follows: D Fin =Δ1+Δ2+Δ3 In the formula, Δ Fin is the final offset.

7. The multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning according to claim 6 is characterized in that: In step 4, the detailed feature extraction mechanism based on discrete detection of feature regions is used to remove redundant information and improve the registration accuracy; specifically: Step 41: Provide a feature detection head, which is composed of N+3 convolutional layers. The first N layers downsample the high-resolution feature map of the current stage N times, the middle two layers extract features, and the last layer is a 1×1 convolutional layer, which is used to calculate the confidence regression of strong features to obtain B×1×H / 2 N ×W / 2 N A confidence map, wherein the value of each position in the confidence map represents the credibility of the strong feature area corresponding to the position; Step 42: In the confidence map, output the coordinates of the first m×m points with the highest confidence, and restore them to the original resolution to obtain the coordinates of the center point of the strong feature area. The formula is as follows: In the formula, x i ,y i is the feature point position coordinate at the downsampled resolution, x i ′,y i ′ is the position coordinate of the feature point at the original resolution, b lth is the width of the block, H and W are the length and width of the feature map respectively; Step 43: After obtaining the coordinates of the center point of the strong feature area based on feature detection, a position mask of the strong feature area is constructed with the coordinates of the center point of the strong feature area as the center and the width of the cutout as the radius, and the high-resolution feature map is subjected to cutout extraction; Step 44: performing position encoding on each pixel in the high-resolution feature map; Step 45: Generate an absolute position information map with a scale of B×2×H×W, where 2 represents information of two channels, and the row and column coordinates of the corresponding pixels are stored in the two channels respectively, and the coordinates are normalized; Step 46: After processing by the convolutional layer, a position code of scale B×C×H×W is generated and added to the original resolution feature to add position information; where C is the number of feature channels; Step 47: Use the strong feature area position mask to segment and block the original resolution features with position encoding added to obtain m×m blocks of scale B×C×H b ×W b image block, where H b and W b are the length and width of the cut-out image, respectively, and the size is b lth ×2; Step 48: Reassemble the image blocks by splicing to obtain a scale of B×C×mH b ×mW b The new local feature map.

Images