Multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning

Abstract

The application discloses a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning, and belongs to the field of remote sensing image processing.The method is as follows: a multi-stage network based on an image pyramid is used to extract feature correlation, so that multi-scale prediction is realized at a feature level; a method of respectively extracting and fusing global features and local features is used, so that the effective improvement of registration accuracy is guaranteed; a Transform module is used to analyze the correlation of features of a target image and a reference image, so that more distant and more accurate feature correlation capture is realized; a detailed feature extraction mechanism based on discrete detection of feature regions is used, so that redundant information is removed, and the improvement of registration accuracy is helpful.The method uses a multi-stage feature extraction network and a Transform module to extract image correlation features, and based on discrete feature detection, while removing abnormal feature regions, effective local feature regions are selected and spliced, so that local high-resolution features and global low-resolution features are fused.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing image processing, and relates to a remote sensing image matching method, in particular to a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning. BACKGROUND

[0002] Infrared remote sensing images have extremely high application value in all-weather remote sensing observation. In order to realize change detection in the overlapping area or obtain imaging results of a larger field of view, it is necessary to register multi-source infrared remote sensing images of different angles. However, the current traditional image stitching method mainly depends on geometric features, and feature detection is a key factor affecting the stitching performance. In recent years, such methods increasingly rely on complex geometric features, resulting in a significant increase in algorithm complexity. At the same time, these methods are difficult to adapt to use scenarios such as infrared remote sensing imaging which lack texture features.

[0003] In recent years, deep learning methods have developed rapidly, which directly extract high-level semantic features of images based on convolutional neural network (CNN) technology, rather than extracting existing geometric features, thereby to a certain extent solving the problem of feature extraction difficulty in infrared remote sensing images. This method avoids some problems existing in traditional stitching and fusion methods, and has robustness for multi-source heterogeneous remote sensing images. CNN extracts image features and regresses homography matrix to realize unsupervised alignment of images. At present, deep learning methods adaptively learn in a supervised, weakly supervised or unsupervised manner in a data-driven mode, and have shown great potential in visual tasks such as optical flow estimation and homography estimation.

[0004] Infrared remote sensing images often have low resolution, making it difficult to extract effective feature points, and there are a large number of low-texture regions. In addition, there is an imaging parallax between the clouds and the ground scene in the image, and the changes in the light angle and the observation angle often also cause slight changes in the shadow and shape of the ground scene. Therefore, it is necessary to weaken the parallax influence as much as possible to improve the feature alignment and registration accuracy of multi-level infrared remote sensing images. SUMMARY

[0005] The purpose of the present application is to use unsupervised deep learning for infrared remote sensing image registration in view of the registration requirements of infrared remote sensing images, and to propose a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning. The method uses a multi-stage progressive homography estimation network model, and in each stage, the vertex motion residual of the previous stage is estimated using the global and local features distorted by the previous stage, and the global homography estimation accuracy is improved from coarse to fine, thereby improving the registration accuracy. Finally, based on discrete feature detection, redundant information is removed to realize the fusion of local high-resolution features and global low-resolution features.

[0006] To achieve the above object, the technical scheme adopted by the present application is:

[0007] The method comprises the following steps:

[0008] Step one: using a multi-stage network based on image pyramid to extract feature correlation to realize multi-scale prediction at the feature level;

[0009] Step two: based on step one, using a method of extracting and fusing global features and local features respectively to ensure effective improvement of registration accuracy;

[0010] Step three: after processing the features in step two, using a Transformer module to analyze the correlation of the features of the target image and the reference image to realize more distant and more accurate feature correlation capture;

[0011] Step four: using a detailed feature extraction mechanism based on feature region discrete detection to remove redundant information and improve registration accuracy.

[0012] Further, in step one, the feature correlation is extracted using a multi-stage network based on image pyramid; specifically:

[0013] Step one: using a convolutional neural network layer with shared weights to extract features and downsample the two or more input infrared remote sensing images, the network structure uses a YOLO-V8 backbone network composed of a basic convolution module and a c2f module, and uses cross-step convolution to realize multi-scale down sampling of the image;

[0014] Step two: the multi-stage network extracts image features at three levels of original resolution, 1 / 2 resolution and 1 / 8 resolution;

[0015] Step three: using the image features extracted at each level to estimate homography level by level, and passing the estimated upper layer homography to the lower layer to gradually improve the accuracy of homography estimation, thereby realizing coarse-to-fine homography prediction.

[0016] Further, in step two, the global feature is obtained through the top layer of the image pyramid, specifically:

[0017] Each level of image feature is reprocessed through the remaining network after being distorted based on the upper layer homography matrix, and finally the top layer feature is obtained, and the resolution and scale of each level of global feature are the same; the local feature is obtained by cutting and splicing each level of image feature, and the scale of each level of local feature is the same, but the resolution increases level by level.

[0018] Further, in step three, the Transformer module is used to analyze the correlation between the features of the target image and the reference image to achieve more distant and more accurate feature correlation capture; specifically:

[0019] Step three one: global features use whole feature map self-attention analysis to capture global correlation under large baseline through the Transformer module to ensure the correlation extraction performance under large baseline; local features use the Swim-Transformer module, set the window size to the block size, and the depth is 1;

[0020] Step three two: after extracting the features at each level by using the multi-stage network and calculating the correlation, a regression network composed of convolutional layers and fully connected layers is designed to predict the displacement of the four vertices for determining homography; except for the top-level features, each level of the multi-stage network extracts the correlation between the distorted target features and the reference features, i.e., only the residual offset Δ k is predicted, not the complete offset;

[0021] Step three three: the homography obtained by solving the final offset calculation is used to distort the target image to achieve coarse alignment of the image; the homography estimation ability of each stage is trained unsupervisedly using the distorted image and the reference image obtained at each level.

[0022] Further, in step three two, the calculation formula of the residual offset Δ k is as follows:

[0023]

[0024] In the formula, A is the target image, B is the reference image, DLT is the homography operation, W is the distortion of the image using the homography, is the operation of the vertex residual offset between the reference image features and the distorted target image features, k is the current stage number, Δ k is the residual offset corresponding to the current stage, is the sum of all residual offsets before the kth stage.

[0025] Further, in step three three, the final offset is calculated as follows:

[0026] Δ Fin = Δ1+ Δ2+ Δ3

[0027] In the formula, Δ Fin is the final offset.

[0028] Further, in step four, a detailed feature extraction mechanism based on feature region discrete detection is used to remove redundant information and improve registration accuracy; specifically:

[0029] Step four one: providing a feature detection head composed of N+3 convolution layers, where the first N layers down-sample the high-resolution feature map of the current stage N times, the middle two layers extract features, and the last layer is a 1x1 convolution layer for confidence regression calculation of strong features to obtain a confidence map of Bx1xH / 2 N xW / 2 N , where the value of each position in the confidence map represents the confidence of the position corresponding to the strong feature region;

[0030] Step four two: output the coordinates of the top m*m points with the highest confidence in the confidence map, and restore them to the original resolution to obtain the strong feature region center point coordinates, as shown in the following formula:

[0031]

[0032] where x i ,y i are the coordinates of the feature points at the down-sampling resolution, x i ′,y i ′ are the coordinates of the feature points at the original resolution, b lth is the patch width, and H and W are the length and width of the feature map, respectively;

[0033] Step four three: based on the strong feature region center point coordinates obtained by feature detection, a position mask of the strong feature region is constructed with the strong feature region center point coordinates as the center and the patch width as the radius, and the high-resolution feature map is extracted by patching;

[0034] Step four four: position encoding is performed on each pixel in the high-resolution feature map;

[0035] Step four five: generating an absolute position information map with a size of Bx2xHxW, where 2 represents two channels of information, and the row and column coordinates of the corresponding pixels are stored in the two channels respectively, and the coordinates are normalized;

[0036] Step four six: after convolution layer processing, a position encoding with a size of BxCxHxW is generated, and is added to the original resolution feature to realize the addition of position information; where C is the number of feature channels;

[0037] Step four seven: using the strong feature region position mask to segment and extract the original resolution feature with added position encoding to obtain m*m image blocks with a size of BxCxH b xW b , where H b and W b are the length and width of the patch image, respectively, and the size is b lth x2;

[0038] Step four eight: the image block is reorganized by splicing to obtain a new local feature map with a size of BxCxmH b XmW b .

[0039] The beneficial effects of the present application relative to the prior art are: the present application is based on multi-source infrared remote sensing image feature alignment and registration, and proposes a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning. The method uses a multi-stage feature extraction network and a Transformer module to extract image correlation features, ensuring the registration performance under a large baseline. Secondly, based on discrete feature detection, while eliminating abnormal feature regions, the effective local feature regions are selected and spliced, realizing the fusion of local high-resolution features and global low-resolution features. The specific innovations are as follows:

[0040] (1) A multi-stage feature extraction and progressive alignment network is designed; multi-scale features are used, which are compatible with cascade estimation and cyclic estimation, effectively improving the registration accuracy and realizing unsupervised image distortion.

[0041] (2) A detailed feature supplement mechanism is proposed; based on feature region discrete detection, while eliminating low-texture regions, local patch sampling and reorganization of strong feature regions at each stage are performed, realizing the supplement and fusion of high-resolution detailed features based on global features, reducing the computational complexity and effectively improving the registration accuracy.

[0042] (3) The method of extracting and fusing global features and local features respectively is used to ensure the effective improvement of registration accuracy.

[0043] (4) The Transformer module is used to analyze the correlation of the features of the target image and the reference image, making full use of global features and local features to realize more distant and more accurate feature correlation capture.

[0044] (5) Based on discrete feature detection, while eliminating abnormal feature regions, the effective local feature regions are selected and spliced, realizing the fusion of local high-resolution features and global low-resolution features. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of the multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning of the present application;

[0046] Figure 2 is a schematic diagram of a multi-stage progressive homography estimation network;

[0047] Figure 3 is a schematic diagram of a feature correlation calculation module structure, wherein:Figure 3 (a) is a global correlation diagram obtained by processing global features using a Transformer module; Figure 3 (b) is a local correlation diagram obtained by processing local features using a Swim-Transformer module after detecting the local features;

[0048] Figure 4 is a high-resolution feature discrete detection process diagram. DETAILED DESCRIPTION

[0049] As shown in Figure 1 , the embodiment discloses a multi-source infrared remote sensing image feature alignment and registration method based on unsupervised learning, which comprises the following steps:

[0050] Step one: using a multi-stage network based on an image pyramid to extract feature correlation to achieve multi-scale prediction at the feature level;

[0051] The multi-stage network based on the image pyramid is used to extract feature correlation, specifically:

[0052] Step one: using a shared weight convolutional neural network (CNN) layer to extract features and downsample two or more input infrared remote sensing images, the network structure uses a YOLO-V8 backbone network composed of a basic convolution module and a c2f (CSP Bottleneck with 2 Convolutions) module, and uses a stride convolution to realize multi-scale downsampling of the image, thereby extracting feature information of the image at different scales to provide high-quality feature representation for subsequent image registration;

[0053] Step two: the multi-stage network extracts image features at three levels of original resolution, 1 / 2 resolution, and 1 / 8 resolution;

[0054] Step three: using the image features extracted at each level to estimate homography level by level, and passing the estimated upper-level homography to the lower level to gradually improve the accuracy of homography estimation, thereby realizing coarse-to-fine homography prediction.

[0055] Step two: based on step one, a method of extracting and fusing global features and local features is used to ensure effective improvement of registration accuracy;

[0056] The global features are obtained through the top layer of the image pyramid, specifically:

[0057] Each level of image features is reprocessed through the remaining network after being distorted based on the upper-level homography matrix, and finally the top-level features are obtained, in which Figure 2The global features are represented by red lines and green lines, and the resolution and scale of the global features of each level are the same; the local features are obtained by patching and splicing the features of each level of images, and the scale of each level of local features is the same, but the resolution increases gradually.

[0058] The global features and the local features are extracted and fused respectively, so as to have the advantages of global low-resolution features and local high-resolution features.

[0059] Step three: after the features are processed in step two, the features of the target image and the reference image are analyzed by using a Transformer module to realize more distant and more accurate feature correlation capture;

[0060] The features of the target image and the reference image are analyzed by using a Transformer module to realize more distant and more accurate feature correlation capture; specifically,

[0061] Step three one: in order to adapt to the image stitching requirements under a large baseline and improve the estimation accuracy, more distant and more accurate feature correlation capture is realized by using a Transformer module to analyze the features of the target image and the reference image. Figure 3 As shown in the figure, the global features are analyzed by using a Transformer module to capture the global correlation under a large baseline, which guarantees the correlation extraction performance under a large baseline; the local features are recombined by using a patching method, which destroys the continuity of the features in space, so when the local features are processed, a Swim-Transformer module is used, the window size is set to the block size, and the depth is 1, which is equivalent to calculating the attention only in the respective local blocks without cross-block calculation, so as to realize independent correlation analysis;

[0062] Step three two: after the features of each level are extracted by using a multi-stage network and the correlation is calculated, a regression network composed of convolutional layers and fully connected layers is designed to predict the displacement of four vertices for determining homography; except for the top-level features, each level of the multi-stage network extracts the correlation of the target features and the reference features after distortion, that is, only the residual offset k is predicted, and the complete offset is not predicted.

[0063] The calculation formula of the residual offset k is as follows:

[0064]

[0065] In the formula, A is the target image, B is the reference image, DLT is homography operation, W is the distortion of the image by using homography, For the operation of the vertex residual offset between the reference image feature and the distorted target image feature, k is the current stage number, Δ k is the residual offset corresponding to the current stage, is the sum of all residual offsets before the kth stage.

[0066] Step three: the target image is distorted by solving the homography obtained by the final offset calculation to achieve coarse alignment of the image; the homography estimation capability of each stage is unsupervisedly trained using the distorted image and the reference image obtained at each level;

[0067] The final offset calculation is as follows:

[0068] Δ Fin = Δ1+ Δ2+ Δ3

[0069] In the formula, Δ Fin is the final offset.

[0070] Step four: using a detailed feature extraction mechanism based on feature region discrete detection to remove redundant information and improve registration accuracy;

[0071] The detailed feature extraction mechanism based on feature region discrete detection is used to remove redundant information and improve registration accuracy; specifically:

[0072] Step four one: a feature detection head is provided, which is composed of N+3 convolution layers, as shown in the figure Figure 4 , wherein the first N layers perform N times down-sampling on the high-resolution feature map of the current stage, the middle two layers extract features, and the last layer is a 1x1 convolution layer for confidence regression calculation of strong features to obtain a confidence map of Bx1xH / 2 N xW / 2 N , the value of each position in the confidence map represents the credibility of the strong feature region corresponding to the position, and the purpose of N times down-sampling is to expand the receptive field while making the feature region as dispersed as possible at the original resolution without overlapping regions;

[0073] Step four two: in the confidence map, the coordinates of the first m*m points with the highest confidence are output, and they are restored to the original resolution to obtain the strong feature region center point coordinates, as shown in the formula:

[0074]

[0075] In the formula, x i ,y i is the feature point position coordinate at the down-sampled resolution, x i ′,y i ′ is the feature point position coordinate at the original resolution, and b lthFor the width of the cutout, H and W are the length and width of the feature map respectively;

[0076] Step four three: based on the feature detection, the strong feature region center point coordinates are obtained, the strong feature region position mask is constructed with the strong feature region center point coordinates as the center and the cutout width as the radius, and the high-resolution feature map is cut out and extracted;

[0077] Step four four: position coding is performed on each pixel in the high-resolution feature map, because the cutout and splicing operation will destroy the coordinate continuity of the original feature map, in order to make the feature still contain the coordinate information under the original resolution after recombination, it is necessary to perform position coding on each pixel in the high-resolution feature map;

[0078] Step four five: generate an absolute position information map with a size of Bx2xHxW, wherein 2 represents the information of two channels, and the row and column coordinates of the corresponding pixels are stored in the two channels respectively, and the coordinates are normalized;

[0079] Step four six: after the convolution layer processing, the position coding with a size of BxCxHxW is generated, and is added to the original resolution feature to realize the addition of position information; wherein C is the number of feature channels;

[0080] Step four seven: using the strong feature region position mask, the original resolution feature added with position coding is segmented and cut out to obtain m*m images with a size of BxCxH b xW b , wherein H b and W b are the length and width of the cutout image respectively, and the size is b lth x2;

[0081] Step four eight: the image blocks are spliced and recombined to obtain a new local feature map with a size of BxCxmH b xmW b . Through the process of segmentation, cutting and splicing and recombination, the feature map size is greatly reduced while maintaining the original resolution, the redundant information is removed and the influence of the low texture feature region is rejected, the subsequent calculation amount is reduced and the registration accuracy is improved.

[0082] The application proposes a detail feature extraction mechanism based on feature region discrete detection, which detects and selects the strong texture feature region in the high-resolution feature map, so as to cut out and splice the effective features, remove the redundant features and significantly reduce the feature map size, so as to better realize the fusion with the global feature.

[0083] The above merely provides the preferred embodiments of the present application, and the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes according to the technical scheme and the inventive concept of the present application within the technical scope disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for feature alignment and registration of multi-source infrared remote sensing images based on unsupervised learning, characterized in that: The method comprises the following steps: Step one: using a multi-stage network based on an image pyramid to extract feature correlation to achieve multi-scale prediction at the feature level; the multi-stage network based on the image pyramid extracts feature correlation; specifically: Step one: two or more input infrared remote sensing images are extracted and down-sampled by using a convolutional neural network layer with shared weights, the network structure uses a YOLO-V8 backbone network composed of a basic convolutional module and a module combination, and multi-scale down-sampling of the image is realized by using a cross-step convolution; Step two: the multi-stage network extracts image features at three levels of original resolution, 1 / 2 resolution and 1 / 8 resolution; Step three: use the image features extracted at each level to estimate homography step by step, and pass the estimated upper-level homography to the lower level to gradually improve the accuracy of homography estimation, thereby achieving coarse-to-fine homography prediction; Step two: based on step one, use a method of extracting and fusing global features and local features respectively to ensure effective improvement of registration accuracy; the global features are obtained through the top layer of the image pyramid, specifically: After the image features at each level are distorted based on the upper-level homography matrix, they are reprocessed through the remaining network to obtain the top-level features, and the resolution and scale of each level of global features are the same; the local features are obtained by patching and splicing each level of image features, and the scale of each level of local features is the same, but the resolution increases step by step; Step three: after processing the features in step two, use a Transformer module to analyze the correlation between the features of the target image and the reference image to capture more distant and more accurate feature correlation; the Transformer module is used to analyze the correlation between the features of the target image and the reference image to capture more distant and more accurate feature correlation; specifically: Step three one: the global features use whole feature map self-attention analysis to capture global correlation under a large baseline through the Transformer module to ensure the performance of correlation extraction under a large baseline; the local features use a Swim-Transformer module with a window size set to the block size and a depth of 1; Step three two: after extracting features at each level and calculating the correlation using the multi-stage network, a regression network composed of convolutional layers and fully connected layers is designed to predict the displacement of the four vertices used to determine homography; in addition to the top-level features, each level of the multi-stage network extracts the correlation between the distorted target features and the reference features, i.e. only the residual displacement from the previous level is predicted, not the complete displacement ​ Step three three: the homography obtained by solving the final offset calculation is used to distort the target image to achieve coarse alignment of the image; the homography estimation capability of each stage is trained unsupervised using the distorted image and the reference image obtained at each level; Step four: use a detailed feature extraction mechanism based on feature region discrete detection to remove redundant information and improve registration accuracy; specifically: Step four one: providing a feature detection head composed of N+3 convolutional layers, where the first N layers down-sample the high-resolution feature map of the current stage N times, the middle two layers extract features, and the last layer is a 1x1 convolutional layer for confidence regression calculation of strong features to obtain a confidence map, where the value of each position in the confidence map represents the credibility of the position corresponding to the strong feature region. ​ Step four two: in the confidence map, output the coordinates of the top points with the highest confidence and restore them to the original resolution to obtain the coordinates of the center points of the strong feature regions, as shown in the following formula: In the formula, is the feature point position coordinate at the down-sampling resolution, is the feature point position coordinate at the original resolution, is the width of the cutout, H and W are the length and width of the feature map, respectively. Step four three: after obtaining the strong feature region center point coordinates based on feature detection, construct a position mask of the strong feature region with the strong feature region center point coordinates as the center and the patch width as the radius, and perform patch extraction on the high-resolution feature map; Step four four: position encoding is performed on each pixel in the high-resolution feature map; Step four five: generate absolute position information map, scale is wherein 2 indicates information of two channels, the two channels respectively store row and column coordinates of corresponding pixels, and the coordinates are normalized. Step four six: generate position encoding with scale of 2 after processing by convolutional layer, and add to original resolution feature to achieve addition of position information; wherein, is the number of feature channels. Step four seven: using the strong feature region position mask to segment and cut the original resolution feature added with position coding, to obtain An image block with a scale of , wherein, and are the length and width of the cut image respectively, and the size is ; Step four eight: reorganize the image blocks via splicing to obtain a new local feature map with a size of .

2. The method of claim 1, wherein the method is based on unsupervised learning. In step three two, the residual offset The calculation formula is as follows: where A is the target image, B is the reference image, DLT is the homography operation, is the homography operation, is the vertex residual offset operation between the reference image feature and the warped target image feature, k is the current stage number, is the residual offset corresponding to the current stage, is the sum of all residual offsets before the kth stage.

3. The method of claim 1 or 2, wherein: In step three three, the final offset calculation is as follows: In the formula, is the final offset.

Images