Infrared image registration method and unsupervised learning image registration model training method

Abstract

The invention discloses an infrared image registration method and an unsupervised learning image registration model training method, and belongs to the technical field of image processing. Aiming at the characteristics of weak texture and few features of a low-overlapping-rate infrared image, a weak texture feature extractor is firstly designed, and the feature extraction capability of a model on a weak texture image is improved through multi-scale feature fusion; then, by adopting multi-scale receptive field correlation calculation, the discrimination capability of the model on similar features is enhanced; furthermore, by adopting a homography transformation estimation method which is optimized gradually from coarse to fine, the estimation precision of a transformation matrix is improved; and finally, realizing registration and splicing of the infrared images through image transformation and overlapping region fusion processing. According to the method, the registration precision of the low-overlapping-rate infrared image can be effectively improved, and application scenes such as large-scene image splicing can be supported.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and more specifically, relates to a method for registering infrared images with low overlap rates and a method for training unsupervised image registration models. Background Technology

[0002] With the advancement of infrared imaging technology, infrared images are increasingly widely used in military, medical, and industrial fields. In complex environments, infrared images can provide unique information, and registration helps improve the accuracy of target recognition and tracking. Registration aligns infrared images from different times, angles, or sensors, improving the precision of target detection and recognition. Registered images facilitate analysis such as change detection and target tracking, enhancing overall system performance. However, registering infrared images acquired from different angles, especially those containing overlapping areas, presents challenges such as changing viewpoints, geometric distortions, occlusion, poor feature consistency, and noise interference. These factors complicate feature extraction and matching, increasing the difficulty of registration. Nevertheless, successful multi-angle infrared image registration is significant. It not only improves the accuracy of target recognition and tracking but also enhances the comprehensiveness of scene understanding through multi-view fusion, supporting 3D reconstruction and depth information extraction. Furthermore, registration of infrared images with other modal data can promote multimodal data fusion, providing richer information support for military reconnaissance, medical diagnosis, and industrial inspection, thereby improving the system's robustness and decision-making capabilities in complex environments. Therefore, despite the high technical difficulty, developing efficient and robust registration algorithms is of great value in promoting the application and development of infrared image processing technology.

[0003] When registering infrared images captured from different angles or positions, the comprehensiveness and reliability of the information can be enhanced. Image registration technology calculates the feature matching relationship between two images with overlapping areas, and then fuses the transformed and aligned images to obtain a registered image. Traditional methods typically involve first extracting feature points, then filtering and matching these feature points, calculating the homography matrix based on the optimal matching points, and finally using this matrix to transform and fuse the images.

[0004] Because low-overlap infrared images typically have few and similar feature points, traditional registration methods face significant challenges in feature extraction and matching, resulting in poor image registration performance. While advanced deep learning methods possess powerful feature extraction capabilities, their ability to distinguish similar features is insufficient, leading to inadequate image registration accuracy. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs in existing technologies, this invention provides a method for training image registration models based on multi-scale progressive optimization of infrared image registration and unsupervised learning, specifically for low-overlap infrared image stitching applications. This method aims to solve the technical problem of poor registration performance for low-overlap infrared images.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an infrared image registration method, comprising the following steps: S1. Input the infrared images to be registered: Image A and Image B, where Image A is the reference image and Image B is the floating image; S2. Use a weak texture extractor to extract features from images A and B respectively, and obtain feature maps of images A and B at three different scales; S3. Based on the feature map, the homography transformation matrix is ​​gradually optimized and estimated using a homography transformation matrix estimation method to obtain the homography transformation matrices of image A and image B; S4. Use a homography transformation matrix to transform image B so that image A is spatially aligned with image B. S5. Based on the alignment relationship, perform fusion processing on the pixels in the overlapping area of ​​image A and image B to obtain the registered fused image of image A and image B.

[0007] Preferably, in step S2, the weak texture feature extractor specifically comprises: The weak texture feature extractor takes an image X, where X = A or B, as input and processes it through three convolutional layers with the same structure. Each convolutional layer contains two cascaded CBR modules. The first CBR module contains a Conv module with a stride of 1 and a kernel size of 7*7, a Batch Normalization module (BN), and an Activation Function module (ReLU). The second CBR module contains a Conv module with a stride of 2 and a kernel size of 7*7, a Batch Normalization module (BN), and an Activation Function module (ReLU). Image X can be processed through three convolutional layers to obtain feature maps of three different scales. , and ; Feature map go through The convolutional processing directly serves as the third-scale feature map. Output the feature map, where x = a or b. go through Convolution, and with feature maps The feature maps from the bilinear interpolation are then summed to obtain the fused feature maps from the third scale and the second scale. Similarly, the processing steps, features go through After convolution, and with feature maps The features obtained after bilinear interpolation are summed to obtain the fused features of the second scale and the first scale. ;in, Here are the feature maps of image A at three different scales. Feature maps of image B at the three different scales. .

[0008] Preferably, step S3 specifically includes the following steps: Third-scale feature correlation calculation: Calculate the third-scale feature maps of image A and image B. and Inter-correlation, to obtain the feature correlation matrix at this scale. ; Third-scale homography transformation estimation: Estimating the encoder's correlation matrix with the features through homography transformation. The processing yields the homography transformation relationship between image B and image A at the third scale, i.e., the third-scale homography transformation matrix. ; Second-scale homography transformation estimation: using right Perform matrix transformations to obtain Then, using the third-scale feature map and The same method is used to calculate the correlation, resulting in... and Feature correlation matrix Using the same method as the third-scale homography transformation estimation, the second-scale homography transformation matrix is ​​obtained. ; First-scale homography transformation estimation: The same processing procedure as the second-scale homography transformation estimation is used to obtain the first-scale homography transformation matrix. ; Synthetic homography transformation estimation: The synthetic homography transformation matrix H is calculated according to the following formula. H is used as the homography transformation matrix for images A and B.

[0009] Preferably, in the third-scale feature correlation calculation step, the correlation is obtained by processing the multi-scale receptive field feature correlation matrix construction module; The input to the multi-scale receptive field feature correlation matrix construction module is: and , First, the feature map is calculated. and Correlation coefficient plot Correlation coefficient plot mid-coordinate point Relevant values The calculation formula is: ; Correlation coefficient plot After processing through two branches, the first branch directly... The input is fed into the Concat module; the second branch is processed by the first CBR module, which consists of a Conv module with a stride of 2 and a kernel size of 3*3, a batch normalization module (BN), and an activation function module (ReLU). After processing, the feature map is obtained. ; After processing through two branches, the first branch obtains the feature map using bilinear interpolation. The input is then directly fed into the Concat module. The second branch first processes the data through the second CBR module, which consists of a Conv module with a stride of 2 and a kernel size of 3*3, a batch normalization module (BN), and an activation function module (ReLU). The resulting feature map... After bilinear interpolation, we obtain Finally, the concat module is used to... , and By combining the results, a multi-receptive field correlation matrix is ​​obtained. As the feature correlation matrix at this scale .

[0010] Preferably, the homography transform estimation encoder comprises two identical CBR modules and one core with a size of [missing information]. Each CBR module consists of a convolutional layer, a fully connected linear transform layer, and a DLT. Each CBR module contains a kernel with a stride of 2 and a size of [missing information]. The convolutional layer, one batch normalization layer, and one ReLU activation layer, along with a multi-receptive field correlation matrix, are used. First, the encoder sequentially encodes the data through two CBR modules in the homography transform estimation encoder, and then passes it through Conv... The convolutional layer adjusts the feature channels, and then the fully connected linear transformation layer outputs the coordinates of the four endpoints of image B, as well as the coordinates of their corresponding points in image A. Finally, the coordinates of the four pairs of endpoints are input into the direct linear transformation model (DLT) for solving, which yields the homography transformation matrices of images A and B at the corresponding scales.

[0011] According to another aspect of the present invention, the present invention also provides a method for training an unsupervised image registration model, comprising the following steps: A1: Obtain image pairs with overlapping areas and construct a training dataset; A2: Randomly select one pair of images from the dataset as the two images to be registered; A3: Input the two images into the unsupervised infrared image registration method according to any one of claims 1-5 to obtain the homography transformation matrix; A4: Obtain the overlapping region of the two images based on the homography transformation matrix, and use the following loss function. Measuring the difference between two images in the overlapping region: + ; in For pixel-level loss, the difference in pixel values ​​of corresponding points in the image is used as the loss, and the root mean square loss function is employed. For feature-level loss, the loss is based on the feature differences between corresponding points in the image, and the root mean square loss function is also used. , where is a weight, and its value is a constant greater than 0 and less than 1; A5: Based on the above loss function To constrain the process, the parameters in the unsupervised infrared image registration method are modified to achieve unsupervised training of the model. A6. Following steps A2-A5, select image pairs from the training dataset one by one for training until all training data has been selected, thus completing the final training.

[0012] In summary, the above technical solutions constructed by this invention can achieve the following beneficial effects: 1. This invention employs a weak texture feature extraction method, which can obtain features of weak texture images at different scales. By fusing multi-scale features, it can obtain weak texture image features with differences, thus meeting the application requirements of weak texture image registration.

[0013] 2. The present invention employs a feature correlation calculation method, which uses receptive fields of different sizes to calculate the correlation of features. This can avoid the influence of local similar regions on image registration and improve the reliability and accuracy of image registration.

[0014] 3. This invention employs a stepwise optimization method for homography transformation estimation. First, coarse-scale features are registered, and the homography transformation matrix is ​​calculated through matching relationships. Then, the transformation matrix is ​​used for coarse transformation, and fine-scale features are used to correct the transformation matrix through transformation relationships to obtain a more accurate transformation matrix. This achieves coarse-to-fine transformation relationship estimation and improves the accuracy of image registration.

[0015] 4. The unsupervised training method provided by this invention does not require the preparation of labeled training samples, which can reduce the difficulty of preparing training dataset samples and can use a large amount of unlabeled data for model training, which is beneficial to improving the generalization ability of the model. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention; Figure 2 The structural diagram of the weak texture feature extractor designed for this invention; Figure 3 The structural diagram of the multi-scale receptive field feature correlation calculation module designed for this invention is shown below. Figure 4 The structure diagram of the homography transform estimation encoder designed in this invention is shown below; Figure 5 This is the result of processing the example image according to the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] refer to Figures 1 to 4 The infrared image registration method of the present invention includes the following steps: S1. Input the infrared images to be registered: Image A and Image B, where Image A is the reference image and Image B is the floating image.

[0019] S2. Use a weak texture extractor to extract features from images A and B respectively, and obtain feature maps of image A at three different scales. and feature maps of image B at three different scales. .

[0020] The weak texture feature extractor is as follows: The weak texture feature extractor takes an image X, where X = A or B, as input and processes it through three convolutional layers with the same structure. Each convolutional layer contains two cascaded CBR modules. The first CBR module contains a Conv module with a stride of 1 and a kernel size of 7*7, a Batch Normalization module (BN), and an Activation Function module (ReLU). The second CBR module contains a Conv module with a stride of 2 and a kernel size of 7*7, a Batch Normalization module (BN), and an Activation Function module (ReLU). Image X can be processed through three convolutional layers to obtain feature maps of three different scales. , and ; Feature map go through The convolutional processing directly serves as the third-scale feature map. Output the feature map, where x = a or b. go through Convolution, and with feature maps The feature maps from the bilinear interpolation are then summed to obtain the fused feature maps from the third scale and the second scale. Similarly, the processing steps, features go through After convolution, and with feature maps The features obtained after bilinear interpolation are summed to obtain the fused features of the second scale and the first scale. .

[0021] S3. Based on the feature map, a stepwise optimization estimation method for the homography transformation matrix is ​​used to obtain the homography transformation matrices of image A and image B. Specifically, this includes the following steps: Third-scale feature correlation calculation: Calculate the third-scale feature maps of image A and image B. and Inter-correlation, to obtain the feature correlation matrix at this scale. The correlation was obtained by processing the correlation matrix of the multi-scale receptive field feature. The input to the multi-scale receptive field feature correlation matrix construction module is: and , First, the feature map is calculated. and Correlation coefficient plot Correlation coefficient plot mid-coordinate point Relevant values The calculation formula is: ; Correlation coefficient plot After processing through two branches, the first branch directly... The input is fed into the Concat module; the second branch is processed by the first CBR module, which consists of a Conv module with a stride of 2 and a kernel size of 3*3, a batch normalization module (BN), and an activation function module (ReLU). After processing, the feature map is obtained. ; After processing through two branches, the first branch obtains the feature map using bilinear interpolation. The input is then directly fed into the Concat module. The second branch first processes the data through the second CBR module, which consists of a Conv module with a stride of 2 and a kernel size of 3*3, a batch normalization module (BN), and an activation function module (ReLU). The resulting feature map... After bilinear interpolation, we obtain Finally, the concat module is used to... , and By combining the results, a multi-receptive field correlation matrix is ​​obtained. As the feature correlation matrix at this scale .

[0022] Third-scale homography transformation estimation: Estimating the encoder's correlation matrix with the features through homography transformation. The processing yields the homography transformation relationship between image B and image A at the third scale, i.e., the third-scale homography transformation matrix. ; The homography transform estimator encoder contains two identical CBR modules and one kernel with a size of [missing information]. Each CBR module consists of a convolutional layer, a fully connected linear transform layer, and a DLT. Each CBR module contains a kernel with a stride of 2 and a size of [missing information]. The convolutional layer, one batch normalization layer, and one ReLU activation layer, along with a multi-receptive field correlation matrix, are used. First, the encoder sequentially encodes the data through two CBR modules in the homography transform estimation encoder, and then passes it through Conv... The convolutional layer adjusts the feature channels, and then the fully connected linear transformation layer outputs the coordinates of the four endpoints of image B, as well as the coordinates of their corresponding points in image A. Finally, the coordinates of the four pairs of endpoints are input into the direct linear transformation model (DLT) for solving, which yields the homography transformation matrices of images A and B at the corresponding scales.

[0023] Second-scale homography transformation estimation: using right Perform matrix transformations to obtain Then, using the third-scale feature map and The same method is used to calculate the correlation, resulting in... and Feature correlation matrix Using the same method as the third-scale homography transformation estimation, the second-scale homography transformation matrix is ​​obtained. .

[0024] First-scale homography transformation estimation: The same processing procedure as the second-scale homography transformation estimation is used to obtain the first-scale homography transformation matrix. .

[0025] Synthetic homography transformation estimation: The synthetic homography transformation matrix H is calculated according to the following formula. H is used as the homography transformation matrix for images A and B.

[0026] S4. Use a homography transformation matrix to transform image B so that image A is spatially aligned with image B. S5. Based on the alignment relationship, perform fusion processing on the pixels in the overlapping area of ​​image A and image B to obtain the registered fused image of image A and image B.

[0027] This invention also provides a method for training an unsupervised image registration model, comprising the following steps: A1: Obtain image pairs with overlapping areas and construct a training dataset; A2: Randomly select one pair of images from the dataset as the two images to be registered; A3: Input the two images into the unsupervised infrared image registration method according to any one of claims 1-5 to obtain the homography transformation matrix; A4: Obtain the overlapping region of the two images based on the homography transformation matrix, and use the following loss function. Measuring the difference between two images in the overlapping region: + ; in For pixel-level loss, the difference in pixel values ​​of corresponding points in the image is used as the loss, and the root mean square loss function is employed. For feature-level loss, the loss is based on the feature differences between corresponding points in the image, and the root mean square loss function is also used. is a weight, and its value is a constant greater than 0 and less than 1.

[0028] A5: Based on the above loss function To constrain the process, the parameters in the unsupervised infrared image registration method are adjusted to achieve unsupervised training of the model; the adjusted parameters include the weight parameters involved in all steps of the infrared image registration method. A6. Following steps A2-A5, select image pairs from the training dataset one by one for training until all training data has been selected, thus completing the final training.

[0029] refer to Figure 5The invention presents three examples, showing how input images yield transformation results, which are then registered and stitched together. Experiments demonstrate that the infrared image registration method and image registration model training method for low overlap rates of this invention belong to the field of image processing technology. Addressing the characteristics of weak texture and few features in low overlap rate infrared images, this invention first designs a weak texture feature extractor, improving the model's feature extraction capability for weak texture images through multi-scale feature fusion. Then, by employing multi-scale receptive field correlation calculation, it enhances the model's ability to distinguish similar features. Furthermore, by adopting a homography transformation estimation method that is progressively optimized from coarse to fine, it improves the estimation accuracy of the transformation matrix. Finally, through image transformation and overlap area fusion processing, it achieves the registration and stitching of infrared images. This invention can effectively improve the registration accuracy of low overlap rate infrared images and supports application scenarios such as large-scene image stitching.

[0030] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An infrared image registration method, characterized in that, It includes the following steps: S1. Input the infrared images to be registered: Image A and Image B, where Image A is the reference image and Image B is the floating image; S2. Use a weak texture extractor to extract features from images A and B respectively, and obtain feature maps of images A and B at three different scales; S3. Based on the feature map, the homography transformation matrix is ​​gradually optimized and estimated using a homography transformation matrix estimation method to obtain the homography transformation matrices of image A and image B; S4. Use a homography transformation matrix to transform image B so that image A is spatially aligned with image B. S5. Based on the alignment relationship, perform fusion processing on the pixels in the overlapping area of ​​image A and image B to obtain the registered fused image of image A and image B.

2. The infrared image registration method according to claim 1, characterized in that, In step S2, the weak texture feature extractor specifically refers to: The weak texture feature extractor takes an image X, where X = A or B, as input and processes it through three convolutional layers with the same structure. Each convolutional layer contains two cascaded CBR modules. The first CBR module contains a Conv module with a stride of 1 and a kernel size of 7*7, a Batch Normalization module (BN), and an Activation Function module (ReLU). The second CBR module contains a Conv module with a stride of 2 and a kernel size of 7*7, a Batch Normalization module (BN), and an Activation Function module (ReLU). Image X can be processed through three convolutional layers to obtain feature maps of three different scales. , and ; Feature map go through The convolutional processing directly serves as the third-scale feature map. Output the feature map, where x = a or b. go through Convolution, and with feature maps The feature maps from the bilinear interpolation are then summed to obtain the fused feature maps from the third scale and the second scale. Similarly, the processing steps, features go through After convolution, and with feature maps The features obtained after bilinear interpolation are summed to obtain the fused features of the second scale and the first scale. ;in, Here are the feature maps of image A at three different scales. Feature maps of image B at the three different scales. .

3. The infrared image registration method according to claim 2, characterized in that, Step S3 specifically includes the following steps: Third-scale feature correlation calculation: Calculate the third-scale feature maps of image A and image B. and Inter-correlation, to obtain the feature correlation matrix at this scale. ; Third-scale homography transformation estimation: Estimating the encoder's correlation matrix with the features through homography transformation. The processing yields the homography transformation relationship between image B and image A at the third scale, i.e., the third-scale homography transformation matrix. ; Second-scale homography transformation estimation: using right Perform matrix transformations to obtain Then, using the third-scale feature map and The same method is used to calculate the correlation, resulting in... and Feature correlation matrix Using the same method as the third-scale homography transformation estimation, the second-scale homography transformation matrix is ​​obtained. ; First-scale homography transformation estimation: The same processing procedure as the second-scale homography transformation estimation is used to obtain the first-scale homography transformation matrix. ; Synthetic homography transformation estimation: The synthetic homography transformation matrix H is calculated according to the following formula. H is used as the homography transformation matrix for images A and B.

4. The infrared image registration method according to claim 3, characterized in that, In the third-scale feature correlation calculation step, the correlation is obtained by processing the multi-scale receptive field feature correlation matrix construction module; The input to the multi-scale receptive field feature correlation matrix construction module is: and , First, the feature map is calculated. and Correlation coefficient plot Correlation coefficient plot mid-coordinate point Relevant values The calculation formula is: ； Correlation coefficient plot After processing through two branches, the first branch directly... The input is fed into the Concat module; the second branch is processed by the first CBR module, which consists of a Conv module with a stride of 2 and a kernel size of 3*3, a batch normalization module (BN), and an activation function module (ReLU). After processing, the feature map is obtained. ; After processing through two branches, the first branch obtains the feature map using bilinear interpolation. The input is then directly fed into the Concat module. The second branch first processes the data through the second CBR module, which consists of a Conv module with a stride of 2 and a kernel size of 3*3, a batch normalization module (BN), and an activation function module (ReLU). The resulting feature map... After bilinear interpolation, we obtain Finally, the concat module is used to... , and By combining the results, a multi-receptive field correlation matrix is ​​obtained. As the feature correlation matrix at this scale .

5. The infrared image registration method according to claim 4, characterized in that, The homography transform estimator encoder includes two identical CBR modules and one kernel size of [missing information]. Each CBR module consists of a convolutional layer, a fully connected linear transform layer, and a DLT. Each CBR module contains a kernel with a stride of 2 and a size of [missing information]. The convolutional layer, one batch normalization layer, and one ReLU activation layer, along with a multi-receptive field correlation matrix, are used. First, the encoder sequentially encodes the data through two CBR modules in the homography transform estimation encoder, and then passes it through Conv... The convolutional layer adjusts the feature channels, and then the fully connected linear transformation layer outputs the coordinates of the four endpoints of image B, as well as the coordinates of their corresponding points in image A. Finally, the coordinates of the four pairs of endpoints are input into the direct linear transformation model (DLT) for solving, which yields the homography transformation matrices of images A and B at the corresponding scales.

6. A method for training an unsupervised image registration model, characterized in that, Includes the following steps: A1: Obtain image pairs with overlapping areas and construct a training dataset; A2: Randomly select one image pair from the dataset as the two images to be registered; A3: Input the two images into the unsupervised infrared image registration method according to any one of claims 1-5 to obtain the homography transformation matrix; A4: Obtain the overlapping region of the two images based on the homography transformation matrix, and use the following loss function. Measuring the difference between two images in the overlapping region: + ； in For pixel-level loss, the difference in pixel values ​​of corresponding points in the image is used as the loss, and the root mean square loss function is employed. For feature-level loss, the loss is based on the feature differences between corresponding points in the image, and the root mean square loss function is also used. , where is a weight, and its value is a constant greater than 0 and less than 1; A5: Based on the above loss function To constrain the process, the parameters in the unsupervised registration method for infrared images are adjusted to achieve unsupervised training of the model. A6. Following steps A2-A5, select image pairs from the training dataset one by one for training until all training data has been selected, thus completing the final training.

Images