Infrared and visible light image end-to-end registration method based on phase consistency enhancement

Abstract

The invention provides an infrared and visible light image end-to-end registration method based on phase consistency enhancement, and belongs to the technical field of electric digital data processing. The invention discloses an infrared and visible light image end-to-end registration method based on phase consistency enhancement, which comprises the following steps of: firstly, preprocessing infrared and visible light source images in an image database, and dividing the infrared and visible light source images into a training set and a test set; and constructing a feature extractor based on a phase consistency attention mask, realizing cross-modal consistency feature extraction, and focusing an important space region by using phase information. A self-defined ResSobeNet network is adopted for parameter estimation, and Haar wavelet transform is used for replacing pooling operation, so that information loss in a traditional down-sampling method is avoided. A double-branch generator structure is designed, affine parameters and an optical flow field are generated respectively, meanwhile, constraint on rigid registration and non-rigid registration is achieved, and finally a registration result is obtained through resampling of a space conversion module. According to the method, the problems that cross-modal consistent features are difficult to extract in the infrared and visible light image registration process and explicit registration steps are complicated and difficult to generalize are solved, and the method has good universality and practicability.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic digital data processing, and in particular relates to an end-to-end registration method of infrared and visible light images based on phase consistency enhancement. Background Art

[0002] The goal of multimodal image registration is to spatially align heterogeneous images from different sensors or different wavelengths, thereby achieving multimodal information fusion. This fusion can significantly improve the richness and accuracy of information, providing more comprehensive data support for various applications. However, compared with conventional same-modal image registration, multimodal image registration faces many challenges. For example, the imaging principles and parameter settings of visible light cameras and infrared cameras differ, resulting in differences in image resolution, contrast, and brightness; infrared cameras are severely affected by internal temperature and external thermal airflow, resulting in noisy and blurred images; in addition, jitter and delay may occur during the capture of cross-modal binocular photos, further increasing the difficulty of registration.

[0003] Traditional feature-based registration methods typically involve complex steps such as feature extraction, feature detection, and feature matching, and require careful design and parameter adjustment, making them difficult to automate and scale. In recent years, deep learning-based image registration technology has rapidly developed, providing new solutions for infrared and visible image registration. However, existing methods, which combat modal differences, mostly use a separate modality conversion module to convert multimodal registration into a single-modality registration. This results in a complex network structure and high computational complexity, and the registration results are heavily dependent on the results of the modality conversion.

[0004] These issues lead to the complexity of existing traditional feature registration methods, which require manual parameter design and are difficult to automate and scale. Existing end-to-end registration methods struggle to handle the nonlinear radiometric differences between infrared and visible light images. Modal conversion-based methods require training additional conversion networks, making them difficult to scale to large datasets. A registration method that can effectively handle modal differences and is easily generalizable is urgently needed. In other words, existing technologies lack the ability to effectively handle modal differences between infrared and visible light images, and the matching process is overly complex and difficult to scale. Summary of the Invention

[0005] In view of this, the present invention provides an end-to-end registration method for infrared and visible light images based on phase consistency enhancement, which can solve the technical problems in the existing technology of lack of effective processing of modal differences between infrared and visible light images and the matching process being too complex and difficult to expand.

[0006] The present invention is implemented as follows: the present invention provides an end-to-end registration method for infrared and visible light images based on phase consistency enhancement, comprising: selecting pairs of infrared and visible light images from a data set, performing affine transformations of varying degrees on each pair of images to obtain an expanded data set, randomly selecting training sets and test sets for the registration network in proportion; constructing a phase consistency attention mask subnetwork, performing consistent feature extraction and feature selection under the mask attention of the subnetwork; using a custom ResSobelNet network architecture as a registration parameter estimator, and using Haar wavelet transform to obtain the best matching result. The convolution is replaced by downsampling to reduce the information loss caused by maximum pooling; a deformation field generation and image conversion module is constructed, and a dual-branch generator is designed to generate affine parameters and optical flow field respectively; the affine transformation deformation field and the optical flow field grid are superimposed to obtain the final deformation field, and the spatial conversion module is used to resample the deformed image according to the deformation field to obtain an infrared image aligned with the template image; under the guidance of the loss function, the infrared and visible light datasets are used to train a bidirectional registration network; the infrared image and visible light image to be registered are input into the trained registration network for end-to-end registration to obtain the registration result.

[0007] Among them, dataset expansion refers to selecting pairs of infrared and visible light images in the public dataset, and performing random affine transformations on each pair of images to varying degrees to simulate various transformations that may occur in images in actual applications.

[0008] Among them, phase consistency is an indicator to measure the phase similarity of frequency components at different positions in the image. By analyzing the phase information after Fourier transform of the image, features with inherent consistency can be identified.

[0009] Among them, the phase consistency enhancement mask sub-network uses the feature map calculated by phase consistency as the attention map to guide the sub-network to generate the enhancement mask.

[0010] Among them, the feature extractor uses enhanced masks to extract cross-modal consistent features, improve feature robustness, and implicitly detect feature edges and corners.

[0011] Among them, the customized ResSobelNet is based on the ResNet18 network, combining the edge detection capabilities of the residual network and the Sobel operator, and directly learns the registration parameters from the extracted consistency features without the need to manually design complex matching strategies.

[0012] Among them, affine parameters refer to the linear transformation parameters of the deformed image after translation, rotation, and scaling to the position of the template image; the optical flow field is a displacement field that describes the local deformation of the deformed image to the template image, and each pixel corresponds to a displacement vector.

[0013] Among them, the final deformation field refers to the affine parameters as coarse registration parameters; the optical flow field generated by the other branch is used as the fine registration parameters, and the two are superimposed to obtain the final deformation field.

[0014] Among them, the spatial transformation module is a differentiable image resampling operation, which performs bilinear interpolation reconstruction on the source image according to the final deformation field to generate a result that is aligned with the template image.

[0015] Among them, the loss function guides the training of the registration network by measuring the difference between the predicted deformation field and the actual deformation field, the pixel difference, structural similarity difference, phase consistency difference between the registered image and the template image, and the smoothness of the deformation field to generate the optimal registration result.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] (1) The present invention provides an end-to-end registration method for infrared and visible light images based on phase congruency enhancement. It combines the traditional phase congruency method and converts the traditional signal processing method into a differentiable PyTorch operation, allowing it to be trained end-to-end in a deep learning model. In the feature extraction stage, it extracts cross-modal consistency features, enhances the robustness of feature representation, and effectively solves the modality difference problem in infrared and visible light image registration.

[0018] (2) Random affine transformations of varying degrees are performed on the original dataset to simulate the various changes that may occur in images in the real world, making the model more robust when facing unknown data in practical applications. The backbone network uses the ResNet18 network for end-to-end deformation field estimation, which can automatically learn the optimal feature representation and registration strategy without the need to manually design complex feature extraction and matching processes. At the same time, a dual-branch generator is used to generate the deformation field, achieving dual constraints on rigid and non-rigid registration.

[0019] (3) The present invention introduces phase consistency features, which not only improves the accuracy of infrared and visible light image registration, but also improves the robustness of the registration model. At the same time, the end-to-end learning model makes the model easier to expand and transplant, effectively solving the problem in the existing technology that the processing of modal differences in the infrared and visible light registration process is too complex and difficult to train. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 It is the overall flow chart of the present invention;

[0022] Figure 2 Schematic diagram of the phase consistency enhancement feature extraction module provided by the present invention;

[0023] Figure 3 Schematic diagram of end-to-end deformation field estimation provided by the present invention;

[0024] Figure 4 The comparison diagram is a display of two sets of original visible light images from RoadScene, a deformed infrared image, an infrared image registered by the method of the present invention, and a checkerboard overlay. DETAILED DESCRIPTION

[0025] The principles and features of the present invention are described below with reference to the accompanying drawings and examples. The examples are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0026] Example 1:

[0027] S1: Select pairs of infrared and visible light images from the dataset, perform random affine transformations of varying degrees on each pair of images to generate an extended dataset, and randomly select image pairs in proportion to form the training and test sets.

[0028] S11: Select paired infrared and visible light images from a public dataset to expand the dataset to form the training and test sets. We selected 1074 pairs of infrared and visible light images from the uncalibrated public dataset FLIR.

[0029] S12: Each pair of images is subjected to different degrees of affine transformation, namely small-scale deformation (rotation angle range is ±10 degrees, translation range is ±10 pixels, and scaling ratio is 0.9 to 1.1), medium-scale deformation (rotation angle range is ±20 degrees, translation range is ±12 pixels, and scaling ratio is 0.88 to 1.12), and large-scale deformation (rotation angle range is ±30 degrees, translation range is ±15 pixels, and scaling ratio is 0.85 to 1.15). Then they are cropped to generate a total of 3222 visible light-thermal infrared image pairs with a size of 256×256.

[0030] S13: Randomly select 322 image pairs from the training set, with a test set to training set ratio of 1:9. This step artificially generates a more diverse set of image samples, simulating the various variations that images may experience in real applications. This allows the model to be exposed to a wider data distribution during training, improving its adaptability to unknown data.

[0031] S2: Construct a phase-consistent attention mask sub-network, gradually extract features through a multi-layer convolutional network, and use phase-consistent features to guide feature attention to enhance the robustness and effectiveness of feature representation.

[0032] S21: Construct a phase congruency module. First, perform a two-dimensional fast Fourier transform on the input image to convert the image from the spatial domain to the frequency domain. Based on the frequency domain representation, a two-dimensional frequency coordinate system is constructed to calculate the radial frequency ω and azimuth angle θ of each sampling point, and establish a complete frequency-angle parameter space. By constructing a hybrid filter group containing a Butterworth low-pass filter and a logarithmic Gabor filter, frequency band decomposition and noise suppression are achieved. Then, the Kovesi orthogonal moment model is used to calculate phase congruency.

[0033] Specifically, a bandpass filter bank with a logarithmic frequency response is constructed, and its transfer function is:

[0034]

[0035] Where ω0 is the center frequency and σ / ω0 is 0.55. Compared with conventional Gabor filters, Log-Gabor has better detail capture capabilities in the low-frequency band.

[0036] Phase consistency theory originates from the field of signal processing. Given a signal F(x,y), e so (x,y) and o so (x,y) are the signal F(x,y) and the even symmetric wavelet and oddly symmetric wavelets The corresponding components of the convolution at scale s and direction o, * indicates the convolution operation:

[0037]

[0038] The calculation formula for the transform amplitude under a given wavelet scale can be expressed as:

[0039]

[0040] The phase consistency value for a given position (x, y) is achieved by weighting all scales s and directions o, and its mathematical form is:

[0041]

[0042] Among them, W o (x,y) represents the frequency expansion weighting function, E(x,y) represents the local energy function, T o represents the noise threshold, ε is a small constant added to avoid division by zero, If the internal value is positive, it is taken as itself, otherwise it is 0. To reduce the influence of noise, only the value exceeding the estimated noise T is calculated. o The energy value of the local energy function is calculated as follows:

[0043]

[0044] S22: Construct a mask generation subnetwork, concatenate the input image and the phase consistency features obtained by the phase consistency module to form a 4-channel input, and perform multi-level feature refinement through 5 layers of 3×3 convolution. The first two layers use the phase consistency features as the attention mechanism to guide the subnetwork to pay more attention to cross-modal consistent features.

[0045] S23: Generate a probability mask in the [0,1] interval through Sigmoid activation.

[0046] Specifically, the entire phase consistency module is based on the Pytorch architecture and GPU accelerated, achieving real-time processing while maintaining the theoretical rigor of traditional methods.

[0047] S3: Construct a phase consistency enhanced feature extraction module to fuse the shared features with the enhanced mask generated by the sub-network to extract cross-modal consistency features. At the same time, phase consistency enables the feature extractor to pay more attention to the edge structure of the image, which has the effect of implicit feature detection.

[0048] S31: The feature extraction module uses dual-path feature extraction to preliminarily extract features from infrared and visible light images. It contains three convolutional layers, each with a kernel size of 3×3, a stride of 1, and 4, 8, and 1 channels, respectively. By using the principle of "wide in, narrow out," the network is forced to learn the most essential feature expressions.

[0049] S32: Perform Hadamard product between the obtained 1-channel feature and the enhanced mask, and adaptively focus on cross-modal common features such as edge contours.

[0050] S4: Construct a parameter learning module, use a custom ResSobelNet as the backbone network to learn the mapping relationship between features, and use Haar wavelet convolution instead of pooling operations to reduce information loss during downsampling.

[0051] S41: ResSobelNet is based on the ResNet18 network. It uses a 7×7 convolutional layer to initially extract input features with a stride of 2 and a padding of 3. It outputs a 32-channel 1 / 2 scale feature map and uses batch normalization and ReLU activation functions to enhance the expressiveness of the network model.

[0052] S42: Use 4 residual layers for multi-scale feature extraction. Each residual layer consists of two residual blocks. Each residual block contains two 3×3 convolutional layers and a batch normalization layer, as well as a skip connection.

[0053] Furthermore, a Sobel operator is added to each residual block, and feature gradients are calculated using convolution kernels in both horizontal and vertical directions, fusing the extracted edge information with the features extracted by the residual block. Haar wavelet transform is used for downsampling between each residual layer.

[0054] Specifically, Haar wavelet transform is used to replace maximum pooling for downsampling. The calculation process based on two-dimensional Haar discrete wavelet transform is as follows:

[0055]

[0056] Among them, Y L 、Y H is the low-frequency component and high-frequency component obtained by wavelet transform, Y HL 、Y LH 、Y HH These are the horizontal, vertical, and diagonal components extracted from the high-frequency components. The spatial resolution of each component subband is half that of the original image. The four component channels are concatenated and then subjected to a 1×1 convolution operation for feature compression. Batch normalization is then performed and ReLU activation is performed.

[0057] The Sobel operator consists of two 3×3 convolution kernels, which are used to calculate the gradient of an image or feature in the horizontal and vertical directions. The edge information of the image can be calculated through these two convolution kernels. X With the vertical convolution kernel G Y Respectively expressed as:

[0058]

[0059] S43: Finally, feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 scales of 64, 128, 256, and 512 channels are obtained.

[0060] S5: Construct a two-branch deformation field generation module. One branch reduces the dimension of features and generates affine transformation parameters. The other branch gradually upsamples the features to generate a two-channel optical flow field of image size, which is superimposed with the affine transformation parameters to generate the final deformation field.

[0061] S51: One branch directly performs average pooling with a pooling window of 8×8 on the feature map of size 1 / 32, flattens the pooled feature map into a one-dimensional vector, and finally maps it to 6 output values, i.e., affine transformation parameters, through a fully connected layer.

[0062] Furthermore, the affine parameters are adjusted to a 2×3 matrix and then expanded to a 3×3 homogeneous coordinate form to facilitate subsequent matrix operations.

[0063] S52: The other branch adopts a U-Net decoding structure. The first layer upsamples the 1 / 8 scale residual network features and concatenates them with the 1 / 4 scale residual network features for deconvolution. The second layer upsamples the features of the first layer and concatenates them with the 1 / 2 scale residual network features for deconvolution. The third layer upsamples the output of the second layer and directly deconvolves it. The fourth layer directly deconvolves and outputs a two-channel optical flow field.

[0064] S53: Generate a 256×256 grid to represent the position of each pixel in the image, transform each point in the grid using the affine transformation matrix, calculate the difference between the transformed point and the original point to obtain the optical flow field, and superimpose it with the optical flow field generated by the other branch to generate the final deformation field.

[0065] S6: Use the final deformation field and the spatial transformation module to perform bilinear interpolation resampling on the deformed image to obtain the registered image.

[0066] Specifically, the spatial transformation module uses the Kornia library to generate a grid representing the position of each pixel in the image, superimposes the deformation field on the grid to generate the final transformed grid, and performs bilinear interpolation resampling on the deformed image according to the transformed grid coordinates to generate the transformed registration image.

[0067] S7: Use the expanded dataset to train a bidirectional registration model to reduce the cumulative error that may be caused by single-direction registration and achieve end-to-end registration of infrared and visible light images based on phase congruency enhancement.

[0068] Specifically, the training process uses a randomly selected augmented dataset, Gaussian weight initialization, and Adam optimizer for parameter optimization. The learning rate is set to 0.000100, the weight decay is 0.00001, and training is performed for 300 epochs. The training process allows the network parameters and optimizer state to be restored from a saved model file, allowing training to continue where it was interrupted. During training, the deformed infrared image is registered to the visible light image, and the visible light image is registered to the deformed infrared image. The bidirectional registration process is supervised using a multi-loss process consisting of a bidirectional symmetry loss, an optical flow loss, and a phase loss.

[0069] Specifically, the bidirectional symmetric loss can be expressed as the sum of the affine matrix and the bidirectional symmetric loss of the optical flow field:

[0070]

[0071] L sym =L aff +L flow

[0072] Among them, H1(x,y) and H2(x,y) represent the positive affine transformation matrix and the inverse affine transformation matrix in the bidirectional registration. Under ideal estimation, the product of the two should be close to the unit matrix E; H and W are the length and width of the image, and are the predicted forward optical flow field and reverse optical flow field, It means that A is spatially transformed according to the displacement of B. Under ideal estimation, the forward optical flow field displacement should be approximately the standard grid field after being moved according to the reverse optical flow field displacement, and vice versa.

[0073] Due to modal differences, light-diverging areas in visible light images and luminous salient targets in infrared images will increase the loss, causing the network to pay more attention to these areas and ignore the texture background. Photometric loss and endpoint loss are the most commonly used loss functions in optical flow field estimation. The endpoint loss quantifies the average Euclidean distance between the predicted optical flow field and the actual optical flow field, constraining the overall smoothness of the optical flow field. Photometric loss measures the pixel-level similarity between the registered image and the template image. The two work together to address the modal differences between infrared and visible light image registration. The endpoint loss and photometric loss can be expressed as:

[0074]

[0075] in, I ir , I vi are the registered infrared image, the registered visible light image, the actual infrared image, and the actual visible light image, respectively. ||·||1 represents the l1 paradigm; is the real optical flow field, and ||·||2 represents the l2 norm.

[0076] Phase consistency loss can be expressed as:

[0077]

[0078] Where pc(·) represents the calculation of phase consistency.

[0079] In summary, the total loss can be expressed as:

[0080] L=λ1L sym +λ2L ph +λ3L ep +λ4L pc

[0081] Among them, λ1=1, λ2=λ3=10, λ4=0.3.

[0082] S8: Input the infrared image and the visible light image to be registered into the trained registration network, and directly perform end-to-end registration to obtain the infrared image registered with the visible light image.

[0083] S81: The visible light image and the infrared image to be registered are input into the registration network together. The registration network is based on four modules: phase consistency enhanced feature extraction, end-to-end parameter learning residual network, dual-branch affine parameter and optical flow field generation, and image conversion. It outputs the registered infrared image and deformation field.

[0084] S82: The registration results are quantitatively evaluated by calculating indicators such as the root mean square error and reprojection error to verify the effectiveness of the registration method.

Claims

1. A method for end-to-end registration of infrared and visible light images based on phase congruency enhancement, characterized in that: include: Paired infrared and visible light images are selected from public datasets, and random affine transformations of varying degrees are performed on the image pairs to obtain an expanded dataset. The training set and dataset of the registration network are randomly selected in proportion. A phase-consistent attention mask subnetwork is constructed, and feature extraction and feature selection are performed under the masked attention of the subnetwork. A customized ResSobelNet network architecture is used as the registration parameter learner, and Haar wavelet transform convolution is used for downsampling to reduce the information loss caused by maximum pooling. A deformation field generation and image conversion module is constructed, and a dual-branch generator is designed to generate affine parameters and optical flow fields respectively. The affine parameters are superimposed on the non-rigid deformation grid to obtain the final deformation field, and the spatial conversion module is used to resample the deformed image according to the deformation field to obtain an infrared image registered with the template image. Under the guidance of the loss function, the infrared and visible light datasets are used to train a bidirectional registration network. The infrared image and visible light image to be registered are input into the trained registration network for end-to-end registration to obtain the registration result.

2. The end-to-end registration method for infrared and visible light images based on phase congruency enhancement according to claim 1, characterized in that: Dataset expansion refers to selecting pairs of infrared and visible light images from the public dataset, and performing random affine transformations on each pair of images to varying degrees to improve the model's adaptability to unknown data.

3. The end-to-end registration method for infrared and visible light images based on phase congruency enhancement according to claim 1, characterized in that: The phase-consistent attention mask sub-network refers to combining the phase-consistent channel-spatial dual attention mechanism on the basis of continuous convolutional layers to generate enhanced masks.

4. The end-to-end registration method for infrared and visible light images based on phase congruency enhancement according to claim 3, characterized in that: Phase consistency refers to a measure of the phase similarity of each frequency component of an image, which is used as an attention guide in the sub-network to generate a consistency enhancement mask.

5. The end-to-end registration method of infrared and visible light images based on phase congruency enhancement according to claim 3, characterized in that: The phase consistency enhancement mask in the consistency feature extraction module extracts cross-modal consistency features by multiplying the shared feature points, while implicitly detecting feature edges and corners.

6. The end-to-end registration method of infrared and visible light images based on phase consistency enhancement according to claim 1 is characterized in that ResSobelNet is based on the ResNet18 network and combines the design of Sobel filters and residual blocks to effectively capture image edge textures while resisting illumination changes and noise.

7. The end-to-end registration method for infrared and visible light images based on phase congruency enhancement according to claim 1, characterized in that: Affine parameters refer to the linear transformation parameters of the deformed image to the template image position after affine transformation; the optical flow field is a displacement field that describes the local deformation from the deformed image to the template image, and each pixel corresponds to a displacement vector.

8. The end-to-end registration method for infrared and visible light images based on phase congruency enhancement according to claim 1, characterized in that: The final deformation field refers to the use of affine parameters as coarse registration parameters; the optical flow field generated by the other branch is used as the fine registration parameters, and the two are superimposed to obtain the final deformation field.

9. The end-to-end registration method for infrared and visible light images based on phase congruency enhancement according to claim 1, characterized in that: The spatial transformation module is a differentiable image resampling operation that interpolates and reconstructs the source image according to the final deformation field to generate a result that is aligned with the template image.

10. The end-to-end registration method of infrared and visible light images based on phase congruency enhancement according to claim 1, characterized in that: The infrared image and visible light image to be registered are input into the trained end-to-end registration network for consistent feature extraction, deformation field estimation and image resampling to obtain the registered infrared image.

Images