Unsupervised infrared and visible light image registration method based on matching cost

Abstract

The invention discloses an unsupervised infrared and visible light image registration method based on matching cost. The unsupervised infrared and visible light image registration method comprises the following steps: acquiring an infrared image and visible light image pair; constructing a cross-modal style conversion module of a U-Net architecture and performing style conversion on the visible light image to obtain a stylized visible light image; constructing an unsupervised registration network based on the matching cost, inputting the infrared image and the stylized visible light image into the unsupervised registration network, calculating the matching cost to evaluate the registration difference between the images, and optimizing the deformation field by minimizing the matching cost to obtain a predicted deformation field; and performing resampling operation on the infrared image based on the predicted deformation field to obtain a registered infrared image. According to the method, through cross-modal style conversion and a light sensation information weighting strategy, the modal difference is remarkably reduced, and the registration precision is improved; the multi-level refining design and the comprehensive loss function optimization of the registration network ensure the stability and the robustness of the registration result, and the method is suitable for registration application in complex scenes.

Description

Technical Field

[0001] The present invention relates to the field of optoelectronic imaging technology, and in particular to an unsupervised infrared and visible light image registration method based on matching cost. Background Art

[0002] The registration of infrared and visible light images is widely used in fields such as military reconnaissance, traffic monitoring, security systems, and aerial photography. Infrared and visible light images reflect different physical properties of a scene: infrared images capture thermal radiation, while visible light images capture reflected light. Due to significant differences in detail, contrast, and noise characteristics, effectively registering these two images for subsequent processing is extremely challenging.

[0003] Traditional registration algorithms can be categorized as feature-based and region-based. These methods typically rely on grayscale similarity or feature matching between images to achieve image registration. However, when dealing with complex scenes and multimodal images, traditional methods struggle to accurately align detailed features and are sensitive to noise and illumination variations. In recent years, deep learning-based registration methods have significantly improved alignment accuracy by leveraging models such as convolutional neural networks (CNNs) to learn complex mapping relationships between images. However, these methods typically rely on large-scale annotated data and have limited generalization capabilities in unsupervised scenarios, which limits their widespread deployment in practical applications.

[0004] Most existing methods fail to fully consider the differences in physical properties between infrared and visible light images, resulting in loss of detail and incomplete target information in multimodal image registration. Furthermore, existing methods are limited in computational efficiency and real-time performance, making them difficult to meet the demands for fast, high-precision registration in practical applications. Therefore, designing a novel unsupervised registration model that more precisely accounts for light-sensing information and pixel displacement while simultaneously accounting for the differences in physical properties between infrared and visible light images has become a pressing technical challenge in this field. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention proposes an unsupervised infrared and visible light image registration method based on matching cost. The visible light image is converted into a stylized infrared image through a cross-modal style transfer module, which reduces modal differences and improves the accuracy of feature matching. The registration network adopts a multi-level refinement strategy, and realizes coarse-to-fine deformation field estimation through a hierarchical feature decoupling extractor and a multi-dimensional information reconstruction decoder to ensure registration accuracy and robustness. Through a light information weighting strategy, the spatial position offset and light brightness change are quantified to optimize the registration difference. This strategy combines local brightness and contrast to design a weight function, which significantly improves the registration accuracy, especially in scenes with large lighting changes.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] The unsupervised infrared and visible light image registration method based on matching cost includes the following steps:

[0008] Acquire infrared image and visible light image pairs;

[0009] Construct a cross-modal style transfer module based on the U-Net architecture, perform style transfer on visible light images based on the cross-modal style transfer module, and obtain stylized visible light images;

[0010] An unsupervised registration network based on matching cost is constructed. The infrared image and the stylized visible light image are input into the unsupervised registration network. The matching cost is calculated to evaluate the registration difference between the images. The deformation field is optimized by minimizing the matching cost to obtain the predicted deformation field.

[0011] The infrared image is resampled based on the predicted deformation field to obtain a registered infrared image.

[0012] Preferably, the cross-modal style transfer module is optimized by a multi-task loss function, and the multi-task loss function includes:

[0013] Structural consistency loss is used to ensure that the global structure of the generated image is aligned with the infrared image. The formula is as follows:

[0014]

[0015] Among them, ψ VGG Represents the feature representation extracted from the pre-trained VGG network, i is the index variable of the summation symbol, which is used to traverse all pixels, i iterates from 1 to N, indicating that each pixel or feature point is calculated, N is the number of pixels, I style To stylize visible light images, I ir is an infrared image;

[0016] Style loss is used to encourage the generated image to retain the texture details of the visible light image while also retaining the thermal radiation characteristics of the infrared image. The formula is as follows:

[0017]

[0018] Among them, G l represents the Gram matrix of the lth layer;

[0019] Edge preservation loss is used to ensure that the edges and details in the generated image are clear and consistent with the infrared image. The formula is as follows:

[0020]

[0021] in, represents the gradient operator.

[0022] Preferably, the calculation of the matching cost includes the following steps:

[0023] Divide the infrared image and the stylized visible light image into several local regions;

[0024] The spatial position offset cost and light sensitivity brightness transformation cost are calculated for each local area in the segmented infrared image and stylized visible light image in turn;

[0025] The spatial position offset cost C spatial and the light sensitivity brightness conversion cost C intensity Perform weighted summation to obtain the matching cost, the formula is as follows:

[0026] C total =C spatial +α·C intensity

[0027] Among them, α is the weight coefficient.

[0028] Preferably, the spatial position offset cost C spatial , the calculation formula is as follows:

[0029]

[0030] Among them, N represents the total number of divided local areas. In the i-th local area R i , I ir (x,y) is the pixel value of the infrared image at position (x,y), I style (x,y) is the pixel value of the stylized visible light image at position (x,y), is the average pixel value of the infrared image in the entire image, is the average pixel value of the stylized visible light image in the entire image, |R i | is the local area R i The total number of pixels in ;

[0031] The light sensitivity brightness conversion cost C intensity , the calculation formula is as follows:

[0032]

[0033] Preferably, the following step is further included: constructing a total loss function of the unsupervised registration network, wherein the total loss function is a weighted sum of bidirectional similarity loss, smoothness loss, and cross-modal consistency loss, and the formula is as follows:

[0034] L total =λ sim Lsim +λ smooth L smooth +λ cmc L cmc

[0035] Among them, λ sim ,λ smooth and λ cmc are the weight coefficients of bidirectional similarity loss, smoothness loss and cross-modal consistency loss, respectively, and L sim is the bidirectional similarity loss, L smooth is the smoothing loss, L cmc is the cross-modal consistency loss.

[0036] Preferably, the bidirectional similarity loss L sim , which is used to align the distorted infrared image and the stylized visible light image at the feature level. The formula is as follows:

[0037] L sim =‖ψ(I ir )-ψ(I RGB )‖1+‖ψ(I RGB )-ψ(I ir )‖1

[0038] Where ψ(·) represents the feature extractor, I RGB is a visible light image;

[0039] The smoothing loss L smooth , used to ensure the smoothness of the deformation field, the formula is as follows:

[0040]

[0041] in, and denote the first and second order derivatives respectively;

[0042] The cross-modal consistency loss L cmc , which is used to measure the similarity between the registered infrared image and the infrared image. The formula is as follows:

[0043] L cmc =‖Θ(I ir )-Θ(I RGB )‖1

[0044] where Θ(·) denotes the feature extractor used for cross-modal comparison.

[0045] Preferably, the method further includes the following steps: determining an evaluation index for registration evaluation, and evaluating the image registration effect based on the evaluation index.

[0046] Preferably, the evaluation indicators of the registration evaluation are mean square error (MSE), normalized correlation coefficient (NCC), mutual information (MI) and matching completion rate.

[0047] Based on the above technical solution, the beneficial effects of the present invention are as follows: the present invention is an unsupervised image registration method based on matching cost, which generates stylized visible light images through cross-modal style conversion module conversion, and combines the light-sensing information weighted matching cost and multi-level refined unsupervised registration network to achieve high-precision deformation field estimation. The network is optimized using bidirectional similarity loss, smoothness loss and cross-modal consistency loss, with the aim of solving the registration misalignment problem caused by imaging differences in multimodal images, improving registration accuracy and enhancing robustness to illumination changes and noise, while retaining more image details, providing a better alignment basis for subsequent image fusion. This innovative method significantly improves the performance of multimodal image registration and is suitable for a wide range of practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of an unsupervised infrared and visible light image registration method based on matching cost in one embodiment;

[0049] Figure 2 is a schematic structural diagram of a modality style transfer module in an unsupervised infrared and visible light image registration method based on matching cost in one embodiment;

[0050] Figure 3 The diagram is a structural diagram of an unsupervised registration network in an unsupervised infrared and visible light image registration method based on matching cost in one embodiment. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0052] See also Figures 1 to 3 This embodiment provides an unsupervised infrared and visible light image registration method based on matching cost, comprising the following steps:

[0053] S1. Acquire an infrared image and a visible light image pair;

[0054] S2. Build a cross-modal style transfer module based on the U-Net architecture, perform style transfer on the visible light image based on the cross-modal style transfer module, and obtain a stylized visible light image.

[0055] S3. Construct an unsupervised registration network based on matching cost. Input the infrared image and the stylized visible light image into the unsupervised registration network, calculate the matching cost to evaluate the registration difference between the images, and optimize the deformation field by minimizing the matching cost to obtain the predicted deformation field.

[0056] S4. Perform a spatial transformation operation on the infrared image based on the predicted deformation field to obtain a registered infrared image.

[0057] In an embodiment of the unsupervised infrared and visible light image registration method based on matching cost, a cross-modal style conversion module is provided to perform a specific process of grid conversion. The cross-modal style conversion module adopts a U-Net architecture, including residual blocks and jump connections. The network converts the visible light image into a stylized visible light image I style , so that it is structurally consistent with the source infrared image I ir To maintain consistency while preserving the texture and edge details of the visible light image, the formula is as follows:

[0058] I style =G(I RGB )

[0059] Among them, I RGB is the input visible light image, I style is the generated stylized visible light image, G is the cross-modal style transfer module

[0060] The cross-modal style transfer module is optimized through a multi-task loss function, which includes:

[0061] Structural consistency loss, which extracts features through the pre-trained VGG network and calculates the L2 norm, is used to ensure that the global structure of the generated image is aligned with the target infrared image. The formula is as follows:

[0062]

[0063] Among them, ψ VGG Represents the feature representation extracted from the pre-trained VGG network, i is the index variable of the summation symbol, which is used to traverse all pixels, i iterates from 1 to N, indicating that each pixel or feature point is calculated, N is the number of pixels, I style To stylize visible light images, I ir is an infrared image;

[0064] Style loss is a loss that obtains style features based on the Gram matrix. The difference between the two Gram matrices is calculated using the L2 norm. This loss is used to ensure that the generated image retains the texture details of the visible light image while also retaining the thermal radiation characteristics of the infrared image. The formula is as follows:

[0065]

[0066] Among them, G l represents the Gram matrix of the lth layer;

[0067] Edge preservation loss, which enhances edge consistency through the Sobel operator, is used to ensure that the edges and details in the generated image are clear and consistent with the source infrared image. The formula is as follows:

[0068]

[0069] in, represents the gradient operator.

[0070] By optimizing the above loss function, the cross-modal style transfer module can generate high-quality stylized visible light images, providing a better foundation for subsequent registration tasks.

[0071] In an embodiment of the unsupervised infrared and visible light image registration method based on matching cost, a process for calculating the matching cost is provided, including the following steps:

[0072] During the registration process, the infrared image and the stylized visible light image are divided into several local regions;

[0073] The spatial position offset cost and light sensitivity brightness transformation cost are calculated for each local area in the segmented infrared image and stylized visible light image in turn;

[0074] The calculation formula of spatial position offset cost is as follows:

[0075]

[0076] Among them, N represents the total number of divided local areas. In the i-th local area R i , calculate the spatial position offset cost between the infrared image and the stylized visible light image, I ir (x,y) is the pixel value of the infrared image at position (x,y), I style (x,y) is the pixel value of the stylized visible light image at position (x,y), is the average pixel value of the infrared image in the entire image, is the average pixel value of the stylized visible light image in the entire image, |R i | is the local area R i The total number of pixels in .

[0077] The calculation formula for light sensitivity brightness transformation cost is as follows:

[0078]

[0079] Among them, N represents the total number of divided local areas. In the i-th local area R i , calculate the light sensitivity brightness transformation cost between infrared image and stylized visible light image, I ir(x,y) is the pixel value of the infrared image at position (x,y), I style (x,y) is the pixel value of the stylized visible light image at position (x,y), |R i | is the local area R i The total number of pixels in .

[0080] The spatial position offset cost C spatial and the light sensitivity brightness conversion cost C intensity Perform weighted summation to obtain the matching cost:

[0081] C total =C spatial +α·C intensity

[0082] Among them, α is the weight coefficient obtained by training.

[0083] By minimizing the matching cost, the unsupervised registration network can accurately calculate the registration parameters and adjust the infrared image to the best alignment with the visible light image.

[0084] In an embodiment of the unsupervised infrared and visible light image registration method based on matching cost, the unsupervised registration network based on matching cost evaluates the registration difference between images and optimizes the deformation field by calculating the matching cost. The structure diagram of the unsupervised registration network is shown in FIG. Figure 3 As shown, it includes the following key modules, specifically:

[0085] (1) Hierarchical feature decoupling extraction module

[0086] The hierarchical feature decoupling extraction module extracts multi-scale features from infrared images and visible light images independently to avoid feature coupling. Its output is a multi-scale feature pyramid:

[0087] For the source infrared image I ir and visible light image I RGB , extract features F respectively IR and F RGB The feature extraction formula is:

[0088]

[0089] Among them, k represents the scale level of the feature pyramid, and H_conv represents the hierarchical feature decoupling extraction convolutional network.

[0090] (2) Multi-dimensional information reconstruction decoding module

[0091] The multi-dimensional information reconstruction decoding module gradually decodes high-level semantic features into low-level detail information and estimates the deformation field. Its output is a series of deformation fields:

[0092] Top-level features and After cascade integration, the multi-dimensional information is input into the top layer of the reconstruction decoding module M_conv to generate the initial deformation field φ (K) The generation formula of the deformation field is:

[0093]

[0094] (3) Deformation field collaborative optimization module

[0095] The deformation field collaborative optimization module deeply integrates and collaboratively optimizes deformation fields of different scales to generate a more refined deformation field:

[0096] Through the neural network D_conv operation, the initial deformation field φ (K) Optimized to a finer deformation field φ (K-1) .

[0097] The optimization formula is:

[0098] φ (K-1) =D_conv(φ (K) )

[0099] Where D_conv represents the deformation field collaborative optimization convolutional network.

[0100] (4) Feature fusion and refinement module

[0101] The feature fusion and refinement module deeply fuses and refines features to improve feature expression capabilities:

[0102] For feature F IR and F RGB Perform deep fusion to generate pure features F refined .

[0103] The feature fusion formula is:

[0104] F refined =F_conv(F IR ,F RGB )

[0105] Where F_conv represents the feature fusion and refinement convolutional network.

[0106] In an unsupervised infrared and visible light image registration method based on matching cost in one embodiment, a total loss function of the unsupervised registration network is provided. The total loss function is a weighted sum of bidirectional similarity loss, smoothness loss, and cross-modal consistency loss, and is expressed as follows:

[0107] L total =λ sim L sim +λ smooth Lsmooth +λ cmc L cmc

[0108] Among them, λ sim ,λ smooth and λ cmc are the weight coefficients of bidirectional similarity loss, smoothness loss and cross-modal consistency loss, respectively, and L sim is the bidirectional similarity loss, L smooth is the smoothing loss, L cmc is the cross-modal consistency loss, specifically,

[0109] Bidirectional similarity loss L sim The pre-trained CNN is used to extract features and calculate the L1 distance, which is used to align the distorted infrared image and the stylized visible light image at the feature level. The formula is as follows:

[0110] L sim =‖ψ(I ir )-ψ(I RGB )‖1+‖ψ(I RGB )-ψ(I ir )‖1

[0111] Where ψ(·) represents the feature extractor, I RGB is a visible light image;

[0112] Smoothing loss L smooth , used to ensure the smoothness of the deformation field, the formula is as follows:

[0113]

[0114] in, and denote the first and second order derivatives respectively;

[0115] Cross-modal consistency loss L cmc , which is used to measure the similarity between the registered infrared image and the infrared image. The formula is as follows:

[0116] L cmc =‖Θ(I ir )-Θ(I RGB )‖1

[0117] where Θ(·) denotes the feature extractor used for cross-modal comparison.

[0118] In one embodiment, the unsupervised infrared and visible light image registration method based on matching cost also includes determining an evaluation index for registration evaluation and evaluating the image registration effect based on the evaluation index.

[0119] In this embodiment, the evaluation indicators of the registration evaluation are determined, which are the mean square error (MSE), the normalized correlation coefficient (NCC), the mutual information (MI), and the matching completion rate. The evaluation is tested on the RoadScene, M3FD, and FLIR datasets.

[0120] Table 1 Test experiment evaluation index results

[0121]

[0122] As shown in Table 1, the experimental results show that the method of this embodiment outperforms other methods (SURF accelerated robust feature algorithm, SIFT scale-invariant feature transformation algorithm, Voxelmorph voxel morphological registration algorithm, DFM deep feature registration algorithm and Ne_MAR neural network-based multimodal adversarial registration algorithm) in terms of mean square error (MSE), normalized correlation coefficient (NCC), mutual information (MI) and matching completion rate. Specifically:

[0123] 1) The low MSE value (0.0027) indicates high pixel-level alignment accuracy, which is attributed to the multi-level refinement registration strategy adopted in this embodiment, which can effectively handle complex deformations;

[0124] 2) High NCC values (0.965) and MI values (1.833) indicate excellent image content consistency and information fidelity, thanks to the cross-modal style transfer module, which converts visible light images into stylized visible light images, bridging the modality gap.

[0125] 3) The high matching completion rate of 97.5% demonstrates the robustness of the method of this embodiment in different scenarios, mainly due to the light information weighting strategy, which optimizes the registration difference calculation and adapts to different lighting conditions.

[0126] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0127] The foregoing description is merely a preferred embodiment of the unsupervised infrared and visible light image registration method based on matching cost disclosed herein and is not intended to limit the scope of protection of the embodiments herein. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the embodiments herein shall be included within the scope of protection of the embodiments herein.

Claims

1. Unsupervised infrared and visible light image registration method based on matching cost, characterized by: The steps include: Acquire infrared image and visible light image pairs; Construct a cross-modal style transfer module based on the U-Net architecture, perform style transfer on visible light images based on the cross-modal style transfer module, and obtain stylized visible light images; An unsupervised registration network based on matching cost is constructed. The infrared image and the stylized visible light image are input into the unsupervised registration network. The matching cost is calculated to evaluate the registration difference between the images. The deformation field is optimized by minimizing the matching cost to obtain the predicted deformation field. The infrared image is resampled based on the predicted deformation field to obtain a registered infrared image.

2. The unsupervised infrared and visible light image registration method based on matching cost according to claim 1, characterized in that: The cross-modal style transfer module is optimized by a multi-task loss function, which includes: Structural consistency loss is used to ensure that the global structure of the generated image is aligned with the infrared image. The formula is as follows: Among them, ψ VGG Represents the feature representation extracted from the pre-trained VGG network, i is the index variable of the summation symbol, which is used to traverse all pixels, i iterates from 1 to N, indicating that each pixel or feature point is calculated, N is the number of pixels, I style To stylize visible light images, I ir is an infrared image; Style loss is used to encourage the generated image to retain the texture details of the visible light image while also retaining the thermal radiation characteristics of the infrared image. The formula is as follows: Among them, G l represents the Gram matrix of the lth layer; Edge preservation loss is used to ensure that the edges and details in the generated image are clear and consistent with the infrared image. The formula is as follows: in, represents the gradient operator.

3. The unsupervised infrared and visible light image registration method based on matching cost according to claim 1, characterized in that: The calculation of the matching cost includes the following steps: Divide the infrared image and the stylized visible light image into several local regions; The spatial position offset cost and light sensitivity brightness transformation cost are calculated for each local area in the segmented infrared image and stylized visible light image in turn; The spatial position offset cost C spatial and the light sensitivity brightness conversion cost C intensity Perform weighted summation to obtain the matching cost, the formula is as follows: C total =C spatial +α·C intensity Among them, α is the weight coefficient.

4. The unsupervised infrared and visible light image registration method based on matching cost according to claim 3, characterized in that: The spatial position offset cost C spatial , the calculation formula is as follows: Among them, N represents the total number of divided local areas. In the i-th local area R i , I ir (x,y) is the pixel value of the infrared image at position (x,y), I style (x,y) is the pixel value of the stylized visible light image at position (x,y), is the average pixel value of the infrared image in the entire image, is the average pixel value of the stylized visible light image in the entire image, |R i | is the local area R i The total number of pixels in ; The light sensitivity brightness conversion cost C intensity , the calculation formula is as follows:

5. The unsupervised infrared and visible light image registration method based on matching cost according to claim 1, characterized in that: The method further includes the following steps: constructing a total loss function of the unsupervised registration network, wherein the total loss function is a weighted sum of bidirectional similarity loss, smoothness loss, and cross-modal consistency loss, and the formula is as follows: L total =λ sim L sim +λ smooth L smooth +λ cmc L cmc Among them, λ sim ,λ smooth and λ cmc are the weight coefficients of bidirectional similarity loss, smoothness loss and cross-modal consistency loss, respectively, and L sim is the bidirectional similarity loss, L smooth is the smoothing loss, l cmc is the cross-modal consistency loss.

6. The unsupervised infrared and visible light image registration method based on matching cost according to claim 5, characterized in that: The bidirectional similarity loss L sim , which is used to align the distorted infrared image and the stylized visible light image at the feature level. The formula is as follows: L sim =‖ψ(I ir )-ψ(I RGB )‖1+‖ψ(I RGB )-ψ(I ir )‖1 Where ψ(·) represents the feature extractor, I RGB is a visible light image; The smoothing loss L smooth , used to ensure the smoothness of the deformation field, the formula is as follows: in, and denote the first and second order derivatives respectively; The cross-modal consistency loss L cmc , which is used to measure the similarity between the registered infrared image and the infrared image. The formula is as follows: L cmc =‖Θ(I ir )-Θ(I RGB )‖1 where Θ(·) denotes the feature extractor used for cross-modal comparison.

7. The unsupervised infrared and visible light image registration method based on matching cost according to claim 1, characterized in that: The method further includes the following steps: determining an evaluation index for registration evaluation, and evaluating the image registration effect based on the evaluation index.

8. The unsupervised infrared and visible light image registration method based on matching cost according to claim 7, characterized in that: The evaluation indicators of the registration evaluation are mean square error (MSE), normalized correlation coefficient (NCC), mutual information (MI) and matching completion rate.

Images