A pyramid feature based progressive matching method for infrared and visible image registration

Abstract

The application provides a pyramid feature-based progressive matching infrared and visible light image registration method, and belongs to the field of image processing and computer vision.Combining the characteristics that the pixel displacement field range of different distances of a scene is large (pixel parallax is too large at a short distance, and pixel parallax is extremely small at a long distance), a pyramid multi-scale iteration module is established, multi-modal features are extracted from coarse to fine at different resolution scales, and registration parameter estimation is carried out, so that fine registration from global to local is realized.In addition, in the case that the application scene lacks labeled data, an unsupervised deep learning framework is established, and a new loss function is designed to evaluate the registration error of the cross-modal images under the current inferred registration parameters, so that the unsupervised problem is converted into a supervised problem.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and computer vision, and relates to multimodal image representation learning and image registration method based on representation similarity. According to the multimodal image feature pyramid, a progressive matching from coarse to fine is performed to achieve high-precision registration of infrared and visible light images. Background Technology

[0002] With technological advancements, new types of sensors are constantly emerging, evolving from reliance on single sensors to the simultaneous use of multiple sensor types to acquire multimodal images with varying resolutions, temporal phases, and spectral attributes. These multimodal images reflect different target attributes and provide complementary information for perception and analysis. For example, infrared images have wavelengths longer than red light. Matter above absolute zero (-273.15℃) can produce infrared radiation. Infrared images are widely used in various fields such as military defense, resource exploration, and weather forecasting due to their ability to observe through fog and rain. Infrared images have excellent target detection and recognition capabilities and can avoid the influence of external environmental factors such as smoke, lighting, and rain. However, they also have some shortcomings, such as low pixel resolution, poor contrast, and blurred background textures. Visible light images are consistent with human visual characteristics, possessing high resolution and reflecting rich scene information, such as texture and detail. However, visible light images are easily affected by environmental factors such as weather, smoke, and obstructions; under interference, visible light images cannot highlight scene targets. To obtain more comprehensive information about the target in the image, the registration of infrared and visible light images is the most crucial step.

[0003] Existing cross-modal remote sensing image registration methods typically use traditional hand-crafted descriptors to extract multimodal image features for subsequent registration. These methods are broadly categorized into feature-based and region-based approaches. Feature-based methods first perform feature detection, identifying two sets of feature points and corresponding feature descriptors from the base image and the floating image respectively, and then perform matching based on these descriptors. Due to the significant differences in grayscale and texture between multimodal images, these hand-crafted local descriptors cannot accurately extract modality-invariant features, thus failing to achieve high-precision matching results. With the widespread application of deep learning, researchers have begun to propose using style transfer networks to align different modal images and address the modality difference problem. This transforms multimodal images into style-consistent images before performing the traditional matching process. However, this approach introduces an additional style transfer network, further increasing the computational cost of traditional methods and reducing their practicality. Furthermore, the style transfer network does not adapt to the feature requirements of the registration method, therefore it still cannot effectively remove modality differences from the feature descriptors and may even introduce significant noise, affecting subsequent matching. Newer deep learning-based methods use Siamese convolutional networks to calculate the similarity between two image patches. The learned feature descriptors are more robust to nonlinear grayscale differences, demonstrating their initial advantages in cross-modal remote sensing image matching. However, existing deep learning-based methods are simple in design and do not address features that are difficult to align between modalities in complex terrain scenes, thus affecting the calculation of similarity. In addition, these methods do not consider the high computational cost caused by the need for fine-grained, large-scale searches for high-precision pixel-level registration, making it difficult to efficiently obtain high-precision registration results.

[0004] To address the aforementioned issues and improve the accuracy and speed of infrared and visible light image registration, this invention proposes a progressive matching method for infrared and visible light image registration based on pyramid features. This invention leverages the large range of pixel displacement fields at different distances within a scene (excessive disparity for near-range pixels and minimal disparity for far-range pixels) to establish a pyramid-shaped multi-scale iterative module. This module extracts multimodal features from coarse to fine at different resolution scales and estimates registration parameters, achieving fine registration from global to local levels. Furthermore, considering the lack of labeled data in application scenarios, an unsupervised deep learning framework is proposed, and a new loss function is designed to evaluate the registration error of cross-modal images under the currently inferred registration parameters, transforming the unsupervised problem into a supervised problem. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies. It provides a progressive matching multimodal image registration method based on pyramid features. A pyramid-shaped multi-scale iterative module is established to obtain the features of different modal data after removing redundant information and aligning them. Multimodal features are extracted from coarse to fine at different resolution scales and registration parameters are estimated to achieve fine registration from global to local.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A progressive matching multimodal image registration method based on pyramid features includes the following steps:

[0008] The first step is to register the infrared image I i and visible light image I o Downsampling to different resolutions yields separate image pyramids, specifically, multi-scale optical image pyramids. Infrared image multi-scale image pyramid The shape of the multi-scale pyramid of optical images is The shape of the infrared multi-scale image pyramid is Where i∈{0,1,…,l-1} is the scale index, l is the pyramid level, and H o W o H represents the height and width of the optical image, respectively. i W i These represent the height and width of the infrared image, respectively.

[0009] The second step involves selecting the visible light-infrared image of the lowest level of the pyramid and feeding it into an unsupervised registration network composed of a feature extraction network and a registration parameter regression network to obtain the registration parameters at that resolution. Specifically:

[0010] 2.1) Select optical images from the two image pyramids respectively. Infrared images The images are fed into a feature extraction network consisting of several convolutional layers to extract optical and infrared image features.

[0011] 2.2) The extracted optical and infrared image features are concatenated and fed into the registration parameter regression network to regress coarse registration parameters.

[0012] The third step involves using the coarse registration parameters obtained in the second step. The infrared images of the higher layers of the pyramid are warped to obtain coarsely registered infrared images. This provides a global initial estimate for the next level of regression.

[0013] Specifically:

[0014] 3.1) Using the deformation function g and the coarse registration parameters obtained in the previous step. Infrared image I i Transform to a visible light viewpoint to obtain the reconstructed image.

[0015]

[0016] 3.2) The coarsely registered infrared image The input is fed into a feature extraction network consisting of several convolutional layers to obtain feature information of different dimensions. High-dimensional features contain semantic and positional information, while low-dimensional features contain more content and structural information. Therefore, a viewpoint loss is constructed on the high-dimensional features, and a content loss is constructed on the low-dimensional features. The L1 viewpoint loss formula is as follows:

[0017]

[0018] Where j represents the convolutional layer, C, W, and H are the number of feature map channels and the width and height of the j-th layer, respectively, and y is the I... i One point about the obtained characteristics, for One of the characteristics obtained.

[0019] In low dimension, compared with infrared image I i To construct the content loss, first calculate the Gram matrix.

[0020]

[0021] Where x c Let C be a point on the C channel feature, and let the Gram matrix represent the relationship between different channels in the feature. Then the content loss L2 is:

[0022]

[0023] The fourth step is to coarsely register the infrared image. Upsampled and corresponding resolution visible light image I o The data is fed into an unsupervised registration network to obtain more accurate registration parameters. Steps two and three are repeated until the original resolution is restored, resulting in the final accurate registration parameters. The infrared image is then warped to obtain the final infrared image registration result.

[0024] The fifth step is to perform coarse registration on the infrared images of each layer. and corresponding resolution infrared image I i The unsupervised loss L1+L2 is applied between the two networks to provide supervised learning for the unsupervised registration network.

[0025] The beneficial effects of this invention are as follows:

[0026] (1) This invention proposes a progressive matching multimodal image registration method based on pyramid features, which realizes a robust and fast multimodal image pixel-level accurate registration method from global to local.

[0027] (2) This method employs a pyramid-shaped multi-scale iterative scheme to progressively estimate registration parameters, effectively addressing the large displacement problem. This framework utilizes a multi-level matching strategy, from coarse to fine, leveraging multi-scale pyramid features to first perform a global coarse search, progressively locating the region, and then a fine search. This effectively avoids searching every pixel within the region and also avoids the missed matching phenomenon caused by overly strong prior regions. Attached Figure Description

[0028] Figure 1 This is a flowchart of the overall solution;

[0029] Figure 2 This is a diagram showing the results of the proposed solution. Figure 2 (a) is the visible light image to be registered; Figure 2 (b) is an infrared image; Figure 2 (c) is the visible light image after registration.

[0030] Detailed Implementation Methods (In practice)

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention are described in further detail below:

[0032] A progressive matching method for multimodal image registration based on pyramid features. Figure 1 This outlines the overall process of the solution, including the following steps:

[0033] First step, such as Figure 2 As shown in (a) and (b), the infrared image I to be registered i and visible light image I o Downsampling to different resolutions yields separate image pyramids, specifically, multi-scale optical image pyramids. Infrared image multi-scale image pyramid The shape of the multi-scale pyramid of optical images is The shape of the infrared multi-scale image pyramid is Where i∈{0,1,…,l-1} is the scale index, l is the pyramid level, and H o W o H represents the height and width of the optical image, respectively. i W i These represent the height and width of the infrared image, respectively.

[0034] The second step involves selecting the visible light-infrared image of the lowest level of the pyramid and feeding it into an unsupervised registration network composed of a feature extraction network and a registration parameter regression network to obtain the registration parameters at that resolution. Specifically:

[0035] 2.1) Select optical images from the two image pyramids respectively. Infrared images The images are fed into a feature extraction network consisting of several convolutional layers to extract optical and infrared image features.

[0036] 2.2) The extracted optical and infrared image features are concatenated and fed into the registration parameter regression network to regress coarse registration parameters.

[0037] The third step involves using the coarse registration parameters obtained in the second step. The infrared images of the higher layers of the pyramid are warped to obtain coarsely registered infrared images. This provides a global initial estimate for the next level of regression.

[0038] Specifically:

[0039] 3.1) Using the deformation function g and the coarse registration parameters obtained in the previous step. Infrared image I i Transform to a visible light viewpoint to obtain the reconstructed image.

[0040]

[0041] 3.2) The coarsely registered infrared image The input is fed into a feature extraction network consisting of several convolutional layers to obtain feature information of different dimensions. High-dimensional features contain semantic and positional information, while low-dimensional features contain more content and structural information. Therefore, a viewpoint loss is constructed on the high-dimensional features, and a content loss is constructed on the low-dimensional features. The L1 viewpoint loss formula is as follows:

[0042]

[0043] Where j represents the convolutional layer, C, W, and H are the number of feature map channels and the width and height of the j-th layer, respectively, and i represents I. i One point about the obtained characteristics, for One of the characteristics obtained.

[0044] In low dimension, compared with infrared image I i To construct the content loss, first calculate the Gram matrix.

[0045]

[0046] Where x c Let C be a point on the C channel feature, and let the Gram matrix represent the relationship between different channels in the feature. Then the content loss L2 is:

[0047]

[0048] The fourth step is to coarsely register the infrared image. Upsampled and corresponding resolution visible light image I o The data is fed into an unsupervised registration network to obtain more accurate registration parameters. Steps two and three are repeated until the original resolution is restored, yielding the final accurate registration parameters for the infrared image I. i The final infrared image registration result is obtained by warping. Figure 2 (c) Visible light image after registration.

[0049] The fifth step is to perform coarse registration on the infrared images of each layer. and corresponding resolution infrared image I i The unsupervised loss L1+L2 is applied between the two networks to provide supervised learning for the unsupervised registration network.

[0050] The embodiments described above are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A progressive matching method for infrared and visible light image registration based on pyramid features, characterized in that, Includes the following steps: The first step is to register the infrared image to be registered. and visible light images Downsampling to different resolutions yields separate image pyramids, specifically, multi-scale optical image pyramids. Infrared image multi-scale image pyramid The shape of a multi-scale pyramid in optical images The shape of the infrared multi-scale image pyramid is ;in, For scale indexing, The number of pyramid levels. These are the height and width of the optical image, respectively. These are the height and width of the infrared image, respectively; The second step is to select the visible light-infrared image of the bottom layer of the pyramid and feed it into an unsupervised registration network composed of a feature extraction network and a registration parameter regression network to obtain the registration parameters at that resolution. The third step involves using the coarse registration parameters obtained in the second step. infrared images Transform to a visible light viewpoint to obtain a coarsely registered infrared image. This provides a global initial estimate for the next level of regression; the fourth step is to coarsely register the infrared images. Upsampled and corresponding resolution visible light images The data is fed into an unsupervised registration network to obtain more accurate registration parameters; the second and third steps are repeated until the original resolution is restored to obtain the final accurate registration parameters; the infrared image is then deformed to obtain the final infrared image registration result. The fifth step is to perform coarse registration on the infrared images of each layer. and infrared images of corresponding resolution Unsupervised loss between ,in Indicates viewpoint loss. This represents content loss, and supervised learning is performed on the unsupervised registration network.

2. The progressive matching infrared and visible light image registration method based on pyramid features according to claim 1, characterized in that, The second step is as follows: 2.1) Select optical images from the two image pyramids respectively. Infrared images The images are fed into a feature extraction network consisting of several convolutional layers to extract optical and infrared image features. 2.2) The extracted optical and infrared image features are concatenated and fed into the registration parameter regression network to regress coarse registration parameters. .

3. The progressive matching infrared and visible light image registration method based on pyramid features according to claim 1, characterized in that, The third step is as follows: 3.1) Using transformation functions and the coarse registration parameters obtained in the second step infrared images Transform to a visible light viewpoint to obtain a coarsely registered infrared image. ; ； 3.2) The coarsely registered infrared image The input is fed into a feature extraction network consisting of several convolutional layers to obtain feature information of different dimensions; High-dimensional features contain semantic and positional information, while low-dimensional features contain more content and structural information; therefore, a viewpoint loss is constructed on the high-dimensional features and a content loss is constructed on the low-dimensional features; viewpoint loss The formula is as follows: ； Where j represents the convolutional layer, and C, W, and H are the number of feature map channels and the width and height of the j-th layer, respectively. for One point about the obtained characteristics, for One point about the obtained characteristics; Low-dimensional and infrared images To construct the content loss, first calculate the Gram matrix. ； in Representing a point on a C-channel feature, the Gram matrix represents the relationship between different channels in the feature; then the content loss... for: 。

Images