Rapid and accurate infrared and visible light image registration method based on deep learning

Abstract

The invention relates to a fast and accurate infrared and visible light image registration method based on deep learning. Firstly, an infrared image and a visible light image are respectively collected for the same or similar scene; secondly, preprocessing the infrared image and the visible light image, namely fusing the infrared image and the visible light image and extracting feature points and descriptors by using a deep learning method; thirdly, establishing a registration network model which comprises an input part, a backbone network and a prediction head, inputting the feature points and the descriptors into the registration network model, performing registration from coarse to fine, and finally outputting a registration result; the method has the advantages of being high in migration generalization ability, short in registration time and accurate, and the engineering application ability of an image registration algorithm is enhanced.

Description

Technical Field

[0001] The present invention relates to a fast and accurate infrared and visible light image registration method based on deep learning. To address the problems of slow speed and low precision in infrared and visible light image registration, the method adopts the idea of ​​coarse-grained to fine-grained registration and local to global registration, thereby improving the speed and precision of infrared and visible light image registration and improving the value of the infrared and visible light image registration method in engineering applications. Background Art

[0002] Infrared and visible light image registration refers to the registration of the same or similar areas captured by different imaging devices. With the development of computer technology, more and more fields need to apply image registration methods, such as judging whether cells are different from normal cells and whether they are cancerous in medicine. In the image registration method, first, it is necessary to find feature points including corner points, edge points, midpoints of line segments and points where pixel values ​​change sharply, and then establish descriptors for these feature points, where the descriptors are vectors. Secondly, calculate the distance between the descriptors in the two images, set a threshold to filter out the descriptors that meet the requirements, and finally, find the corresponding feature points and correspond them to complete the registration process. In the traditional registration method for infrared and visible light images, the method based on manual design can guarantee a certain degree of accuracy, but it brings great challenges to the accuracy of texture images and cross-domain images. Whether the expected performance of the registration effect can still be guaranteed is the key to the image registration problem. The method based on deep learning is difficult to train and slow, which makes the research more practical. In addition, the image registration process is also affected by the uncertainty of the registration method, data noise, cross-modal and other internal interferences and external interferences. Under the influence of multi-source interference, the accuracy of the registration method must still be guaranteed, which puts great demands on the anti-interference ability of the registration method of infrared and visible light images. On the basis of ensuring accuracy, how to improve the speed of infrared and visible light images has become an urgent problem to be solved. The registration methods of infrared and visible light images are mainly divided into two categories: one is for data processing, so that the input infrared and visible light images are at the same color depth and then use traditional methods for registration; the other is for registration methods that are currently dominated by deep learning methods. In order to meet the accuracy and speed requirements of the medical field, image fusion technology is first used to make the infrared image and the visible light image at the same color depth, which provides the necessary conditions for solving the data interference caused by different modal images. The processed data is further input into the tracking first and then registration module and the local first and then global registration module, which ultimately improves the speed and accuracy of infrared and visible light images. Summary of the invention

[0003] The technical problem solved by the present invention is: to provide a method that takes both accuracy and speed into consideration in view of the low accuracy and slow speed of infrared and visible light image registration, thereby enhancing the engineering application capability of infrared and visible light image registration.

[0004] The technical solution of the present invention is: first, infrared images and visible light images are collected respectively for the same or similar scenes; second, the infrared images and visible light images are preprocessed, that is, the infrared images are fused with the visible light images; third, the feature points are extracted using a deep learning method and descriptors are established for the feature points; then, a registration network model is established, including an input part, a backbone network and a prediction head, the feature points and descriptors are input into the registration network model, and a coarse-to-fine registration is performed, and finally the registration results are output to complete the registration method of infrared and visible light images.

[0005] The specific steps are: The first step is to fuse the infrared image and the visible light image to extract the feature points and descriptors of the visible light image and the fused image. The second step is to design a tracking-then-registration module into which feature points and descriptors are input.

[0006] Among them, A is the affine transformation matrix, F is the feature detection model in deep learning, T is the tracking algorithm function, and E is the affine matrix calculation function. is the input image; The third step is to design a local-first-global registration module, align the images using the obtained affine transformation matrix, and input it into the high-level module. Given a feature point Match the feature points on another image Set a radius r around the two feature points and define the area as The distance of the feature points within the calculation radius is as follows:

[0007] For matching points The estimated affine transformation matrix is ​​as follows:

[0008] For matching points not found locally, a global search is performed.

[0009] Using all matching point pairs, including the results of global and local matching, the overall transformation matrix A is estimated as follows:

[0010] The entire registration process can be viewed as a function R, which maps the fused image and the visible light image to the final transformation matrix A as follows: Where: D is the descriptor function, is in the visible light image Matching points; The fourth step is to output the prediction results and complete the registration of infrared and visible light images. Compared with the prior art, the advantages of the present invention are as follows: the infrared and visible light image registration method of the present invention analyzes and models the infrared and visible light image data registration method in view of the different characteristics of the two types of image data, which makes the registration difficult, and then solves the registration problem of the two types of images by designing image fusion, tracking first and then registration, and local first and then overall registration modules, thereby improving the speed and accuracy of infrared and visible light image registration; the infrared and visible light image registration method designed by the present invention greatly improves the registration capability, has the characteristics of fast speed and high accuracy, and enhances the engineering application capability of the infrared and visible light image registration method. The engineering application capability of the infrared and visible light image registration method is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 The present invention provides a fast and accurate infrared and visible light image registration method based on deep learning. DETAILED DESCRIPTION

[0012] like Figure 1 As shown, the specific implementation steps of the present invention are as follows (the following takes early screening of breast cancer in medical imaging as an example to illustrate the specific implementation of the method): 1. Fuse the infrared image and the visible light image, and extract the feature points and descriptors of the visible light image and the fused image Input: infrared image size 512*512, visible light image size 512*512, number of feature points 1000, descriptor is 128-dimensional vector 2. Design a tracking-then-registration module and input feature points and descriptors into it The transformation matrix A is obtained to align the image in a coarse-grained manner as follows: 3. Design a local-first-global registration module One of the feature points is (254,316) and matches the feature point (167,351) on the other image. A radius of 10 is set around the two feature points, and the definition area is The distance of the feature points within the calculation radius is as follows: For matching points The estimated affine transformation matrix is ​​as follows: For matching points not found locally, a global search is performed.

[0013] Using all matching point pairs, including the results of global and local matching, the overall transformation matrix A is estimated as follows: The entire registration process can be viewed as a function R, which maps the fused image and the visible light image to the final transformation matrix A as follows:

[0014] Where: D is the descriptor function; 4. Output the prediction results and complete the registration of infrared and visible light images The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. Claim 1 A fast and accurate infrared and visible light image registration method based on deep learning, characterized by The following steps are involved: The first step is to fuse the infrared image and the visible light image and extract feature points and descriptors; The second step is to design a tracking-then-registration module, which uses the tracking algorithm to quickly make a preliminary estimate of the aligned images. The third step is to design a local-first-global registration module. A pair of matching feature points are selected on the aligned image, and a radius is set around the feature points. The matching points within the radius are matched locally. After the matching is completed, the unmatched points are matched globally. The fourth step is to build the backbone network and predict the head of the registration network model. The backbone network includes a tracking-then-matching module and a local-then-global module, and the head is predicted to output the registration result. Claim 2 The fast and accurate infrared and visible light image registration method based on deep learning according to claim 1 is characterized in that: in the second step, the tracking-then-registration module is designed as follows:

2. Among them, A is the affine transformation matrix, F is the feature detection model in deep learning, T is the tracking algorithm function, and E is the affine matrix calculation function. is the input image; Claim 3 The fast and accurate infrared and visible light image registration method based on deep learning according to claim 1 is characterized in that: in the third step, the local-first-global registration module is designed as follows: Given a feature point Match the feature points on another image Set a radius r around the two feature points and define the area as ,The distance of the feature points within the calculation radius is as follows:

3. For matching points The estimated affine transformation matrix is ​​as follows:

4. For matching points not found locally, perform a global search.

5. Using all matching point pairs, including the results of global and local matching, estimate the overall transformation matrix A as follows:

6. The entire registration process can be viewed as a function R, which maps the fused image and the visible light image to the final transformation matrix A as follows: in: D is the descriptor function, is in the visible light image Matched points.

Images