An infrared and visible image registration method that integrates depth estimation

Abstract

This invention discloses a method for registering infrared and visible light images using depth estimation, comprising: extracting depth maps from visible light and infrared images through monocular depth estimation; fusing the obtained depth maps with the original images to generate fused image pairs; extracting features from the fused image pairs to generate initial matching feature point pairs; performing hierarchical screening of the initial matching feature point pairs, and simultaneously compensating for segmented images without matching points to obtain final matching point pairs; and transforming the infrared image based on the matching point pairs to obtain the registered image. This invention fuses depth information into RGB images, effectively improving the accuracy of infrared and visible light image registration. This invention is compatible with any imaging system composed of visible light and infrared cameras and has broad practical value.

Description

Technical Field

[0001] This invention relates to a method for registering infrared and visible light images by fusing depth estimation, belonging to the field of image registration technology. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, heterogeneous image registration has been widely applied in fields such as multimodal fusion, target detection, autonomous driving, and night vision enhancement. Registration technology between infrared and visible light images is an important research direction within this field. Infrared images primarily focus on the thermal radiation characteristics of targets, possessing strong anti-interference capabilities and all-weather adaptability, while visible light images provide rich texture and detail information. Effectively combining the two can achieve complementary advantages, providing crucial support for multimodal analysis in complex scenes.

[0003] However, due to significant modal differences between infrared and visible light images, such as inconsistencies in spectral response, texture features, and resolution, traditional registration algorithms face numerous challenges in feature extraction, matching, and geometric correction. In dynamic scenes or weakly textured backgrounds, existing methods may lead to registration errors due to feature instability. Furthermore, the high computational cost of complex algorithms limits their application in scenarios with high real-time requirements. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an infrared and visible light image registration method that integrates depth estimation. By integrating depth maps, depth information is added to the image to be registered, and feature points matched by a deep learning network are filtered through a hierarchical filtering strategy, which effectively reduces mismatches and achieves the final image registration.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] An infrared and visible light image registration method that integrates depth estimation includes the following steps:

[0007] Step 1: Preprocess the infrared and visible light images to be registered, and obtain the depth maps of the preprocessed infrared and visible light images;

[0008] Step 2: Fuse the infrared image to be registered with its corresponding depth map to obtain an infrared image containing depth information; fuse the visible light image to be registered with its corresponding depth map to obtain a visible light image containing depth information.

[0009] Step 3: Divide the infrared image and the visible light image containing depth information into rows and columns according to a preset number of rows and columns, so that the resulting infrared image blocks and visible light image blocks are the same size. Treat the infrared image blocks and visible light image blocks that are in the same position in their respective images as a pair of images.

[0010] Step 4: Extract and match feature points for each pair of images to obtain an initial set of matching point pairs;

[0011] Step 5: Set the D0 value, perform the first filtering on the initial set of matching point pairs based on D0, delete the matching point pairs that fail the filtering, and obtain the set of matching point pairs after the first filtering.

[0012] Step 6: Set the D1 value, and perform a second filtering on the set of matching point pairs after the first filtering based on D1. Delete the matching point pairs that fail the filtering, and obtain the set of matching point pairs after the second filtering.

[0013] Step 7: For image pairs that do not have matching point pairs after feature point extraction and matching in Step 4, and image pairs that do not have matching point pairs after the second screening in Step 6, a compensation mechanism is used to compensate for the matching points, so that each image pair has at least one pair of matching point pairs, and the final set of matching point pairs is obtained.

[0014] Step 8: Based on the final set of matching point pairs, perform thin strip transformation on the infrared image to be registered to obtain the registered infrared image.

[0015] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0016] 1. This invention fuses the original image with a depth map, adding depth information to the image to be registered. This effectively increases the amount of prior information for the task of selecting matching feature points, making feature point matching more accurate and registration results more reliable.

[0017] 2. This invention uses a multi-level screening mechanism to effectively check for matching errors that occur during the feature point extraction and matching stages, making the registration results more stable and reliable and improving the matching accuracy.

[0018] 3. This invention compensates for missing matching points in regions based on successfully matched feature points, which can effectively remedy the situation where multispectral image features do not correspond, making the registration algorithm highly robust.

[0019] 4. The method of the present invention has broad practical value as a pre-task for any image processing task that requires registration of multispectral images, such as target detection and target tracking. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an infrared and visible light image registration method based on depth estimation according to the present invention.

[0021] Figure 2 This is a schematic diagram of the deep feature extraction and fusion process;

[0022] Figure 3 This is a flowchart illustrating the process of filtering matching points twice. Detailed Implementation

[0023] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0024] This invention provides a method for registering infrared and visible light images by fusing depth estimation. By fusing depth image information into the original image and performing hierarchical filtering of matching points, it can effectively achieve the registration task of infrared and visible light images. Figure 1 As shown, it includes the following steps:

[0025] Step 1: Perform monocular depth estimation on the infrared and visible light images respectively to obtain their respective depth maps. In this invention, the monocular depth estimation method uses Depth-Anything-V2, utilizing its officially provided pre-trained model to obtain the relative depth image required by this invention.

[0026] It should be noted that the depth estimation method is not limited to Depth-Anything-V2; any monocular depth estimation method can be used as the depth estimation method adapted to this invention.

[0027] Step 2: Perform image fusion between the visible light image and the infrared image and their generated depth map, as follows: Figure 2 As shown, infrared and visible light images containing depth information are generated.

[0028] It should be noted that the present invention adopts a method of first extracting the contour from the depth map, and then weightedly fusing the original image, the depth map, and the contour map to achieve image fusion. This is a fusion method with low complexity. In fact, various image fusion methods such as wavelet transform image fusion, multi-scale pyramid image fusion, or even dedicated deep learning network-based image fusion methods can be used to replace this module.

[0029] Step 3: Segment the visible light and infrared images to generate a series of image pairs of suitable size for feature point extraction and matching.

[0030] Specifically, the specified values ​​of cols and rows are used to determine the number of image pairs to be segmented. Here, cols represents the number of columns into which the original image is segmented, and rows represents the number of rows into which the original image is segmented. cols * rows represents the total number of image pairs to be segmented.

[0031] Step 4: Extract and match feature points for each pair of images to obtain an initial set of matching points.

[0032] It should be noted that this invention uses a LoFTR deep learning network to collect feature matching points, performs denoising processing on each pair of images, converts the array from NumPy format to a tensor, uses the LoFTR feature matching deep learning network to extract the initial feature point set, and converts the result back to NumPy format.

[0033] In fact, classic feature point extraction and matching methods such as SIFT feature point extraction and matching and Harris corner detection can replace this module, at the cost of differences in accuracy.

[0034] Step 5: Perform the first matching point filtering: Perform D0 filtering on the matching points in each pair of images, update the matching point set, and delete the matching point pairs that did not pass the filtering.

[0035] Specifically, first, set the value of D0, and the calculation formula is as follows:

[0036] D0 = w / (8 * cols)

[0037] Where w represents the width of the infrared and visible light images, and cols represents the number of columns into which the original image is divided. Then, the length of the matching line for each pair of matching points in the images is calculated. The three pairs of matching points with the shortest matching line lengths are selected. For these three pairs of matching points, the distance between any two matching points in their respective images is calculated. If there are matching points with a distance less than D0, the matching point pair with the longer matching line is deleted, and the matching point set is updated.

[0038] Step 6: Perform a second matching point filtering: Filter the matching points that passed the first matching point filtering using D1, update the matching point set, and delete the matching point pairs that did not pass the filtering.

[0039] Specifically, the positions of the matching points that passed the first matching point screening are mapped onto the original visible light and infrared images to generate a new set of matching points. The value of D1 is set, and the calculation formula is as follows:

[0040] D1 = h / (4 * rows)

[0041] Where h represents the height of the infrared and visible light images, and rows represents the number of rows into which the original image is divided.

[0042] The absolute value of the slope of the matching line is calculated using the following formula:

[0043] grad = |(y vi -y ir ) / (x vi -x ir)|

[0044] Among them, y vi The y-coordinate represents the ordinate of the matching point in the visible light image. ir The x-coordinate represents the ordinate of the matching point in the infrared image. vi The x-coordinate represents the x-coordinate of the matching point in the visible light image. ir This represents the x-coordinate of the matching point in the infrared image. If the absolute value of the slope of the matching point is greater than a preset threshold thres, then the x-coordinate of that matching point is calculated. i ,y i The shortest distance L between other matching points i The formula for calculating the shortest distance is:

[0045]

[0046] If the distance is within the specified distance range D1, then delete the matching point pair.

[0047] The matching point filtering process involved in steps 5 and 6 is as follows: Figure 3 As shown, specifically, the first matching point selection in step 5 focuses on the selection of local feature points, while the second matching point selection in step 6 focuses on the global matching point selection.

[0048] Step 7: Compensate for matching points: Retrieve the regions in the original image corresponding to the segmented image pairs that have no matching points after the second matching point filtering. Then, through a compensation mechanism, add new matching points to the corresponding regions and update the matching point set. Specifically, for visible light images, find the original image regions corresponding to the image pairs that have no matching points after the second matching point filtering, and find the nearest matching point (x1, y1) to its center point (a1, b1). The calculation formula is:

[0049]

[0050] Where, x i ,y i These represent the x and y coordinates of the matching points retained after two rounds of filtering. For the infrared image, the matching point of the center point (a1, b1) on the infrared image is (a2, b2), and a2 = a1 + x2 - x1, b2 = b1 + y2 - y1; finally, (a1, b1) and (a2, b2) are added to the matching point set as compensation matching point pairs.

[0051] It should be noted that step 7 establishes an anchor point for the area without a matching point to avoid abnormal distortion of the infrared image when using a thin template strip in the subsequent step 8, thus making the registration error smaller.

[0052] Step 8: Perform registration transformation: Based on the final set of matching points, perform a thin spline transformation on the infrared image to obtain the registered infrared image. Specifically, the thin spline transformation uses the correspondence between matching points to establish a smoothing function to minimize deformation energy, thereby achieving accurate mapping from the source image to the target image. The thin spline transformation combines global smoothness with local flexibility, effectively handling nonlinear distortions, and is particularly suitable for solving complex deformation problems caused by modal differences between infrared and visible light images.

[0053] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned infrared and visible light image registration method for fusion depth estimation.

[0054] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned infrared and visible light image registration method for fusion depth estimation.

[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for registering infrared and visible light images by fusing depth estimation, characterized in that, Includes the following steps: Step 1: Preprocess the infrared and visible light images to be registered, and obtain the depth maps of the preprocessed infrared and visible light images; Step 2: Fuse the infrared image to be registered with its corresponding depth map to obtain an infrared image containing depth information; fuse the visible light image to be registered with its corresponding depth map to obtain a visible light image containing depth information. Step 3: Divide the infrared image and the visible light image containing depth information into rows and columns according to a preset number of rows and columns, so that the resulting infrared image blocks and visible light image blocks are the same size. Treat the infrared image blocks and visible light image blocks that are in the same position in their respective images as a pair of images. Step 4: Extract and match feature points for each pair of images to obtain an initial set of matching point pairs; Step 5, settings Value, based on The initial set of matching point pairs is first filtered, and matching point pairs that fail the filter are removed, resulting in the set of matching point pairs after the first filter; the specific process is as follows: set up The value is calculated using the following formula: ， Where w represents the width of the infrared or visible light image to be registered, and cols represents the number of columns into which the infrared or visible light image containing depth information is divided. Calculate the length of the matching line corresponding to each pair of matching points in the initial set of matching point pairs, sort all the matching line lengths in ascending order, and select the three pairs of matching points corresponding to the first three matching line lengths. For three pairs of matching points, considering the three matching points on the infrared image, calculate the distance between any two matching points. If there exists a distance less than... The distance will be less than The matching point with the longer matching line among the two matching points corresponding to the distance is deleted, and the matching point corresponding to the deleted matching point on the visible light image is also deleted; if it does not exist, it is not deleted. If there are two or more remaining matching point pairs, then for several matching points on the visible light image, calculate the distance between any two matching points. If there exists a distance less than 2 / 3... The distance will be less than The matching point with the longer matching line among the two matching points corresponding to the distance is deleted, and the matching point corresponding to the deleted matching point on the infrared image is also deleted; if it does not exist, it is not deleted; thus obtaining the set of matching point pairs after the first filtering. Step 6, Settings Value, based on The set of matching point pairs after the first filtering is then filtered a second time, and matching point pairs that fail the first filtering are deleted, resulting in the set of matching point pairs after the second filtering. The specific process is as follows: Each matching point pair in the first set of matching point pairs is mapped to the infrared and visible light images to be registered, generating a new set of matching point pairs. set up The value is calculated using the following formula: ， Where h represents the height of the infrared or visible light image to be registered, and rows represents the number of rows into which the infrared or visible light image containing depth information is divided. Calculate the absolute value of the slope of the matching line corresponding to each pair of matching points in the new set of matching point pairs. The formula is as follows: ， in, The ordinate of the matching point in the visible light image. Represents the ordinate of the matching point in the infrared image. Represents the x-coordinate of the matching point in the visible light image. Represents the x-coordinate of the matching point in the infrared image; Determine if there exists a matching line whose absolute slope is greater than a preset threshold (thres). If so, calculate the matching point pairs corresponding to those matching lines whose absolute slope is greater than thres. Other matching points shortest distance between The formula for calculating the shortest distance is: ， If the shortest distance is less than Then the matching point pair will be Delete the match points if necessary, otherwise do not delete them, and obtain the set of matching point pairs after the second filtering. Step 7: For image pairs that do not have matching point pairs after feature point extraction and matching in Step 4, and image pairs that do not have matching point pairs after the second screening in Step 6, a compensation mechanism is used to compensate for the matching points, so that each image pair has at least one pair of matching point pairs, and the final set of matching point pairs is obtained. Step 8: Based on the final set of matching point pairs, perform thin strip transformation on the infrared image to be registered to obtain the registered infrared image.

2. The infrared and visible light image registration method based on fusion depth estimation according to claim 1, characterized in that, The specific process of step 1 is as follows: The infrared and visible light images to be registered are preprocessed to ensure that the resolution of the preprocessed infrared and visible light images is the same. Monocular depth estimation is then performed on the preprocessed infrared and visible light images using the Depth-Anything-V2 model to obtain the corresponding depth maps for each of the preprocessed infrared and visible light images.

3. The infrared and visible light image registration method based on fusion depth estimation according to claim 1, characterized in that, The specific process of step 2 is as follows: When fusing the infrared image to be registered with its corresponding depth map, the contour of the depth map is first extracted to obtain the contour map; then the infrared image to be registered, the depth map and the contour map are fused in a weighted fusion manner to obtain an infrared image containing depth information. When fusing the visible light image to be registered with its corresponding depth map, the contour of the depth map is first extracted to obtain the contour map; then the visible light image to be registered, the depth map and the contour map are fused in a weighted fusion manner to obtain a visible light image containing depth information.

4. The infrared and visible light image registration method based on fusion depth estimation according to claim 1, characterized in that, In step 4, feature points are extracted and matched for each pair of images using the LoFTR deep learning network.

5. The infrared and visible light image registration method based on fusion depth estimation according to claim 1, characterized in that, The specific process of step 7 is as follows: For image pairs that do not have matching point pairs after feature point extraction and matching in step 4, and for image pairs that do not have matching point pairs after the second filtering in step 6; For the visible light image patch A, the center point of A is... As the matching point on A; find the center point of the visible light image patch A on the visible light image to be registered. The nearest matching point a And find the matching point a on the infrared image to be registered. The corresponding matching point b ; For infrared image block B, the center point The matching point on infrared image block B is ,and , ; Will and As a matching point pair between visible light image block A and infrared image block B, it is added to the matching point pair set after the second filtering to obtain the final matching point pair set.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the infrared and visible light image registration method for fusion depth estimation as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the infrared and visible light image registration method for fusion depth estimation as described in any one of claims 1 to 5.

Images