An edge extraction-based hetero-spectral image registration method

Abstract

The application discloses a multispectral image registration method based on edge extraction, and belongs to the field of image registration. The method comprises the following steps: performing edge extraction on source images of infrared images and source images of visible light images respectively to obtain two edge images, taking one of the two edge images as a reference image and the other as a to-be-registered image; dividing the reference image into a plurality of non-overlapping sub-image blocks, selecting, in each sub-image block, a region with the largest number of edge points from all regions with a specified size as a sub-edge template in the corresponding sub-image block, and combining the sub-edge templates according to positions to obtain a matching edge template; matching the to-be-registered image by using the matching edge template, calculating a transformation matrix of the to-be-registered image to the reference image according to a matching result; and performing position transformation on each pixel in a source image corresponding to the to-be-registered image according to the transformation matrix to complete image registration. The application can ensure registration accuracy and has good registration speed.

Description

Technical Field

[0001] This invention belongs to the field of image registration, and more specifically, relates to a heterospectral image registration method based on edge extraction. Background Technology

[0002] Among the various multispectral imaging methods, infrared and visible light images are the most widely used. Infrared images are formed through the thermal radiation of objects, thus they are less affected by environmental factors such as lighting and occlusion, and can effectively highlight bright targets in the image, but the image resolution is relatively low. Visible light images are formed through the light reflected from objects, containing rich details and textures, and have higher resolution, which is more in line with human visual perception, but are easily affected by environmental factors such as lighting. By registering and fusing infrared and visible light images, the advantages of both can be effectively combined, thus ensuring both rich details and textures in the image and highlighting bright targets.

[0003] In the actual imaging process of infrared and visible light images, the imaging spectrum and imaging angle of the two sensors often differ. To better combine and process the two types of images, it is often necessary to first register them, and then use the registered image for further image processing, such as image fusion. Currently, infrared and visible light image registration technology is widely used in many fields such as military detection, power line inspection, medical imaging analysis, geological monitoring, remote sensing imagery, and automatic control. In same-spectrum image registration, the grayscale information of the image or the corresponding feature information calculated from the grayscale is mostly utilized. However, infrared and visible light images have different imaging mechanisms and are heterospectral images, with very weak grayscale correlation between them. This grayscale-based image registration method cannot achieve ideal results.

[0004] Considering the strong correlation between edge information between infrared and visible light images, existing technologies have proposed methods for registering infrared and visible light images based on edge information. These registration methods avoid the problem of mismatch in grayscale, gradient and other information, and improve the registration accuracy of heterospectral images to a certain extent. However, in practical applications, the data structure of edge contour feature information is complex to construct, the matching process is time-consuming and the registration speed is slow. Summary of the Invention

[0005] To address the shortcomings and improvement needs of existing technologies, this invention provides a heterospectral image registration method based on edge extraction, aiming to achieve both high registration accuracy and fast registration speed.

[0006] To achieve the above objectives, according to one aspect of the present invention, a heterospectral image registration method based on edge extraction is provided, comprising the following steps:

[0007] (S1) Extract the edges of the two source images to be registered to obtain two edge images. Use one of them as the reference image and the other as the image to be registered. Of the two source images, one is an infrared image and the other is a visible light image.

[0008] (S2) Divide the reference image into multiple non-overlapping sub-image blocks. Within each sub-image block, starting from a size of... The region with the most edge points is selected from all regions as the sub-edge template in the corresponding sub-image block. The sub-edge templates are combined according to their positions to obtain the matching edge template. and These represent the width and height of the region, respectively. It does not exceed the width of the corresponding sub-image patch, and It does not exceed the height of the corresponding sub-image block;

[0009] (S3) Use the matching edge template to match the image to be registered, and calculate the transformation matrix from the image to be registered to the reference image based on the matching result;

[0010] (S4) Based on the transformation matrix, perform position transformation on each pixel in the source image corresponding to the image to be registered to complete image registration.

[0011] In some optional implementations, step (S3) involves matching the images to be registered using a matching edge template, including:

[0012] Using the image to be registered as the registration object and the matching edge template as the template image, fast edge registration is performed on the registration object and the template image to obtain the optimal registration position between the image to be registered and the matching edge template.

[0013] Fast edge registration includes:

[0014] (S31) Set a sliding box with the same size as the template image and align the sliding box with the upper left position of the registered image. Match the edge points contained in the sliding box with the edge points in the template image. Record the number of registered edge points as the number of optimal registration points NUM, and record the current position of the sliding box as the optimal edge registration position (x,y). NUM ;

[0015] (S32) If all positions have been slid, proceed to step (S35); otherwise, slide the slider to the next position according to the preset step distance, and then proceed to step (S33).

[0016] (S33) Record the number of edge points contained in the current sliding box. k If num kIf the result is greater than NUM, proceed to step (S34); otherwise, proceed to step (S32).

[0017] (S34) Match the edge points contained within the sliding box with the edge points in the template image to obtain the number of registered edge points NUM. c If NUM c If the value is greater than NUM, then NUM = NUM. c Update the optimal number of registration points NUM and set the optimal edge registration position (x,y). NUM Update the slider to its current position, then proceed to step (S32); otherwise, proceed to step (S32).

[0018] (S35) Output the current optimal edge registration position (x, y). NUM , complete the matching.

[0019] In some optional implementations, step (S3) involves matching the images to be registered using a matching edge template, including:

[0020] The image to be registered and the matching edge template are downsampled at the same downsampling rate to obtain image features and template features;

[0021] Using image features as the registration object and template features as the template image, fast edge registration is performed on the registration object and the template image to obtain the first optimal registration position between the image features and the template features.

[0022] Find the position corresponding to the first best registration position in the image to be registered and the matching edge template, and delineate the region containing this position in the image to be registered as the registration object. Delineate the region containing this position in the matching edge template as the template image. Then perform fast edge registration to obtain the second best registration position between the image to be registered and the matching edge template.

[0023] Fast edge registration includes:

[0024] (S31) Set a sliding box with the same size as the template image and align the sliding box with the upper left position of the registered image. Match the edge points contained in the sliding box with the edge points in the template image. Record the number of registered edge points as the number of optimal registration points NUM, and record the current position of the sliding box as the optimal edge registration position (x,y). NUM ;

[0025] (S32) If all positions have been slid, proceed to step (S35); otherwise, slide the slider to the next position according to the preset step distance, and then proceed to step (S33).

[0026] (S33) Record the number of edge points contained in the current sliding box. k If num k If the result is greater than NUM, proceed to step (S34); otherwise, proceed to step (S32).

[0027] (S34) Match the edge points contained within the sliding box with the edge points in the template image to obtain the number of registered edge points NUM. c If NUM c If the value is greater than NUM, then NUM = NUM. c Update the optimal number of registration points NUM and set the optimal edge registration position (x,y). NUM Update the slider to its current position, then proceed to step (S32); otherwise, proceed to step (S32).

[0028] (S35) Output the current optimal edge registration position (x, y). NUM , complete the matching.

[0029] Further, in step (S34), the edge points contained within the sliding box are matched with the edge points in the template image to obtain the number of registered edge points NUM. c ,include:

[0030] At the current position, fix the center of the sliding box and rotate the sliding box one revolution at preset angle intervals. At each angle, match the edge points contained in the sliding box with the edge points in the template image and record the number of registered edge points.

[0031] The maximum number of registered edge points at each angle is taken as the number of registered edge points NUM at the current position. c And record the corresponding rotation angle.

[0032] Furthermore, in step (S35), the optimal edge registration position (x, y) is output. NUM At the same time, it will output the corresponding rotation angle.

[0033] Furthermore, downsampling is performed as maximum value downsampling.

[0034] Furthermore, in step (S1), before selecting the reference image and the image to be registered, the following steps are performed for edge cleaning for each edge image:

[0035] The edge points contained in the edge image are clustered to obtain multiple edge categories; edge categories containing fewer than a preset first threshold are removed.

[0036] Furthermore, in step (S2), before combining each sub-edge template according to its position to obtain a matching edge template, the method further includes: removing sub-edge templates that contain fewer than a preset second threshold number of edge points.

[0037] Furthermore, in step (S2), the width and height of the sub-edge template constructed within each sub-image block are as follows:

[0038]

[0039]

[0040] in, and These represent the width and height of the sub-image patch, respectively; ratio represents the scaling factor, and ratio = 0.6.

[0041] Furthermore, step (S4) also includes: after transforming the position of each pixel in the source image corresponding to the image to be registered, weighted fusion of the transformed image with another source image.

[0042] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising: a stored computer program; when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to execute the heterospectral image registration method based on edge extraction provided by the present invention.

[0043] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0044] (1) This invention utilizes edge information with stronger correlation between infrared and visible light images for registration calculation, avoiding the problem of mismatch in grayscale, gradient and other information, and improving the registration accuracy of heterospectral images. During registration, it does not register the complete edge image, but divides the selected reference image into sub-image blocks, and further finds the region of specified size containing the most edge points in each sub-image block to construct sub-edge templates. Finally, these sub-edge templates constitute the matching edge template. Compared with the complete edge image, the matching edge template greatly reduces the number of edge points while fully preserving the edge features. Image registration based on the matching edge template can effectively improve the registration speed while ensuring registration accuracy.

[0045] (2) In a preferred embodiment of the present invention, after determining the matching edge template, a partial calculation method is adopted when performing image registration. Specifically, a sliding box of the same size as the matching edge template is constructed. The optimal registration position is determined by moving the sliding box and calculating the degree of edge registration between the sliding box and the matching edge template at different positions. During the movement of the sliding box, the number of optimal registration points is recorded in real time. Each time the sliding box moves to a position, the number of edge points contained in the sliding box is compared with the current number of optimal registration points. When the number of edge points contained in the sliding box is small, no registration calculation is performed, thereby avoiding invalid calculations and further improving the registration speed.

[0046] (3) In a preferred embodiment of the present invention, after determining the matching edge template, image registration is performed using a combination of pyramid algorithm and partial calculation. Specifically, the image to be registered and the matching edge template are first downsampled, and the downsampled result is registered using partial calculation. After determining the optimal registration position, it is mapped to the image to be registered and the matching edge template. A region containing the optimal registration position is then delineated in the image to be registered and the matching edge template, and registration is performed again using partial calculation to determine the final optimal registration position. By registering the downsampled result, the optimal registration position can be quickly and roughly determined with less computation. Based on this, a region is delineated in the image to be registered and the matching edge template for secondary registration, which can accurately determine the optimal registration position. Therefore, by combining pyramid algorithm and partial calculation, the present invention can further improve the registration speed. More preferably, maximum value downsampling is used, which ensures that the image retains edge information well after downsampling.

[0047] (4) When performing fast edge registration by partial calculation, the present invention calculates the registration results at different angles at each position to which the sliding box slides, thereby assisting in determining the optimal registration position. This fully considers the relative rotation relationship between the source images and ensures the registration accuracy.

[0048] (5) After edge extraction, before selecting the reference image and the image to be registered, the present invention will first cluster the edge points and remove the edge categories with fewer edge points. The edge categories with fewer edge points are often noise. By removing such edge categories, the present invention can effectively avoid the influence of noise on the registration accuracy and further reduce the amount of registration calculation.

[0049] (6) After constructing the sub-edge template, the present invention will remove the sub-edge template with fewer edge points before constructing the matching edge template. This can avoid the influence of noise on the registration accuracy and further reduce the amount of registration calculation.

[0050] (7) In constructing the sub-edge template, the present invention sets the ratio of the width and height of the sub-edge template to the width and height of the corresponding sub-image block to 0.6, which can ensure registration accuracy while also having a faster registration speed. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of a heterospectral image registration method based on edge extraction provided in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the source image to be registered and the fused image provided in an embodiment of the present invention; wherein, (a) is an infrared image, (b) is a visible light image, and (c) is the fused image;

[0053] Figure 3 This is a schematic diagram of a heterospectral image registration method based on edge extraction, provided in another embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0055] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0056] To address the technical issues that existing grayscale-based registration methods are not suitable for registering infrared and visible light images, while edge-based registration methods, while improving registration accuracy, are time-consuming, this invention provides a heterospectral image registration method based on edge extraction. The overall idea is to process the image edges before registration based on the image edges, minimizing the computational load while preserving edge features, thereby ensuring both registration accuracy and good registration speed.

[0057] The following is an example.

[0058] Example 1:

[0059] A metaspectral image registration method based on edge extraction, such as Figure 1 As shown, it includes the following steps:

[0060] (S1) Extract the edges of the two source images to be registered to obtain two edge images. Use one of them as a reference image and the other as the image to be registered. Of the two source images, one is an infrared image and the other is a visible light image.

[0061] Optionally, in step (S1) of this embodiment, the Canny operator is used to extract edges from the two source images; it is easy to understand that in other embodiments of the present invention, other methods such as the Sobel operator and the Prewitt operator can also be used to extract image edges.

[0062] Considering that images are susceptible to noise interference, in order to avoid the impact of image noise on registration accuracy, step (S1) of this embodiment further cleans the image edges after extracting them to remove noise. Specifically, for each edge image, the edge points contained therein are clustered using the DBSCAN clustering idea to obtain multiple edge categories; the edge categories containing too few edge points (less than a preset first threshold) are caused by noise, and therefore, they are removed; the cleaned image edges are clearer and more complete, and are more conducive to accurate registration.

[0063] Optionally, in this embodiment, the edge image of the visible light image is selected as the reference image, and the edge image of the infrared image is selected as the image to be registered.

[0064] This embodiment also includes: (S2) dividing the reference image into multiple non-overlapping sub-image blocks, and within each sub-image block, starting from a size of... The region with the most edge points is selected from all regions as the sub-edge template in the corresponding sub-image block. The sub-edge templates are combined according to their positions to obtain the matching edge template. and These represent the width and height of the region, respectively. It does not exceed the width of the corresponding sub-image patch, and It does not exceed the height of the corresponding sub-image block.

[0065] This embodiment divides the selected reference image into sub-image blocks, and further identifies the region of a specified size containing the most edge points within each sub-image block to construct a sub-edge template. These sub-edge templates are then used to form a matching edge template for subsequent image registration. This matching edge template significantly reduces the number of edge points while fully preserving edge features, effectively improving registration speed while maintaining registration accuracy in the subsequent registration process.

[0066] Optionally, in this embodiment, when constructing the matching edge template, the reference image is specifically divided into four sub-image blocks of 2*2. The size of the sub-edge template is set to a certain proportion of the size of the sub-image block, and the calculation formula is as follows:

[0067]

[0068]

[0069] in, and These represent the width and height of the sub-edge template, respectively. and The width and height of the sub-image patch are represented by , and 'ratio' is a scaling factor. A larger 'ratio' results in a larger sub-edge template and higher registration accuracy, but also a slower calculation speed. As a preferred implementation, this embodiment sets the 'ratio' value to 0.6, which can ensure registration accuracy while maintaining a relatively fast registration speed. It should be noted that the sub-image patch division scheme and the sub-edge template setting method can be flexibly set according to actual applications. This is only a preferred implementation and should not be construed as the sole limitation of the invention.

[0070] Considering the influence of noise, the sub-edge templates constructed in some sub-image blocks may be invalid during image registration. Therefore, in order to further reduce the amount of computation and improve the registration speed, in step (S2) of this embodiment, after constructing the sub-edge templates of each sub-image block, if the number of edge points contained in the sub-edge template is greater than a certain proportion of the number of pixels in the sub-image block, then the sub-edge template is considered to be valid; otherwise, it is considered invalid, and invalid sub-edge templates will be discarded.

[0071] Specifically, a flag can be set for each sub-edge template. Set each flag as follows:

[0072]

[0073] Where, sum(template) i ) represents the number of edge points contained in the sub-edge template, sum(I i The value ) represents the number of pixels contained in the sub-image block, and ratio′ is a scaling factor, which is set to 0.01 in this embodiment. Using the above calculation method, the flag bit of the sub-edge template is used when the number of edge points is less than 0.01 of the total number of pixels in the entire sub-image block. This will be set to 0, indicating that the sub-edge template is invalid; otherwise, the sub-edge template flag will be set to 0. This will be set to 1, indicating that the sub-edge template is a valid sub-edge template. Finally, sub-edge templates with a flag of 0 will be discarded.

[0074] This embodiment avoids the impact of noise on registration accuracy by eliminating invalid sub-edge templates, and at the same time, further reduces the amount of computation required for registration.

[0075] This embodiment also includes: (S3) matching the image to be registered using the matching edge template, and calculating the transformation matrix from the image to be registered to the reference image based on the matching result.

[0076] For matching edge templates, save the position coordinates of the edge points to an array Inf = {(x i ,y i In the formula |i=1,2,3…}, for the corresponding region in the image to be registered, it is only necessary to calculate the coordinate point position (x) in Inf. i ,y i If a point is an edge point, it is considered that the edges of the two at that location are registered. By counting the number of registered edge points at all locations in Inf, the degree of edge registration between the edge template and the corresponding region of the image to be registered can be obtained. The region with the highest degree of edge registration in the image to be registered is the optimal registration position.

[0077] To further improve the registration speed, this embodiment proposes a partial calculation method. During the registration process, a sliding box of the same size as the matching edge template is constructed. By moving the sliding box and calculating the edge registration degree between the sliding box and the matching edge template at different positions, the optimal registration position is determined. During the movement of the sliding box, the number of optimal registration points is recorded in real time. Each time the sliding box moves to a position, the number of edge points contained within the sliding box is compared with the current number of optimal registration points. When the number of edge points contained within the sliding box is small, registration calculation is not performed, thereby avoiding unnecessary computation and further improving the registration speed. Specifically, in step (S3) of this embodiment, matching the image to be registered is performed using the matching edge template, including:

[0078] Using the image to be registered as the registration object and the matching edge template as the template image, fast edge registration is performed on the registration object and the template image to obtain the optimal registration position between the image to be registered and the matching edge template.

[0079] Fast edge registration includes:

[0080] (S31) Set a sliding box with the same size as the template image and align the sliding box with the upper left position of the registered image. Match the edge points contained in the sliding box with the edge points in the template image. Record the number of registered edge points as the number of optimal registration points NUM, and record the current position of the sliding box as the optimal edge registration position (x, y). NUM ;

[0081] (S32) If all positions have been slid, proceed to step (S35); otherwise, slide the slider to the next position according to the preset step distance, and then proceed to step (S33).

[0082] (S33) Record the number of edge points contained in the current sliding box. k If num k If the result is greater than NUM, proceed to step (S34); otherwise, proceed to step (S32).

[0083] To quickly calculate the number of edge points (num) contained within the current slider. k The specific method used in this embodiment is as follows:

[0084] Let num be the number of edge points contained within the previous position's sliding box. k-1 ; Calculate the number of edge points (num) contained in the newly added pixels of the current slider relative to the previous slider position. a And the number of edge points contained in the reduced pixels (num) p The difference num d =num q -num p Update the number of edge points contained in the slider to num. k =num k-1 +num d ;

[0085] The number of edge points contained within the current slider (num) k Not satisfied with num k When the value is greater than NUM, the edge registration degree at that position cannot be the highest, and that position cannot be the optimal registration position. At this time, registration calculation is not performed on the current sliding box, but the process is directly transferred to step (S32) to slide the sliding box to the next position. This reduces invalid calculations and further improves registration accuracy without affecting the registration accuracy.

[0086] (S34) Match the edge points contained within the sliding box with the edge points in the template image to obtain the number of registered edge points NUM. c If NUM c If the value is greater than NUM, then NUM = NUM. cUpdate the optimal number of registration points NUM and set the optimal edge registration position (x,y). NUM Update the slider to its current position, then proceed to step (S32); otherwise, proceed to step (S32).

[0087] (S35) Output the current optimal edge registration position (x, y). NUM , complete the matching.

[0088] Considering the potential relative rotation between source images, to ensure registration accuracy, in step (S34) of this embodiment, the edge points contained within the sliding box are matched with the edge points in the template image to obtain the number of registered edge points NUM. c ,include:

[0089] At the current position, fix the center of the sliding box and rotate the sliding box one revolution at preset angle intervals. At each angle, match the edge points contained in the sliding box with the edge points in the template image and record the number of registered edge points.

[0090] The maximum number of registered edge points at each angle is taken as the number of registered edge points NUM at the current position. c And record the corresponding rotation angle.

[0091] Furthermore, in step (S35), the optimal edge registration position (x, y) is output. NUM At the same time, it will output the corresponding rotation angle.

[0092] After determining the optimal edge registration position and the corresponding rotation angle, the rigid body model transformation formula is applied.

[0093]

[0094] The final transformation matrix parameters (x) of the image can be obtained. d ,y d ,θ),(x d ,y d ) represents the translation of the image to be registered relative to the reference image, θ represents the rotation angle of the image to be registered relative to the reference image, and (x,y) and (x',y') represent the pixel coordinates in the image to be registered and the reference image, respectively.

[0095] This embodiment also includes: (S4) transforming the position of each pixel in the source image corresponding to the image to be registered according to the transformation matrix to complete image registration.

[0096] To facilitate subsequent image processing, step (S4) of this embodiment further includes: after transforming the position of each pixel in the source image corresponding to the image to be registered, weighted fusion of the transformed image with another source image.

[0097] Figure 2 As shown, the registration result is based on the heterospectral image registration method provided in this embodiment, where (a) is the infrared image to be registered, (b) is the visible light image to be registered, and (c) is the fused image. Figure 2 As shown in (c), the registered image details are clearer, which demonstrates that the registration method provided in this embodiment can accurately complete the registration. Based on the registration result, the fused image obtained in this embodiment not only ensures the rich detail and texture information in the image, but also highlights the bright targets in the image.

[0098] To further verify the beneficial effects of the heterospectral image registration method provided in this embodiment, this embodiment calculates evaluation metrics for the fused image. These metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Mean Absolute Error (MAE). Higher PSNR and SSIM indicate better fusion performance and higher image registration accuracy, while lower MAE also indicates better fusion performance and higher image registration accuracy. Since many other methods cannot calculate the image transformation model in certain scenarios and therefore cannot achieve registration, the mismatch rate is also used as an evaluation metric for image registration. The formula for calculating the mismatch rate is as follows:

[0099]

[0100] The evaluation metrics for the fused images are shown in Table 1.

[0101] Table 1

[0102] index PSNR SSIM MAE Mismatch rate Runtime value 15.4926 0.6629 24.9165 0 93.59

[0103] As shown in Table 1, the fused image obtained by the heterospectral image registration method based on edge extraction provided in this embodiment has a good fusion effect, and the registration process has a short running time.

[0104] To verify the effectiveness and superiority of the proposed edge-extraction-based multi-sub-template matching algorithm, the registration method provided in this invention was compared and analyzed with three existing template-based matching methods. The basic processes of the three methods are the same, the difference lies in the similarity calculation methods. These three similarity calculation methods are explained below.

[0105] (1) Mutual Information MI

[0106] The mutual information (MI) of two images is equal to the sum of their entropies minus their joint entropy. For two images I and J, the formula for calculating their mutual information is as follows:

[0107] MI(I,J)=H(I)+H(J)-H(I,J)

[0108] In the above formula, H(I) and H(J) represent the entropy of image I and image J, respectively, and represent the joint entropy of image I and image J. They can be solved by the following formula.

[0109] H(I)=-∑ i P I (i)logP I (i)

[0110] H(J)=-∑ i P J (j)logP J (j)

[0111] H(I,J)=-∑ i，j PI J (i, j)logP IJ (i, j)

[0112]

[0113]

[0114]

[0115] In the above formula, P I (i) and P J (j) represents the probability distribution function of the image, i.e., the probability of each pixel's gray level appearing in the image, and N represents the number of gray levels in the image. The joint probability distribution function of two images is represented by P. IJ The values ​​(i, j) are used to represent the two images. The larger the mutual information (MI) between the two images, the greater the correlation between them. Specifically, in the template matching algorithm, by statistically analyzing the mutual information between the template image and the floating image, the position corresponding to the maximum mutual information is selected, which can be considered as the position with the greatest correlation between the template image and the floating image. Therefore, this position is used as the final matching position.

[0116] (2) Normalized Mutual Information (NMI)

[0117] In some cases, an image may contain multiple regions with similar grayscale distributions. In such situations, normalized mutual information can achieve better registration results compared to mutual information. The calculation of image entropy and joint entropy in normalized mutual information is the same as in mutual information; the only difference lies in the calculation of the final normalized mutual information index. The calculation formula is shown below.

[0118]

[0119] Similar to mutual information, the greater the normalized mutual information, the greater the correlation between the two images. Therefore, by finding the matching position corresponding to the maximum normalized mutual information during the template matching process, the registration of the two images can be achieved.

[0120] (3) Normalized cross-correlation coefficient NCC

[0121] The normalized cross-correlation coefficient can also characterize the degree of correlation between two images. The formula for calculating the normalized cross-correlation coefficient between images I and J is shown below.

[0122]

[0123] In the above formula, I(i,j) and J(i,j) represent the pixel values ​​at corresponding positions in image I and image J, respectively. and The average pixel value within the template region. The greater the correlation between two images, the higher their calculated NCC values. Therefore, by using this index to find the matching position in the floating image where the correlation with the template image is greatest, the registration of the two images can be achieved.

[0124] Three existing feature-based heterospectral image registration algorithms, namely SURF-PIIFD-RPM, PSO-SIFT and CAO-C2F, were selected for comparison. Together with template matching methods based on MI, NMI and NCC, a total of 6 methods were selected. The evaluation metrics of these 6 methods are shown in Table 2.

[0125] Table 2 Evaluation Indicators for Different Registration Methods

[0126] Registration method PSNR SSIM MAE Mismatch rate MI-based template matching 14.1630 0.5851 34.3952 0 NMI-based template matching 14.2951 0.6040 33.2263 0 NCC-based template matching 13.8778 0.5785 35.4041 0 SURF-PIIFD-PRM 14.2556 0.5951 33.6998 51.43％ PSO-SIFT 14.2469 0.5896 33.9196 80％ CAO-CF 14.4738 0.6242 30.6524 28％

[0127] Comparing the results in Tables 1 and 2, it is evident from the various indicators that the image fusion effect after registration processing using the method provided in this invention is the best, with no mismatch occurring. This demonstrates that this invention has superior registration performance. When registering heterospectral images such as infrared and visible light, which have large spectral distances, accurate registration is difficult to achieve using grayscale information due to significant differences in grayscale between the images. Therefore, this invention starts from the edge information, which has stronger correlation between such heterospectral images, and employs a template matching method based on edge extraction, achieving better registration results. Experimental results prove that the multi-sub-template matching algorithm based on edge extraction proposed in this paper is more suitable for registering infrared and visible light images.

[0128] Example 2:

[0129] A metaspectral image registration method based on edge extraction, such as Figure 3As shown, this embodiment is similar to Embodiment 1 above. The difference is that, in order to further improve the registration speed, step (S3) of this embodiment uses a matching edge template to match the image to be registered, and adopts a pyramid algorithm combined with partial calculation. Specifically, step (S3) uses the matching edge template to match the image to be registered, including:

[0130] The image to be registered and the matching edge template are downsampled at the same downsampling rate to obtain image features and template features. Considering that the edge image is a binary image with only two values, 0 and 1, in order to ensure that the image can still retain edge information well after downsampling, preferably, maximum value downsampling is used in this embodiment; optionally, in this embodiment, downsampling is specifically 16 times maximum value downsampling; the image to be registered and the matching edge template can be regarded as the bottom layer of the pyramid, and the image features and template features obtained by downsampling can be regarded as the second layer of the pyramid.

[0131] Using image features as the registration object and template features as the template image, fast edge registration is performed on the registration object and the template image to obtain the first optimal registration position between the image features and the template features.

[0132] Find the position corresponding to the first optimal registration position in the image to be registered and the matching edge template, and delineate a region containing this position in the image to be registered as the registration object. Delineate a region containing this position in the matching edge template as the template image. Then perform fast edge registration to obtain the second optimal registration position between the image to be registered and the matching edge template. This second optimal registration position is the final optimal registration position. Optionally, in this embodiment, when performing secondary matching, the size of the delineated region is 60*60 pixels.

[0133] In this embodiment, the fast edge registration method is the same as in Embodiment 1 above. For details, please refer to the description in steps (S31) to (S35) above, which will not be repeated here.

[0134] This embodiment can quickly and roughly determine the optimal registration position with less computation by registering the downsampling results. Based on this, a region is delineated in the image to be registered and the matching edge template for secondary registration, which can accurately determine the optimal registration position. Therefore, by combining the pyramid algorithm with partial computation, this embodiment can further improve the registration speed.

[0135] Example 3:

[0136] A computer-readable storage medium includes: a stored computer program; when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to execute the heterospectral image registration method based on edge extraction provided in Embodiment 1 or 2 above.

[0137] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of edge extraction based hetero-spectral image registration, characterized in that, The method comprises the following steps: (S1) performing edge extraction on two source images to be registered respectively to obtain two edge images, taking one of the two edge images as a reference image and the other as an image to be registered; one of the two source images is an infrared image and the other is a visible light image; (S2) dividing the reference image into a plurality of mutually non-overlapping sub-image blocks, and within each sub-image block, selecting a region containing the largest number of edge points from all regions of size as a sub-edge template within the corresponding sub-image block, and combining each sub-edge template according to position to obtain a matching edge template; and represent the width and height of the region, respectively, does not exceed the width of the corresponding sub-image block, and does not exceed the height of the corresponding sub-image block. (S3) performing matching on the image to be registered by using the matching edge template, and calculating a transformation matrix of the image to be registered to the reference image according to a matching result; (S4) performing position transformation on each pixel in the source image corresponding to the image to be registered according to the transformation matrix to complete image registration; In the step (S3), the matching edge template is used to match the image to be registered, which comprises: taking the image to be registered as a registration object and the matching edge template as a template image, performing fast edge registration on the registration object and the template image to obtain a best registration position between the image to be registered and the matching edge template; The fast edge registration comprises: (S31) setting a sliding frame equal in size to the template image, and aligning the sliding frame to the upper left position of the registration image, matching the edge points contained in the sliding frame with the edge points in the template image, recording the number of registration edge points as the optimal registration point number NUM, and recording the current position of the sliding frame as the optimal edge registration position x , y ) NUM ; (S32) if the sliding of all positions is completed, the step (S35) is entered; otherwise, the sliding frame is slid to the next position according to a preset step distance, and the step (S33) is entered; (S33) Record the number of edge points currently contained in the sliding frame If go to step (S34); otherwise, go to step (S32); (S34) Match the edge points contained within the sliding box with the edge points in the template image to obtain the number of registered edge points. NUM c ,like Then according to For the optimal number of registration points NUM Update and set the optimal edge registration position ( x , y ) NUM Update the slider to its current position, then proceed to step (S32); otherwise, proceed to step (S32). (S35) output the current best edge registration position x , y ) NUM , complete matching; Alternatively, in the step (S3), the matching edge template is used to match the image to be registered, which comprises: performing down-sampling on the image to be registered and the matching edge template respectively according to the same down-sampling rate to obtain image features and template features; taking the image features as a registration object and the template features as a template image, performing fast edge registration on the registration object and the template image to obtain a first best registration position between the image features and the template features; finding a position corresponding to the first best registration position in the image to be registered and the matching edge template, and delimiting a region containing the position in the image to be registered as a registration object and delimiting a region containing the position in the matching edge template as a template image, and then performing fast edge registration to obtain a second best registration position between the image to be registered and the matching edge template; The fast edge registration comprises: (S31) setting a sliding frame equal in size to the template image, and aligning the sliding frame to the upper left position of the registration image, matching the edge points contained in the sliding frame with the edge points in the template image, recording the number of registration edge points as the optimal registration point number NUM, and recording the current position of the sliding frame as the optimal edge registration position x , y ) NUM ; (S32) if the sliding of all positions is completed, the step (S35) is entered; otherwise, the sliding frame is slid to the next position according to a preset step distance, and the step (S33) is entered; (S33) Record the number of edge points currently contained in the sliding frame If , go to step (S34); otherwise, go to step (S32); (S34) matching the edge points contained in the sliding frame with the edge points in the template image to obtain the number of registered edge points NUM c , if , then updating the number of best registration points , and updating the best edge registration position NUM ( x , y ) NUM to the current position of the sliding frame, and then turning to step (S32); otherwise, turning to step (S32);​ (S35) output the current best edge registration position x , y ) NUM , complete matching.

2. The edge extraction based hetero-spectral image registration method of claim 1, wherein, In the step (S34), the edge points contained in the sliding frame are matched with the edge points in the template image, to obtain the number NUM of matched edge points c comprising: In the current position, the center of the sliding frame is fixed, and the sliding frame is rotated for one revolution according to a preset angle interval, in each angle, the edge points contained in the sliding frame are matched with the edge points in the template image, and the number of registered edge points is recorded; The maximum number of edge points registered under each angle is taken as the number of edge points registered under the current position NUM c and the corresponding rotation angle is recorded And, the step (S35) in, in output best edge registration position x , y ) NUM at the same time, will output the corresponding rotation angle.

3. The edge extraction based hetero-spectral image registration method of claim 1, wherein, The down-sampling is maximum value down-sampling.

4. The edge extraction based hetero-spectral image registration method of claim 1, wherein, In the step (S1), before the reference image and the image to be registered are selected, the following steps are further performed for each edge image to perform edge cleaning: clustering the edge points contained in the edge image to obtain a plurality of edge categories; and eliminating the edge categories whose number of edge points is less than a preset first threshold.

5. The edge extraction based hetero-spectral image registration method of claim 1, wherein, The step (S2) further comprises: eliminating the sub-edge template whose contained edge points are less than a preset second threshold before combining the sub-edge templates to obtain the matching edge template.

6. The edge extraction based hetero-spectral image registration method of claim 5, wherein, In the step (S2), the width and height of the sub-edge template constructed in each sub-image block are as follows: wherein, and denote the width and height of the sub-picture tile, respectively; ratio denotes a scale factor, and ratio = 0.

6.

7. The edge extraction based hetero-spectral image registration method of claim 1, wherein, The step (S4) further comprises: performing position transformation on each pixel in the source image corresponding to the image to be registered, and then performing weighted fusion of the transformed image and another source image.

8. A computer-readable storage medium, characterized in that, The method comprises: A computer program stored in the computer readable storage medium, wherein the computer program is executed by a processor to control the device where the computer readable storage medium is located to perform the edge extraction based multispectral image registration method according to any one of claims 1-7.

Images