An infrared and visible light image registration method

Abstract

The present application relates to a kind of image registration methods, specifically to a kind of infrared and visible light image registration method.The present application solves the technical problem that the existing infrared and visible light image registration method has lower contrast of infrared image, makes feature point extraction difficult, and leads to poor matching accuracy.The present application includes the following steps:1) respectively collect infrared image and visible light image of the same scene;2) by image enhancement algorithm, pixel enhancement processing is carried out to infrared image to obtain enhanced image;3) respectively to visible light image and enhanced image are carried out contour extraction, obtain infrared contour image and visible light contour image;4) using feature extraction algorithm, all feature points of enhanced image and visible light image are extracted, and finally feature points of enhanced image and visible light image are obtained from all feature points;5) using dynamic point matching algorithm, the final feature points of enhanced image and visible light image are matched, and image registration is completed.

Description

TECHNICAL FIELD

[0001] The present application relates to an image registration method, in particular to an infrared and visible light image registration method. BACKGROUND

[0002] Image registration is an important research direction in the field of computer vision, pattern recognition and medical image analysis, including single-mode image and multi-mode image registration, wherein multi-mode image registration refers to registration of heterogeneous images collected by different image sensors, so as to satisfy the alignment of images under the coordinate axis. The purpose of image registration is to obtain the spatial mapping relationship of different images, and align the spatial positions of the same target in different images. Image registration is an indispensable preprocessing step for image fusion and three-dimensional reconstruction technology.

[0003] The existing infrared and visible light image registration methods can be divided into three categories: calibration parameter-based, region-based and feature-based. The calibration parameter-based method is a non-automatic registration method, which can only register a group of images taken at the same time, and its registration accuracy depends on the calibration parameter accuracy. The region-based registration method depends on the linear correlation degree of image gray scale and the field of view overlap degree, and has poor adaptability to complex scenes with perspective difference, spectral difference and distortion, and high computational complexity, such as the registration method based on gradient mutual information. Compared with the former two registration methods, the feature-based registration method has strong stability and better registration effect for images of various complex scenes, among which the point feature-based registration method is widely applied and researched. The existing point feature extraction methods include SIFT, Harris, SURF and PIIFD, etc. However, due to the low contrast of infrared images, it is difficult to extract image feature points, resulting in poor matching accuracy. SUMMARY

[0004] The purpose of the present application is to solve the technical problem of the existing infrared and visible light image registration method, which has low contrast of collected infrared images, making it difficult to extract image feature points and resulting in poor matching accuracy, and to provide an infrared and visible light image registration method. The present application improves the contrast of infrared images through an image enhancement algorithm, extracts the contour maps of infrared images and visible light images using a contour extraction method, extracts feature points using a SURF feature extraction algorithm, and finally completes the matching of feature points using a dynamic point matching algorithm.

[0005] The technical solution of the present application is:

[0006] An infrared and visible light image registration method, characterized in that it comprises the following steps:

[0007] 1) Collect infrared images and visible light images of the same scene respectively;

[0008] 2) obtaining an enhanced image by performing pixel enhancement processing on the infrared image through an image enhancement algorithm;

[0009] 3) performing contour extraction on the visible light image and the enhanced image respectively by using a multi-directional gradient mean method to obtain an infrared contour image and a visible light contour image;

[0010] 4) extracting all feature points of the enhanced image and the visible light image respectively through the infrared contour image and the visible light contour image by using a feature extraction algorithm, and then obtaining the final feature points of the enhanced image and the visible light image from all the feature points;

[0011] 5) matching the final feature points of the enhanced image and the visible light image by using a dynamic point matching algorithm to complete image registration.

[0012] Further, in step 2), the image enhancement algorithm comprises the following steps:

[0013] A) performing filter processing on the pixel value p r (i,j) of each pixel point in the infrared image through a guided filter formula to obtain the base layer image pixel value

[0014]

[0015] The GF(·) function is a guided filter;

[0016] (i,j) is the coordinate of the pixel point;

[0017] s is the filter radius of the filter;

[0018] ε is the filter coefficient;

[0019] B) obtaining the detail layer image pixel value d (i,j) of each pixel point in the infrared image through the following formula:

[0020]

[0021] C) performing pixel enhancement on the base layer image pixel value b (i,j) and the detail layer image pixel value d of each pixel point respectively through the following formula to obtain the enhanced base layer image pixel value b and the enhanced detail layer image pixel value d

[0022]

[0023] α is the enhancement coefficient;

[0024] μ is the average value of the gray values of all pixel points in the image;

[0025] β is a detail enhancement coefficient, and is 2;

[0026] abs(·) represents an absolute value;

[0027] D) the final image pixel value P of each pixel point after enhancement is obtained by the following formula r E (i,j):

[0028]

[0029] E) the final image pixel value P of all pixel points in the infrared image after enhancement is obtained r E (i,j), to obtain an enhanced image.

[0030] Further, in step 3, the multi-direction gradient mean method adopts seven directions, and the corresponding angles are 0°, 30°, 60°, 90°, 120°, 150° and 180°, respectively.

[0031] The contour extraction of the image by the multi-direction gradient mean method is specifically as follows: first, the gradient values of the image in 0°, 30°, 60°, 90°, 120°, 150° and 180° directions are calculated respectively, then the mean values of the gradient values of the image in the seven directions are calculated by the mean() function, and the contour image GM is obtained by the following formula:

[0032]

[0033] θ: gradient direction;

[0034] g θ : gradient value corresponding to the gradient direction;

[0035] mean() function: mean value of multiple direction gradient values.

[0036] Further, in step 4), the feature extraction algorithm adopts a SURF feature point extraction algorithm.

[0037] Further, in step 5), the dynamic point matching algorithm specifically includes the following steps:

[0038] S1) the feature point outliers of the enhanced image and the visible light image are eliminated by a single Gaussian model;

[0039] The Gaussian model is as follows:

[0040]

[0041] E(f,σ 2 ) is an energy function about affine transformation;

[0042] f is a radio transform function;

[0043] σ is a filter scale;

[0044] M and N respectively represent the final number of feature points of the visible light image and the enhanced image;

[0045] x j is a feature point of the enhanced image, and j is the corresponding feature point serial number;

[0046] y i is a feature point of the visible light image, and i is the corresponding feature point serial number;

[0047] η is the corresponding relationship between the feature matching points, and its value is 1 when the two feature points are in a matching relationship, and 0 otherwise;

[0048] φ is a Gaussian function;

[0049] S2) using weighted regularization least squares to iteratively calculate the energy function of the Gaussian model, and obtaining the projection transformation matrix when the energy function is minimum;

[0050] S3) obtaining all matching feature points of the enhanced image and the visible light image through the projection transformation matrix, and then projecting the enhanced image to the visible light image through the projection transformation matrix to obtain an infrared rotation image, and completing image registration.

[0051] Further, in step 4), a 3*3*3 neighborhood non-maximum suppression method is used to obtain the final feature points.

[0052] Advantages of the present application:

[0053] 1. The infrared and visible light image registration method of the present application enhances the pixels of the infrared image through an image enhancement algorithm, improves the contrast of the infrared image, makes the image feature point extraction accurate and efficient, and improves the matching accuracy of the infrared and visible light image registration.

[0054] 2. The infrared and visible light image registration method of the present application provides an image enhancement algorithm, which first uses a guided filter to decompose the image, enhances the decomposed base layer image, enhances the detail layer image, and then adds the processed base layer image and the detail layer image to obtain the final enhanced image, and the enhancement effect is remarkable.

[0055] 3、The profile extraction of the infrared and visible light image registration method of the application adopts a seven-direction gradient mean method to respectively acquire infrared and visible light profile images, with 30° as an interval, and seven directions corresponding angles being 0°, 30°, 60°, 90°, 120°, 150° and 180°. If the interval is large, the image information cannot be accurately expressed, and if the interval is small, the calculation amount is large. On the basis of reducing the calculation amount, the seven-direction gradient mean method can completely retain the profile feature information of the infrared image and the visible light image, and the feature points are conveniently and quickly extracted. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the infrared and visible light image registration flowchart of the application;

[0057] Figure 2 is the infrared original image and the enhanced image contrast diagram in the embodiment of the application (a is the infrared image, and b is the enhanced image);

[0058] Figure 3 is the enhanced image, the visible light original image, the infrared profile image and the visible light profile image in the embodiment of the application (b is the enhanced image, c is the visible light image, d is the infrared profile image, and e is the visible light profile image);

[0059] Figure 4 is the infrared image feature point and the visible light image feature point extracted by using the SURF feature extraction algorithm in the embodiment of the application (f is the enhanced image feature point, and g is the visible light image feature point);

[0060] Figure 5 is the feature point matching diagram of the infrared image and the visible light image obtained after iterative calculation in the embodiment of the application (b is the enhanced image, and c is the visible light image);

[0061] Figure 6 is the infrared rotation image obtained by projection transformation in the embodiment of the application;

[0062] Figure 7 is the infrared and visible light fusion image after registration in the embodiment of the application. DETAILED DESCRIPTION

[0063] The application will be described in detail below with reference to the embodiments and the drawings. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments of the application, other embodiments obtained by those skilled in the art without creative work are within the protection scope of the application.

[0064] The infrared and visible light image registration method of the application, the registration process is as shown in Figure 1 , comprising the following steps:

[0065] 1) Collecting infrared image and visible light image of the same scene respectively;

[0066] Collecting original image of the same scene by infrared image acquisition device and visible light image acquisition device respectively.

[0067] 2) Enhancing pixel of infrared image by image enhancement algorithm to obtain enhanced image;

[0068] Firstly, the original infrared image is decomposed into base layer image and detail layer image by guided filter, then the pixels of base layer image and detail layer image are enhanced respectively, and finally the processed base layer image and detail layer image are added to obtain the final enhanced image, so as to realize pixel enhancement of infrared image and improve the contrast of image. The image enhancement algorithm specifically includes the following steps:

[0069] A) The pixel value P r (i,j) of each pixel point in the infrared image is filtered by guided filter formula to obtain the base layer image pixel value

[0070]

[0071] The GF(·) function is a guided filter;

[0072] (i,j) is the coordinate of the pixel point;

[0073] s is the filter radius of the filter;

[0074] ε is the filter coefficient;

[0075] B) The detail layer image pixel value P (i,j) of each pixel point in the infrared image is obtained by the following formula:

[0076]

[0077] C) The base layer image pixel value P and the detail layer image pixel value P of each pixel point are enhanced respectively by the following formula to obtain the enhanced base layer image pixel value P and the enhanced detail layer image pixel value P

[0078]

[0079] α is the enhancement coefficient;

[0080] μ is the average value of the gray values of all pixel points in the image;

[0081] Beta is a detail enhancement coefficient, and is 2;

[0082] abs() represents the absolute value;

[0083] D) the final image pixel value P of each pixel point after enhancement is obtained by the following formula r E (i,j):

[0084]

[0085] E) the final image pixel value P of all pixel points in the infrared image after enhancement is obtained r E (i,j), and the enhanced image is obtained, as shown in the following formula Figure 2 After the image is enhanced by the pixel points, the contrast of the image is obviously improved, and the subsequent feature point extraction is facilitated.

[0086] 3) In order to reduce the complexity of feature point extraction, a multi-directional gradient mean method is used to extract the contours of the visible light image and the enhanced image respectively, and infrared contour images and visible light contour images are obtained;

[0087] The multi-directional gradient mean method in the embodiment of the application adopts seven directions, and the corresponding angles are 0°, 30°, 60°, 90°, 120°, 150° and 180°. In the application, the interval between the gradients is 30°. In other embodiments, other gradient mean methods with other intervals and covering 0-180° can also be adopted, but it should be noted that if the interval is too small, the algorithm for image contour extraction becomes complex, the calculation amount becomes large, and the extraction rate is greatly slowed down. If the interval is too large, the image information cannot be accurately expressed, the contour extraction is blurred or incorrect, and the difficulty of feature point extraction is increased.

[0088] The specific method of the above multi-directional gradient mean method for contour extraction of the image is as follows: the gradient values of the image in 0°, 30°, 60°, 90°, 120°, 150° and 180° are calculated respectively, and then the gradient value mean of the image in the seven directions is calculated through the mean() function, and the contour image GM is obtained through the following formula:

[0089]

[0090] Theta: gradient direction;

[0091] g θ : gradient value corresponding to the gradient direction;

[0092] Mean() function: mean value of multiple direction gradient values.

[0093] As shown in the following formula Figure 3As shown, it can be seen that using this method can completely retain the contour feature information of the infrared image and the visible light image, and lay a foundation for subsequent feature point extraction.

[0094] 4) using feature extraction algorithm through infrared contour image and visible light contour image respectively to extract all feature points of enhanced image and visible light image;

[0095] The present feature extraction methods include SIFT, Harris, SURF and PIIFD, and the SURF feature point extraction algorithm is selected in the embodiment of the application to extract the feature points of the infrared image and the visible light image.

[0096] The SURF feature point extraction operator is mainly based on the Hessian matrix, and the Hessian matrix of any point u in the image I is defined as follows:

[0097]

[0098] The first-order Gaussian convolution is as follows: G(u,σ) is a Gaussian filter, σ is a filter scale, is a convolution operator. xx , L xy and L yy are the second-order Gaussian convolutions of the image I in the horizontal direction, the horizontal and vertical direction and the vertical direction respectively. The first-order Gaussian difference function D(u,σ) is obtained through the following formula: D(u,σ) = L(u,kσ)-L(u,σ), wherein k is a scale difference. The determinant det(H L (u,σ)) of the Hessian matrix is obtained through the following formula, and is as follows:

[0099] det(H) = D xx D yy -(wD xy ) 2

[0100] wherein the relative weight w of the filter response is used to balance the expression of the determinant of the Hessian matrix, D xx , D xy and D yy are the second-order Gaussian difference functions of the original image I in the horizontal direction, the horizontal and vertical direction and the vertical direction respectively. The determinant det(H L(u,σ)) to obtain all feature points of the enhanced image and all feature points of the visible light image. Then, a 3x3x3 neighborhood non-maximum suppression method is used to obtain the final feature points in different sizes of the image, i.e. each pixel point is compared with 26 points in its three-dimensional neighborhood to obtain the final feature points. The purpose of obtaining the final feature points is to select the most critical feature points in the image by comparison, thereby reducing the number of feature points and the calculation amount of feature point registration, and enhancing the feature points of the enhanced image and the feature points of the visible light image as shown in Figure 4 .

[0101] 5) The dynamic point matching algorithm is used to match the final feature points of the enhanced image and the visible light image to complete image registration. The dynamic point matching algorithm includes the following steps:

[0102] S1) The feature point outliers of the enhanced image and the visible light image are eliminated by a single Gaussian model; the Gaussian model is as follows:

[0103]

[0104] E(f,σ 2 ) is an energy function about affine transformation;

[0105] f is a radio transformation function;

[0106] σ is a filter scale;

[0107] M and N respectively represent the number of final feature points of the visible light image and the enhanced image;

[0108] x j is a feature point of the enhanced image, and j is the serial number of the corresponding feature point;

[0109] y i is a feature point of the visible light image, and i is the serial number of the corresponding feature point;

[0110] η is the corresponding relationship between the feature matching points, which is 1 when the two feature points are in matching relationship, and 0 otherwise;

[0111] φ is a Gaussian function;

[0112] In order to calculate accurately, it is assumed that y i -f(x j ) satisfies a single Gaussian distribution.

[0113] S2) The energy function of the Gaussian model is iteratively calculated by using a weighted regularization least squares method, and the projection transformation matrix (i.e. the corresponding matching feature point relationship) is obtained when the energy function is minimized. In this embodiment, as shown in Figure 5As shown, through iterative calculation, the final enhanced image and the visible light image obtain 9 sets of matching feature points, and the projection transformation matrix of the infrared image can be obtained through the 9 sets of matching feature points, and the specific definition is as follows:

[0114]

[0115] Wherein, [x, y] is the original image coordinate, [u, v] is the transformed image coordinate, a, b, c, d, e, g is the projection transformation coefficient. It can be seen from the above formula that there are six position coefficients, so at least six or more matching feature points are obtained through iterative calculation, and the above projection transformation matrix can be obtained.

[0116] S3) All matching feature points of the enhanced image and the visible light image are obtained through the projection transformation matrix, and then the enhanced image is projected to the visible light image through the projection transformation, as shown in Figure 6 As shown, the infrared rotation image obtained through the projection transformation can be seen that the enhanced image obtained after the pixel enhancement of the infrared image has high matching accuracy with the visible light image, and the two images are basically overlapped and aligned.

[0117] In order to verify the registration effect of the present application, the images after registration are fused to obtain a fused image, as shown in Figure 7 As can be seen from the figure, the fused image has been completely aligned, indicating that the registration effect in the early stage is good.

Claims

1. A method for registering infrared and visible light images, characterized in that, Includes the following steps: 1) Acquire infrared and visible light images of the same scene separately; 2) An enhanced image is obtained by performing pixel enhancement processing on the infrared image using an image enhancement algorithm; In step 2), the image enhancement algorithm includes the following steps: A) The pixel value P of each pixel in the infrared image is obtained by using the guided filtering formula. r (i,j) is filtered to obtain the base image pixel value of the corresponding pixel point. The GF(·) function is a guided filter; (i,j) are the coordinates of the pixel; s is the filter radius; ε is the filter coefficient; B) Obtain the detail layer pixel value of each pixel in the infrared image using the following formula. C) The pixel values ​​of the base image for each pixel are calculated using the following formulas. and detail layer image pixel values Pixel enhancement is performed to obtain the enhanced pixel values ​​of the base image. and detail layer image pixel values α is the enhancement coefficient; μ is the average gray value of all pixels in the image; β is the detail enhancement factor, with a value of 2; abs(·) represents absolute value; D) Obtain the final image pixel value P after enhancement for each pixel using the following formula. r E (i,j): E) The final image pixel value P after enhancement of all pixels in the infrared image. r E (i,j) yields the enhanced image; 3) The multi-directional gradient averaging method is used to extract contours from the visible light image and the enhanced image respectively, to obtain infrared contour images and visible light contour images; 4) Using feature extraction algorithms, all feature points of the enhanced image and the visible light image are extracted from the infrared contour image and the visible light contour image respectively, and then the final feature points of the enhanced image and the visible light image are obtained from all feature points; 5) Use a dynamic point matching algorithm to match the final feature points of the enhanced image and the visible light image to complete image registration.

2. The infrared and visible light image registration method according to claim 1, characterized in that: In step 3, the multi-directional gradient mean method uses seven directions, with corresponding angles of 0°, 30°, 60°, 90°, 120°, 150°, and 180°. Contour extraction from an image using the multi-directional gradient mean method involves: first, calculating the gradient values ​​of the image in seven directions (0°, 30°, 60°, 90°, 120°, 150°, and 180°); then, calculating the mean of these gradient values ​​using the mean() function; and finally, obtaining the contour image GM using the following formula: θ: Gradient direction; g θ : The gradient value corresponding to the gradient direction; The mean() function: the average of gradient values ​​in multiple directions.

3. The infrared and visible light image registration method according to claim 2, characterized in that: In step 4), the feature extraction algorithm uses the SURF feature point extraction algorithm.

4. The infrared and visible light image registration method according to claim 3, characterized in that: Step 5) specifically includes the following steps: S1) Eliminate feature point outliers in enhanced and visible light images using a single Gaussian model; The Gaussian model is as follows: E(f,σ 2 Let be the energy function with respect to the affine transformation; f is the radial transformation function; σ is the filtering scale; M and N represent the final number of feature points in the visible light image and the enhanced image, respectively; x j To enhance the feature points of the image, j is the corresponding feature point index; y i For the feature points of the visible light image, i is the corresponding feature point number; η represents the correspondence between feature matching points. Its value is 1 when two feature points are matched, and 0 otherwise. φ is a Gaussian function; S2) The energy function of the Gaussian model is iteratively calculated using the weighted regularized least squares method, and the projection transformation matrix is ​​obtained when the energy function is minimized; S3) Obtain all matching feature points of the enhanced image and the visible light image through the projection transformation matrix, and then project the enhanced image onto the visible light image through the projection transformation matrix to obtain the infrared rotated image, thus completing the image registration.

5. The infrared and visible light image registration method according to claim 4, characterized in that: In step 4), the 3×3×3 neighborhood non-maximum suppression method is used to obtain the final feature points.

Images