A multi-source image registration method at different scales

Abstract

This invention belongs to the technical field of multi-source image registration methods, specifically involving a multi-source image registration method at different scales. Step 1: Multi-source image feature point extraction. Considering that feature points in multi-source images remain unchanged in scale and direction, the method incorporates the inverse gradient changes of the same target in the image caused by different sensor acquisition methods, improving the effectiveness of feature point extraction. Step 2: Geometric constraint matching. Geometric constraints on target images acquired by multiple sensors are added, taking into account the inherent geometric conditions between sensor-acquired images. The geometrically constrained feature points overcome the difficulty of incorrect matching caused by the complexity of describing feature point gradient changes in multi-source images, improving the accuracy and robustness of feature point matching. Step 3: Image registration parameter optimization. By employing a least-squares model, the registration parameters are optimized, improving the accuracy of the final image registration.

Description

Technical Field

[0001] This invention relates to the technical field of multi-source image registration methods, specifically to a multi-source image registration method at different scales. Background Technology

[0002] With the rapid development of new materials and processes, sensors are becoming increasingly miniaturized and integrated. Data from single sensors is no longer sufficient to meet user needs, and multi-modal, multi-sensor integrated image acquisition is gradually becoming the primary method. Among these, thermal infrared imaging and visible light imaging are the most widely used combinations. Visible light imaging utilizes reflected light from the target, sharing the same wavelength as human vision, providing rich and clear target information. Its large information content facilitates the establishment of corresponding optical imaging models. Its main drawback is that in poor imaging conditions, such as smoke or low light, reflected light is significantly reduced, affecting image quality. Infrared detectors receive the thermal radiation from the target and its surrounding environment, converting the detected radiation into image grayscale information, thus forming an infrared image. Typically, the image grayscale value is proportional to the target's radiation intensity. When the temperature difference between the target and the background is large, the grayscale difference in the generated infrared image is more pronounced. Infrared images have a unique advantage in highlighting thermal targets such as humans and vehicles, facilitating target identification even in low-light conditions. The main drawbacks of infrared images are that, due to the influence of the application environment and hardware conditions, infrared images have high noise, poor visual effects, low resolution, and significant loss of the appearance features of the target.

[0003] Because infrared and visible light sensors have different imaging mechanisms and application scenarios, integrating the two types of images has significant application value. Utilizing infrared and visible light fusion detection can significantly improve the ability to acquire target images in all weather conditions and at all times. Simultaneously, leveraging the texture of the fused image can significantly improve target recognition capabilities. Effectively registering visible light and infrared images at different resolutions is of great importance in fields such as intelligent driving, environmental monitoring, and medical diagnosis.

[0004] Image registration is the process of matching two or more images from different times, spaces, and modes according to certain criteria.

[0005] In the process of image overlay, the key to registration is finding the optimal mapping relationship between different images. Based on different registration methods, there are two main categories: pixel-based registration techniques and feature-based registration techniques. Pixel-based registration utilizes global or local pixel grayscale information for registration; it is simple to implement but computationally expensive. Feature-based registration primarily establishes correspondences and mapping relationships between salient features of images; it is relatively less computationally expensive, more robust, and suitable for time-sensitive scenarios such as autonomous driving. For feature-based matching methods, the primary problem to solve is feature point extraction and the construction of matching relationships. Lowe et al. proposed Scale Invariant Feature Transform (SIFT) to address rotation, scaling, and contrast differences between images to be matched. This method has good robustness and has been widely used in various matching scenarios. Bay et al. proposed a fast robust feature algorithm that improves the SIFT algorithm using integral images and Hessian matrices, increasing its efficiency. Rublee et al. proposed the ORB feature extraction algorithm based on FAST feature point detection and BRIEF feature descriptors, improving the speed of feature extraction and matching. Liu et al. proposed an affine and contrast-invariant description method, improving the success rate of image matching. However, existing methods still have many problems in matching different types of sensors, especially in the registration of infrared and visible light images. The gradient changes of the same target in multi-mode images are often different, and sometimes the gradient transformations of corresponding regions may be reversed. In addition, the gray-level distribution and variation characteristics are also different, resulting in inconsistent gradient changes and intensity distributions. On the one hand, infrared and visible light images represent two different physical phenomena and show different target characteristics. Infrared images mainly show the spectral characteristics of the target in the 8-14μm band, while visible light mainly shows the spectral characteristics in the 380nm-780nm band. The difference in spectral ranges reduces the matching features between the same target, and sometimes even leads to incorrect matching. In addition, the resolution of infrared images is usually low, with a large scale difference compared to the resolution of visible light images. All of these factors make matching infrared and visible light images difficult. To address this, a multi-source image registration method with different scales is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-source image registration method at different scales to solve the technical problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a multi-source image registration method at different scales, the method comprising the following steps:

[0008] Step 1: Feature point extraction from multi-source images

[0009] Using the Harris method, control points of the image to be registered are calculated. A Gaussian window function is used to detect pixel grayscale variation features in various directions, identifying points of maximum local curvature change. For image I, the Gaussian window function is w(u,v), and the structure matrix N is as follows:

[0010]

[0011] In the formula, I u I v This represents the derivative of the image coordinates (u, v) along the two coordinate axes, respectively.

[0012] Let λ1 and λ2 be the eigenvalues ​​of N, and define R = λ1λ2 - k(λ1 + λ2). 2 R is the cost function, where k is an empirical constant, ranging from 0.04 to 0.06. When R exceeds the threshold, the image coordinate point is considered to be a candidate feature point.

[0013] Step 2: Matching Geometric Constraints

[0014] During data acquisition, different sensors for the same target exhibit certain spatial geometric relationships, which in turn lead to geometric spatial relationships between the acquired image data. Specifically, the distances between feature points and the relationships in the horizontal and vertical directions are geometrically related. Therefore, based on the candidate feature points with scale-invariant properties selected in the first step, geometric constraints are applied to the X, Y, and R directions of the matched feature point pairs. The expected value of the absolute differences in the X, Y, and R directions is then statistically analyzed for all initially matched feature point pairs. Given the variances δx, δy, and δr, analyze point-by-point whether the feature point pairs satisfy the following relationship:

[0015]

[0016] In the formula, ΔX, ΔY, and ΔR represent the differences in distances between feature point pairs at X, Y, and R, respectively;

[0017] If the candidate feature point pairs satisfy the above three geometric constraints, the feature points are retained for subsequent parameter calculations for final image registration; otherwise, the points are discarded.

[0018] Step 3: Image registration parameter optimization

[0019] The transformation model G between the pixel values ​​I1(x,y) and I2(x,y) at image (x,y) is calculated, i.e.

[0020] I1(x,y)=G(I2(x,y))

[0021] To this end, different scale factors between multi-source images are incorporated into the transformation model, while considering the effects of translation, rotation, and perspective transformation. The parameters are optimized using the least squares method through the matched feature point pairs.

[0022]

[0023] In the formula, x, y, x', and y' are the image coordinates corresponding to the matched feature points, r and r' are the scale factors between the images, and A is the transformation matrix. 11 A 12 A 21 A 22 Represents the rotation matrix between images, A 13 and A 23 Let A be a translation matrix. 31 A 32 A 33 As the scaling factor matrix, let A 31 A 32 Set to 0, A 33 Set it to 1, and define the objective function C:

[0024] C = min(Δx, Δy)

[0025] In the formula, Δx and Δy represent X and Y obtained from the solution of redundant feature points and the statistical results obtained from the least squares model.

[0026] The difference between the expected values ​​of Y is used to calculate the image registration model G according to the objective function, thereby completing the registration between multiple source images.

[0027] Preferably, after extracting candidate feature points in step 1, the gradient direction of each point in the 4*4 neighborhood window of the feature point is calculated using the SIFT feature descriptor algorithm.

[0028] Preferably, two pairs of opposite directions are re-examined to check whether they are reverse feature points. If they are reverse feature points, the feature points are retained. Candidate feature points in the multi-modal image are retained according to the correspondence between feature descriptors, and candidate feature points that satisfy the scale orientation invariant property are retained.

[0029] Preferably, in step 2, if the final number of feature points to be solved is less than 5, the K value in step 1 is reduced, candidate feature points are increased, and matching is performed again to select a sufficient number of feature points.

[0030] The beneficial effects of this invention are:

[0031] (I) Multi-source image feature point extraction method. This method, based on the fact that the feature points in multi-source images are invariant in scale and direction, adds the reverse change in gradient of the same target in the image caused by different sensor acquisition methods, which improves the effectiveness of feature point extraction, improves the efficiency of the algorithm in direction matching, and uses 4 directions as descriptive operators to reduce the computational load of the algorithm.

[0032] (ii) Geometric constraint matching method. This method adds geometric constraints to the target images acquired by multiple sensors, takes into account the inherent geometric conditions between the images acquired by the sensors, and overcomes the difficulty of incorrect matching caused by the complexity of the gradient change description of feature points in multi-source images by using geometrically constrained feature points, thereby improving the accuracy and robustness of feature point matching.

[0033] (III) Image registration parameter optimization method. This method improves the problem of low image registration accuracy caused by the inconsistency of scale between multi-source images. During image registration, the scale factor is used as a parameter for translation and rotation calculation. By adopting the least squares model, the registration parameters are optimized, thereby improving the accuracy of the final image registration. Attached Figure Description

[0034] Figure 1 This is a flowchart of the multi-source image registration method at different scales according to the present invention;

[0035] Figure 2 The image registration results for visible light and near-infrared images in an embodiment of the present invention are shown. Detailed Implementation

[0036] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0037] Example

[0038] Data is collected from the same scene using visible light and near-infrared sensors. Due to the different sensors, the resolution and range of the acquired images differ. By using the method proposed in this invention, corresponding target points in two images can be automatically identified, the geometric transformation relationship between the two images can be calculated, and the registration between the same target pixels in the two images can be achieved. Figure 2 The two endpoints of the middle line represent the corresponding target points in the two images after registration.

[0039] Step 1: Multi-source image feature point extraction. First, the Harris method is used to calculate the control points of the image to be registered. Then, a Gaussian window function is used to detect the pixel grayscale variation features in various directions of the image, and to detect whether it is a point with the maximum local curvature change. Image I, the Gaussian window function is w(u,v), and the structure matrix N is as follows:

[0040]

[0041] In the formula, I u I v Let λ1 and λ2 be the eigenvalues ​​of N, and define R = λ1λ2 - k(λ1 + λ2). 2 R is the cost function, where k is an empirical constant with a value of 0.04-0.06. When R exceeds the threshold, the image coordinate point is considered to be a candidate feature point.

[0042] After extracting candidate feature points, the SIFT feature descriptor algorithm is used to calculate the gradient direction of each point within a 4x4 neighborhood window. Since gradient distributions differ in multi-modal images, four gradient directions are set. Simultaneously, two pairs of opposite directions are re-checked to verify if they are reverse feature points; if so, the feature point is retained. Based on the correspondence between feature descriptors, candidate feature points in the multi-modal image that satisfy the scale-invariant property are retained.

[0043] Step 2: Geometric Constraint Matching. During data acquisition, different sensors for the same target exhibit certain spatial geometric relationships, leading to geometric spatial relationships between the acquired image data. Specifically, feature points exhibit geometric spatial relationships in the distance, horizontal, and vertical directions. Therefore, based on the candidate feature points with scale-invariant properties selected in Step 1, geometric constraints are applied to the X, Y, and R directions of the matched feature point pairs. First, the expected value of the absolute differences in the X, Y, and R directions is statistically analyzed for all initially matched feature point pairs. Given the variances δx, δy, and δr, analyze point-by-point whether each pair of feature points satisfies the following relationship:

[0044]

[0045] In the formula, ΔX, ΔY, and ΔR represent the differences between feature point pairs in X, Y, and distance, respectively.

[0046] If candidate feature point pairs satisfy the above three geometric constraints, the feature points are retained for subsequent parameter calculations in the final image registration. If the geometric constraints are not satisfied, the point is discarded. Furthermore, if the final number of feature points calculated is less than 5, the K value in the first step is reduced, candidate feature points are increased, and matching is performed again until a sufficient number of feature points are selected.

[0047] Step 3: Image registration parameter optimization. Image registration mainly involves calculating the transformation model G between the pixel values ​​I1(x,y) and I2(x,y) at (x,y) in the image from multiple sources.

[0048] I1(x,y)=G(I2(x,y))

[0049] Therefore, this invention incorporates different scale factors between multi-source images into the transformation model, while also considering the effects of translation, rotation, perspective transformation, etc., and optimizes the parameters using the least squares method through the matched feature point pairs.

[0050]

[0051] In the formula, x, y, x', and y' are the image coordinates corresponding to the matched feature points, r and r' are the scale factors between the images, and A is the transformation matrix. 11 A 12 A 21 A 22 Represents the rotation matrix between images, A 13 and A 23 Let A be a translation matrix. 31 A 32 A 33 As the scaling factor matrix, let A 31 A 32 Set to 0, A 33 Set it to 1, and define the objective function C:

[0052] C = min(Δx, Δy)

[0053] In the formula, Δx and Δy are the differences between X and Y obtained from the solution of redundant feature points and the expected values ​​of X and Y obtained from the least squares model.

[0054] Based on the objective function, the image registration model G is calculated to complete the registration between multi-source images. This embodiment solves the problem of inconsistent gradient information between multi-source data in practical applications. It comprehensively considers issues such as image scale, gradient, geometry, and overall transformation optimization to achieve automatic matching results between images to be matched, thus solving the problem of difficulty in matching feature points due to scaling between images of different scales.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for registering multi-source images at different scales, characterized in that: The method includes the following steps: Step 1: Feature point extraction from multi-source images Using the Harris method, control points of the image to be registered are calculated. A Gaussian window function is used to detect pixel grayscale variation features in various directions, identifying points of maximum local curvature change. For image I, the Gaussian window function is w(u,v), and the structure matrix N is as follows: In the formula, I u I v This represents the derivative of the image coordinates (u, v) along the two coordinate axes, respectively. Let λ1 and λ2 be the eigenvalues ​​of N, and define R = λ1λ2 - k(λ1 + λ2). 2 R is the cost function, where k is an empirical constant, ranging from 0.04 to 0.

06. When R exceeds the threshold, the image coordinate point is considered to be a candidate feature point. Step 2: Matching Geometric Constraints Geometric constraints are applied to the X, Y, and R directions of the matched feature point pairs. The expected value of the absolute differences in the X, Y, and R directions is statistically analyzed for all initially matched feature point pairs. Given the variances δx, δy, and δr, analyze point-by-point whether the feature point pairs satisfy the following relationship: In the formula, ΔX, ΔY, and ΔR represent the differences in distances between feature point pairs at X, Y, and R, respectively; If the candidate feature point pairs satisfy the above three geometric constraints, the feature points are retained for subsequent parameter calculations for final image registration; otherwise, the points are discarded. Step 3: Image registration parameter optimization The transformation model G between the pixel values ​​I1(x,y) and I2(x,y) at image (x,y) is calculated, i.e. I1(x,y)=G(I2(x,y)) Different scale factors among multi-source images are incorporated into the transformation model, and the parameters are optimized using the least squares method through the matched feature point pairs. In the formula, x, y, x', and y' are the image coordinates corresponding to the matched feature points, r and r' are the scale factors between the images, and A is the transformation matrix. 11 A 12 A 21 A 22 Represents the rotation matrix between images, A 13 and A 23 Let A be a translation matrix. 31 A 32 A 33 As the scaling factor matrix, let A 31 A 32 Set to 0, A 33 Set it to 1, and define the objective function C: C = min(Δx, Δy) In the formula, Δx and Δy represent X and Y obtained from the solution of redundant feature points and the statistical results obtained from the least squares model. The difference between the expected values ​​of Y is used to calculate the image registration model G according to the objective function, thereby completing the registration between multiple source images.

2. The multi-source image registration method at different scales according to claim 1, characterized in that: After extracting candidate feature points in step 1, the gradient direction of each point in the 4*4 neighborhood window of the feature point is calculated using the SIFT feature descriptor algorithm.

3. The multi-source image registration method at different scales according to claim 2, characterized in that: Two pairs of opposite directions are re-examined to check whether they are reverse feature points. If they are reverse feature points, the feature points are retained. Candidate feature points in the multi-modal image are retained based on the correspondence between feature descriptors, and those that satisfy the scale orientation invariant property are retained.

4. The multi-source image registration method at different scales according to claim 1, characterized in that: In step 2, if the final number of feature points to be solved is less than 5, the K value in step 1 is reduced, candidate feature points are increased, and matching is performed again to select a sufficient number of feature points.

Images