Cross-view-angle different-source image matching method for airborne sensor

Abstract

The invention relates to the technical field of image processing, in particular to a cross view angle different-source image matching method for an airborne sensor, and the method comprises the steps: obtaining a to-be-matched registration image and a to-be-matched reference image; respectively extracting edge features of the registration image and the reference image; performing coarse alignment on the registration image and the reference image; matching is carried out on the basis of the coarse alignment position to obtain feature pairs, and the feature pairs are sorted into a multi-scale feature pair set; and on the basis of the multi-scale feature pair set, estimating a transformation matrix between the images through an RANSAC algorithm, and finally completing fine image matching. According to the method, through a mechanism of combining rapid coarse registration based on image gradient edge features and multi-scale feature fine registration based on image phase information, the balance between the matching precision and the calculation speed is effectively realized, and meanwhile, the method has relatively good robustness for airborne different-source images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a cross-viewing angle heterogeneous image matching method for airborne sensors. Background Art

[0002] With the rapid development of airborne sensor technology, in order to adapt to future complex battlefield situations and diversified combat missions, the capabilities that airborne air-to-surface detection systems can cover are expanding, and the functional compatibility and performance complementarity between payloads within the system are becoming more and more obvious. Multi-spectral optoelectronic detection and heterogeneous sensor collaboration have become key features of modern advanced airborne air-to-surface imaging detection equipment.

[0003] In advanced airborne air-to-surface detection systems, in order to maximize combat effectiveness, heterogeneous sensors with different imaging spectrum bands and different imaging mechanisms are not simply stacked, but highly coordinated at all levels from task planning to information processing. Taking the coordination of optoelectronic-SAR sensors in Rafael's Litening pod as an example, SAR sensors make up for the shortcomings of optoelectronic sensors that are significantly restricted by meteorological conditions and difficult to locate at long distances with high precision, while optoelectronic sensors solve the problems of low upper limit of close-range imaging resolution and poor real-time imaging of SAR sensors. For heterogeneous imaging detection sensors, high-precision real-time registration technology of cross-view heterogeneous images ensures pixel-level alignment of images across sensors and high-precision fusion of detection information across systems, which is a prerequisite for efficient coordination of multiple sensors within the system. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a cross-view heterogeneous image matching method for airborne sensors, which is mainly a cross-view heterogeneous image matching method between cross-spectral heterogeneous airborne sensors or between airborne sensors and digital maps. The present invention preliminarily aligns the structural features of ground scenes through projection transformation, and then effectively achieves a balance between matching accuracy and calculation speed through a mechanism combining fast coarse registration based on image gradient edge features and multi-scale feature fine registration based on image phase information. At the same time, the method has good robustness for airborne heterogeneous images.

[0005] The first object of the present invention is to provide a cross-view heterogeneous image matching method for airborne sensors, comprising: Acquire a registration image and a reference image to be matched; The projection transformation method is used to correct the viewing angle difference between the registration image and the reference image, and then the gray gradient information is used to extract the edge features of the registration image and the reference image respectively. The registered image and the reference image are roughly aligned based on the Hausdorff distance between the edge features of the registered image and the reference image; The sizes of the registration image and the reference image are compressed to construct the corresponding multi-scale image sequence, and the HOPC template features are extracted iteratively. Then, feature pairs are obtained by matching based on the rough alignment positions and organized into a multi-scale feature pair set. Based on a set of multi-scale feature pairs, the transformation matrix between images is estimated through the RANSAC algorithm, and finally the image precise matching is completed.

[0006] Preferably, a projection transformation method is used to correct the viewing angle difference between the registration image and the reference image, including using the following projection transformation matrix to achieve the transformation:

[0007] In the formula, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system respectively; and are the position vectors from the center of the scene to the reference image sensor coordinate system and the registration image coordinate system respectively; is the internal parameters of the reference image sensor; To align the image sensor internal parameters, the sensor internal parameters can be obtained based on design or calibration.

[0008] Preferably, the Hausdorff distance between edge features is used as the objective function for coarsely aligning the registration image and the reference image:

[0009] In the formula, and and are the edge feature point sets of the registration image and the reference image respectively.

[0010] Preferably, the sizes of the registration image and the reference image are compressed to construct corresponding multi-scale image sequences, including: is the minimum compression ratio, The registration image and the reference image are compressed to construct a multi-scale image sequence, forming a registration image sequence. and the reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and The image is consistent with the original size.

[0011] Preferably, the process of acquiring a set of multi-scale feature pairs includes: First, in the scaled image and Window size Extract the HOPC template features, then search for matching feature pairs centered on the rough alignment position to form a feature pair set , based on this set, the RANSAC algorithm is run to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix, the registration position is corrected, and then based on the new registration position, the process is repeated with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

[0012] Preferably, all feature pairs are set Integrate into a multi-scale feature pair set , based on the multi-scale feature set, the RANSAC algorithm is run to estimate the final transformation matrix , complete the precise matching between the registered image and the reference image.

[0013] Preferably, the edge features are obtained by using Canny or Sobel, an edge extraction algorithm based on the grayscale gradient information of the image.

[0014] Preferably, the registration image to be matched and the reference image both include airborne photoelectric sensor images, synthetic aperture radar sensor images or digital map images.

[0015] The second object of the present invention is to provide a cross-view heterogeneous image matching system for airborne sensors, comprising: The image acquisition module is used to obtain the registration image and the reference image to be matched; the projection transformation method is used to correct the viewing angle difference between the registration image and the reference image, and then the grayscale gradient information is used to extract the edge features of the registration image and the reference image respectively; A coarse alignment module, for coarsely aligning the registration image with the reference image based on the Hausdorff distance between the edge features of the registration image and the reference image; The feature pair extraction module is used to compress the sizes of the registration image and the reference image to construct the corresponding multi-scale image sequence, iteratively extract the HOPC template features, and then match them based on the rough alignment position to obtain feature pairs, and organize them into a multi-scale feature pair set; The precise alignment module is used to estimate the transformation matrix between images through the RANSAC algorithm based on a set of multi-scale feature pairs, and finally complete the precise matching of images.

[0016] The present invention has at least the following beneficial effects: The present invention provides a cross-view heterogeneous image matching method for airborne sensors. The present invention targets an input image pair (a registration image and a reference image). The present invention first aligns the visual axis of the reference image with the registration image through a projection transformation. This projection transformation takes the posture information of the two sensors and the internal parameter information of the sensors as input to calculate the projection transformation matrix between the images. The calculation of the projection transformation matrix is ​​divided into two parts. The first step is to project the reference image to the world coordinate system, and the second step is to project the image again to the registration image sensor coordinate system.

[0017] Although the visual axis difference between image pairs can be solved to a certain extent through projection transformation, the significant nonlinear radiometric difference between image pairs must be adapted through matching algorithms. Existing high-precision matching methods often have high computational complexity, making it difficult to directly apply them in airborne environments and under high real-time requirements. The present invention designs a coarse-and-fine combined two-level matching method, which uses an algorithm with relatively low accuracy but high computational efficiency in coarse registration to significantly reduce the search range of the registration position at a faster speed; and uses an algorithm with high accuracy but complex computation in fine registration to ensure its computational efficiency with a small search range.

[0018] In the coarse registration, the present invention uses edge features based on image gradient information for image matching. The methods that can be used are typically Canny algorithm and Sobel algorithm. This type of algorithm has high computational efficiency and can quickly complete the extraction of image structural features. The coarse registration process uses the Hausdorff distance of the image to the edge feature point set as the optimization objective function, and takes the minimum Hausdorff distance position as the coarse registration position.

[0019] In the precise registration, the present invention uses the HOPC feature based on the image phase information for image matching, and this feature has good adaptability to the nonlinear radiometric differences between heterogeneous images. The present invention designs a multi-scale feature pair set extraction mechanism, and obtains a multi-scale image sequence by compressing the image according to a series of equally spaced ratios. By iteratively running the HOPC feature and matching feature pair search, RANSAC registration position estimation and registration position update process in the image sequence, feature pair sets of different scales can be obtained, and these sets are integrated to obtain a multi-scale feature pair set. Finally, the present invention runs the RANSAC algorithm based on the multi-scale feature pair set to obtain the transformation matrix between the registered image and the reference image, and completes the image matching process.

[0020] The method provided by the present invention can achieve a balance between the operation efficiency of the matching algorithm and the registration accuracy, and can also adapt to the problem that the viewing angle and scene scale of the airborne sensor image vary widely with the detection distance, thereby effectively improving the real-time, accuracy and robustness of image matching between airborne heterogeneous sensors and between airborne sensors and digital maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the cross-view heterogeneous image matching algorithm flow for airborne sensors.

[0022] Figure 2 This is the effect of cross-view heterogeneous image matching.

[0023] Figure 3 This is the effect of cross-view heterogeneous image matching. DETAILED DESCRIPTION

[0024] In order to illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description in conjunction with embodiments.

[0025] The present invention mainly aims at the cross-view heterogeneous image matching task of airborne sensors, which needs to complete the alignment between two images with different imaging perspectives, different imaging mechanisms (imaging bands, detector systems, etc.), different sensor internal parameters (focal length, sensitivity, optical transmittance, etc.) and different environment parameters (ambient illumination, atmospheric transmittance, etc.). The difference in perspective will cause obvious changes in the outline of the scene in the image, while the differences in imaging mechanism, sensor parameters and environmental parameters will cause significant nonlinear radiometric differences between images. At the same time, since the airborne sensor needs to observe ground targets at different distances, the range of variation of the visual axis inclination during imaging is large, resulting in large scale changes in the typical ground scene features within the frame, further improving the requirements for the adaptability and robustness of the registration algorithm. The present invention is designed for the task of cross-view heterogeneous image matching on airborne sensors, and the structural features of the ground scene are preliminarily aligned through projection transformation. Then, through the mechanism of combining fast coarse registration based on image gradient edge features and multi-scale feature fine registration based on image phase information, the balance between matching accuracy and calculation speed is effectively achieved, and the method has good robustness for airborne heterogeneous images.

[0026] In order to achieve the above object, the present invention provides a cross-view heterogeneous image matching method for airborne sensors, comprising: S1, obtaining a registration image and a reference image to be matched; The registration image and the reference image to be matched both include airborne photoelectric sensor images, synthetic aperture radar sensor images or digital map images.

[0027] S2, using the projection transformation method to correct the viewing angle difference between the registration image and the reference image, and then using the gray gradient information to extract the edge features of the registration image and the reference image respectively; The projection transformation method is used to correct the perspective difference between the registration image and the reference image, including using the following projection transformation matrix to achieve the transformation:

[0028] In the formula, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system respectively; and are the position vectors from the center of the scene to the reference image sensor coordinate system and the registration image coordinate system respectively; is the internal parameters of the reference image sensor; To align the image sensor internal parameters, the sensor internal parameters can be obtained based on design or calibration.

[0029] The edge features are obtained by using Canny or Sobel edge extraction algorithm based on image grayscale gradient information.

[0030] S3, coarsely aligning the registration image and the reference image based on the Hausdorff distance between the edge features of the registration image and the reference image; The Hausdorff distance between edge features is used to roughly align the registration image with the reference image:

[0031] In the formula, and and are the edge feature point sets of the registration image and the reference image respectively.

[0032] S4, compressing the sizes of the registration image and the reference image respectively to construct corresponding multi-scale image sequences, iteratively extracting HOPC template features, then matching based on the coarsely aligned positions to obtain feature pairs, and arranging them into a multi-scale feature pair set; Compress the sizes of the registration image and the reference image to construct the corresponding multi-scale image sequence, including: is the minimum compression ratio, The registration image and the reference image are compressed to construct a multi-scale image sequence, forming a registration image sequence. and the reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and The image is consistent with the original size.

[0033] The process of acquiring a multi-scale feature pair set includes: First, in the scaled image and Window size Extract the HOPC template features, then search for matching feature pairs centered on the rough alignment position to form a feature pair set , based on this set, the RANSAC algorithm is run to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix, the registration position is corrected, and then based on the new registration position, the process is repeated with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

[0034] S5. Based on the multi-scale feature pair set, the transformation matrix between images is estimated through the RANSAC algorithm, and finally the image precise matching is completed.

[0035] Set all feature pairs Integrate into a multi-scale feature pair set , based on the multi-scale feature set, the RANSAC algorithm is run to estimate the final transformation matrix , complete the precise matching between the registered image and the reference image.

[0036] In order to further illustrate the cross-view heterogeneous image matching method for airborne sensors provided by the present invention, it is described in conjunction with the accompanying drawings.

[0037] See also Figure 1 As shown in the figure, the process of the cross-view heterogeneous image matching method for airborne sensors is as follows: Step 1: Input the registration image to be matched and the reference image, and use the following projection transformation matrix to achieve the transformation:

[0038] in, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system, respectively. and are the position vectors from the view center to the reference image sensor coordinate system and the registration image coordinate system, respectively. is the internal parameter of the reference image sensor, To align the image sensor internal parameters, the sensor internal parameters can be obtained based on design or calibration.

[0039] The calculation of the projection transformation matrix in the above formula is divided into two parts. The first step is to project the reference image to the world coordinate system, and the second step is to project the image again to the registration image sensor coordinate system. When the reference image is a digital map, the first step transformation matrix can take the unit matrix.

[0040] Step 2: Extract edge features using the grayscale gradient information of the image pair. The operators that can be used include Canny, Sobel, etc. The edge extraction algorithm based on the image grayscale gradient information obtains edge features. The extracted edge features are represented in the form of a point set, denoted as and .

[0041] Step 3: Fast coarse alignment of images based on the Hausdorff distance between edge features. The objective function is the set and The Hausdorff distance between

[0042] The minimum position of the objective function is taken as the coarse registration position.

[0043] Step 4: Multi-scale feature pair extraction.

[0044] by For the minimum reduction ratio, For the proportional interval, reduce the registration image and the reference image to form a registration image sequence and the reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and The image is consistent with the original size.

[0045] First, in the scaled image and Window size Extract the HOPC template features, and then search for matching feature pairs with the rough alignment position obtained in step 3 as the center to form a feature pair set , based on this set, the RANSAC algorithm is run to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix, the registration position is corrected, and then based on the new registration position, the process is repeated with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

[0046] Step 5: Collect all feature pairs Integrate into a multi-scale feature pair set , based on the multi-scale feature set, the RANSAC algorithm is run to estimate the final transformation matrix , complete the matching between the registered image and the reference image.

[0047] See also Figure 2~3 As shown in FIG. 1 , the image captured by the near-infrared camera and the digital map are matched by the method provided by the present invention. Figure 2~3 It can be seen that the algorithm successfully matches the near-infrared black-and-white image used as the registration image with the color digital map used as the reference image. The details in the image are enlarged and displayed on the right in the form of a checkerboard image. The edge structures of the roads, bridges, greenhouses and other scenes in the image are accurately aligned. The image matching results achieve pixel-level accuracy, which well verifies the effectiveness of the method of the present invention.

[0048] The present invention provides a cross-view heterogeneous image matching system for airborne sensors, comprising: The image acquisition module is used to obtain the registration image and the reference image to be matched; the projection transformation method is used to correct the viewing angle difference between the registration image and the reference image, and then the grayscale gradient information is used to extract the edge features of the registration image and the reference image respectively; A coarse alignment module, for coarsely aligning the registration image with the reference image based on the Hausdorff distance between the edge features of the registration image and the reference image; The feature pair extraction module is used to compress the sizes of the registration image and the reference image to construct the corresponding multi-scale image sequence, iteratively extract the HOPC template features, and then match them based on the rough alignment position to obtain feature pairs, and organize them into a multi-scale feature pair set; The precise alignment module is used to estimate the transformation matrix between images through the RANSAC algorithm based on a set of multi-scale feature pairs, and finally complete the precise matching of images.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-view heterogeneous image matching method for airborne sensors, characterized in that: include: Acquire a registration image and a reference image to be matched; The projection transformation method is used to correct the viewing angle difference between the registration image and the reference image, and then the gray gradient information is used to extract the edge features of the registration image and the reference image respectively. The registered image and the reference image are roughly aligned based on the Hausdorff distance between the edge features of the registered image and the reference image; The sizes of the registration image and the reference image are compressed to construct the corresponding multi-scale image sequence, and the HOPC template features are extracted iteratively. Then, feature pairs are obtained by matching based on the rough alignment positions and organized into a multi-scale feature pair set. Based on a set of multi-scale feature pairs, the transformation matrix between images is estimated through the RANSAC algorithm, and finally the image precise matching is completed.

2. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: The projection transformation method is used to correct the perspective difference between the registration image and the reference image, including using the following projection transformation matrix to achieve the transformation: In the formula, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system respectively; and are the position vectors from the center of the scene to the reference image sensor coordinate system and the registration image coordinate system respectively; is the internal parameters of the reference image sensor; To align the image sensor internal parameters, the sensor internal parameters can be obtained based on design or calibration.

3. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: The Hausdorff distance between edge features is used to roughly align the registration image with the reference image: In the formula, and and are the edge feature point sets of the registration image and the reference image respectively.

4. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: Compress the sizes of the registration image and the reference image to construct the corresponding multi-scale image sequence, including: is the minimum compression ratio, The registration image and the reference image are compressed to construct a multi-scale image sequence, forming a registration image sequence. and the reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and The image is consistent with the original size.

5. The cross-view heterogeneous image matching method for airborne sensors according to claim 4, characterized in that: The process of acquiring a multi-scale feature pair set includes: First, in the scaled image and Window size Extract the HOPC template features, then search for matching feature pairs centered on the rough alignment position to form a feature pair set , based on this set, the RANSAC algorithm is run to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix, the registration position is corrected, and then based on the new registration position, the process is repeated with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

6. The cross-view heterogeneous image matching method for airborne sensors according to claim 5, characterized in that: Set all feature pairs Integrate into a multi-scale feature pair set , based on the multi-scale feature set, the RANSAC algorithm is run to estimate the final transformation matrix , complete the precise matching between the registered image and the reference image.

7. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: The edge features are obtained by using Canny or Sobel edge extraction algorithm based on image grayscale gradient information.

8. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: The registration image and the reference image to be matched both include airborne photoelectric sensor images, synthetic aperture radar sensor images or digital map images.

9. A cross-view heterogeneous image matching system for airborne sensors, characterized in that: include: The image acquisition module is used to obtain the registration image and the reference image to be matched; the projection transformation method is used to correct the viewing angle difference between the registration image and the reference image, and then the grayscale gradient information is used to extract the edge features of the registration image and the reference image respectively; A coarse alignment module, for coarsely aligning the registration image with the reference image based on the Hausdorff distance between the edge features of the registration image and the reference image; The feature pair extraction module is used to compress the sizes of the registration image and the reference image to construct the corresponding multi-scale image sequence, iteratively extract the HOPC template features, and then match them based on the rough alignment position to obtain feature pairs, and organize them into a multi-scale feature pair set; The precise alignment module is used to estimate the transformation matrix between images through the RANSAC algorithm based on a set of multi-scale feature pairs, and finally complete the precise matching of images.

Images