A cross-view heterogeneous image matching method for airborne sensors

Abstract

The present invention relates to the field of image processing technology, and more specifically to a cross-view heterogeneous image matching method for airborne sensors. The method comprises obtaining a registration image and a reference image to be matched; extracting edge features of the registration image and the reference image respectively; coarsely aligning the registration image and the reference image; matching based on the coarsely aligned positions to obtain feature pairs, and organizing them into a multi-scale feature pair set; and estimating the transformation matrix between the images using a RANSAC algorithm based on the multi-scale feature pair set, ultimately completing precise image matching. The present invention effectively achieves a balance between matching accuracy and computational speed by combining a mechanism of fast coarse registration based on image gradient edge features with precise multi-scale feature registration based on image phase information. Furthermore, the method exhibits good robustness for airborne heterogeneous images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a cross-viewing angle heterogeneous image matching method for airborne sensors. Background Art

[0002] With the rapid development of airborne sensor technology, in order to adapt to the complex battlefield situation and diversified combat missions in the future, the capabilities of airborne air-to-surface detection systems are expanding, and the functional compatibility and performance complementarity between payloads within the system are becoming more and more obvious. Multi-spectral optoelectronic detection and heterogeneous sensor collaboration have become key features of modern advanced airborne air-to-surface imaging detection equipment.

[0003] To maximize combat effectiveness in advanced airborne air-to-surface detection systems, heterogeneous sensors with different imaging spectrum bands and mechanisms are not simply stacked together, but rather highly coordinated at all levels, from mission planning to information processing. Taking the Rafael Litening pod's electro-optical-SAR sensor collaboration as an example, SAR sensors address the shortcomings of electro-optical sensors, which are significantly constrained by meteorological conditions and have difficulty performing high-precision positioning at long distances. Electro-optical sensors, on the other hand, address the SAR sensors' low upper resolution limit for close-range imaging and poor real-time imaging. For heterogeneous imaging and detection sensors, high-precision real-time registration technology for cross-view heterogeneous images ensures pixel-level alignment of images across sensors and high-precision fusion of detection information across systems, a prerequisite for efficient collaboration among multiple sensors within the system. Summary of the Invention

[0004] To address the aforementioned technical issues, the present invention provides a cross-view heterogeneous image matching method for airborne sensors. This method primarily performs cross-spectral heterogeneous image matching between airborne sensors or between airborne sensors and digital maps. This method uses a projective transformation to initially align the structural features of ground objects. Subsequently, through a combination of rapid coarse registration based on image gradient edge features and fine registration based on multi-scale features using image phase information, the method effectively achieves a balance between matching accuracy and computational speed. Furthermore, the method exhibits good robustness for airborne heterogeneous imagery.

[0005] The first object of the present invention is to provide a cross-view heterogeneous image matching method for airborne sensors, comprising:

[0006] Obtaining a registration image and a reference image to be matched;

[0007] The projection transformation method is used to correct the perspective difference between the registered image and the reference image, and then the grayscale gradient information is used to extract the edge features of the registered image and the reference image respectively;

[0008] The registered image and the reference image are roughly aligned based on the Hausdorff distance between the edge features of the registered image and the reference image;

[0009] The sizes of the registration image and the reference image are compressed to construct corresponding multi-scale image sequences, and the HOPC template features are extracted iteratively. Then, feature pairs are matched based on the coarse alignment positions and organized into a multi-scale feature pair set.

[0010] Based on the multi-scale feature pair set, the transformation matrix between images is estimated through the RANSAC algorithm, and finally the image precise matching is completed.

[0011] Preferably, a projective transformation method is used to correct the perspective difference between the registered image and the reference image, including using the following projective transformation matrix to implement the transformation:

[0012]

[0013] Where, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system respectively;

[0014] and are the position vectors from the center of the scene to the reference image sensor coordinate system and the registration image coordinate system respectively;

[0015] is the internal parameter of the reference image sensor;

[0016] To align the image sensor internal parameters, the sensor internal parameters can be obtained based on design or calibration.

[0017] Preferably, the Hausdorff distance between edge features is used to roughly align the registration image with the reference image:

[0018]

[0019] Where, and and are the edge feature point sets of the registration image and the reference image respectively.

[0020] Preferably, the sizes of the registration image and the reference image are compressed to construct corresponding multi-scale image sequences, including: is the minimum compression ratio, For the scale interval, compress the registration image and the reference image respectively to construct a multi-scale image sequence to form a registration image sequence and reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and Consistent with full-size image.

[0021] Preferably, the process of acquiring a multi-scale feature pair set includes:

[0022] First, in the scaled image and Window size Extract the HOPC template features, and then search for matching feature pairs centered on the rough alignment position to form a feature pair set , based on this set, run the RANSAC algorithm to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix to correct the registration position, and then based on the new registration position, repeat this process with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

[0023] Preferably, all feature pairs are set Integrate into a multi-scale feature pair set , run the RANSAC algorithm based on the multi-scale feature set to estimate the final transformation matrix , complete the precise matching between the registered image and the reference image.

[0024] Preferably, the edge features are obtained by using Canny or Sobel edge extraction algorithms based on image grayscale gradient information.

[0025] Preferably, the registration image to be matched and the reference image both include airborne photoelectric sensor images, synthetic aperture radar sensor images or digital map images.

[0026] A second object of the present invention is to provide a cross-view heterogeneous image matching system for airborne sensors, comprising:

[0027] The image acquisition module is used to obtain the registration image and the reference image to be matched; the projection transformation method is used to correct the perspective difference between the registration image and the reference image, and then the grayscale gradient information is used to extract the edge features of the registration image and the reference image respectively;

[0028] A coarse alignment module is used to coarsely align the registration image and the reference image based on the Hausdorff distance between the edge features of the registration image and the reference image;

[0029] The feature pair extraction module is used to compress the sizes of the registration image and the reference image to construct corresponding multi-scale image sequences, iteratively extract the HOPC template features, and then match them based on the coarse alignment positions to obtain feature pairs, and organize them into a multi-scale feature pair set;

[0030] The precise alignment module is used to estimate the transformation matrix between images through the RANSAC algorithm based on a set of multi-scale feature pairs, and ultimately complete the precise image matching.

[0031] The present invention has at least the following beneficial effects:

[0032] This paper provides a cross-view heterogeneous image matching method for airborne sensors. Taking an input image pair (a registration image and a reference image), the method first aligns the visual axis of the reference image with the registration image through a projective transformation. This projective transformation takes the pose information of both sensors and their internal parameters as input and calculates the projective transformation matrix between the images. The calculation of the projective transformation matrix consists of two steps: the first step projects the reference image into the world coordinate system; the second step reprojects the image into the sensor coordinate system of the registration image.

[0033] Although the difference in the visual axis between image pairs can be solved to a certain extent through projection transformation, the significant nonlinear radiometric difference between the image pairs must be accommodated through a matching algorithm. Existing high-precision matching methods often have high computational complexity, making it difficult to directly apply them in airborne environments and under high real-time requirements. The present invention designs a coarse-and-fine two-stage matching method. In the coarse registration, an algorithm with relatively low accuracy but high computational efficiency is applied to significantly reduce the search range of the registration position at a faster speed; in the fine registration, an algorithm with high accuracy but complex computation is applied to ensure its computational efficiency with a small search range.

[0034] In coarse registration, the present invention uses edge features based on image gradient information for image matching. Typical methods used include the Canny and Sobel algorithms, which are highly computationally efficient and can quickly extract image structural features. The coarse registration process uses the Hausdorff distance between the image and the edge feature point set as the optimization objective function, and the position with the minimum Hausdorff distance is taken as the coarse registration position.

[0035] In fine registration, the present invention uses HOPC features based on image phase information for image matching. This feature has good adaptability to nonlinear radiometric differences between heterogeneous images. The present invention designs a multi-scale feature pair set extraction mechanism, which compresses the image according to a series of equally spaced ratios to obtain a multi-scale image sequence. By iteratively running the process of HOPC feature and matching feature pair search, RANSAC registration position estimation and registration position update in the image sequence, feature pair sets of different scales can be obtained, and these sets are integrated to obtain a multi-scale feature pair set. Finally, the present invention runs the RANSAC algorithm based on the multi-scale feature pair set to obtain the transformation matrix between the registered image and the reference image, thereby completing the image matching process.

[0036] The method provided by the present invention can achieve a balance between the computational efficiency of the matching algorithm and the registration accuracy. At the same time, it can adapt to the problem that the viewing angle and scene scale of the airborne sensor image vary widely with the detection distance, and effectively improve the real-time, accuracy and robustness of image matching between airborne heterogeneous sensors and between airborne sensors and digital maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the cross-view heterogeneous image matching algorithm process for airborne sensors.

[0038] Figure 2 This is the cross-view heterogeneous image matching effect.

[0039] Figure 3 This is the cross-view heterogeneous image matching effect. DETAILED DESCRIPTION

[0040] In order to illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description with reference to the embodiments.

[0041] This invention primarily addresses the cross-view heterogeneous image matching task for airborne sensors, requiring the alignment of two images captured at different viewing angles, using different imaging mechanisms (such as imaging bands and detector structures), with different internal sensor parameters (such as focal length, sensitivity, and optical transmittance), and different environmental parameters (such as ambient illumination and atmospheric transmittance). Differences in viewing angle can lead to significant changes in the outline of objects within the images, while differences in imaging mechanisms, sensor parameters, and environmental parameters can cause significant nonlinear radiometric differences between the images. Furthermore, because airborne sensors need to observe ground targets at varying distances, the range of line of sight angles varies significantly during imaging, resulting in significant scale variations in typical ground features within the image frame, further increasing the requirements for adaptability and robustness of the registration algorithm. Designed specifically for airborne cross-view heterogeneous image matching, this invention uses a projective transformation to initially align the structural features of the ground scene. Subsequently, a mechanism combining fast coarse registration based on image gradient edge features with fine registration of multi-scale features based on image phase information effectively achieves a balance between matching accuracy and computational speed. Furthermore, the method exhibits good robustness for airborne heterogeneous images.

[0042] To achieve the above objectives, the present invention provides a cross-view heterogeneous image matching method for airborne sensors, comprising:

[0043] S1. Obtain the registration image and reference image to be matched;

[0044] The registration image and the reference image to be matched both include airborne photoelectric sensor images, synthetic aperture radar sensor images or digital map images.

[0045] S2, using the projection transformation method to correct the perspective difference between the registration image and the reference image, and then using the grayscale gradient information to extract the edge features of the registration image and the reference image respectively;

[0046] The projective transformation method is used to correct the perspective difference between the registered image and the reference image, including the use of the following projective transformation matrix to achieve the transformation:

[0047]

[0048] Where, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system respectively;

[0049] and are the position vectors from the center of the scene to the reference image sensor coordinate system and the registration image coordinate system respectively;

[0050] is the internal parameter of the reference image sensor;

[0051] To align the image sensor internal parameters, the sensor internal parameters can be obtained based on design or calibration.

[0052] Canny or Sobel is used to obtain edge features based on the edge extraction algorithm of image grayscale gradient information.

[0053] S3, coarsely aligning the registration image and the reference image based on the Hausdorff distance between the edge features of the registration image and the reference image;

[0054] The objective function for coarse alignment of the registered image and the reference image using the Hausdorff distance between edge features is:

[0055]

[0056] Where, and and are the edge feature point sets of the registration image and the reference image respectively.

[0057] S4. Compress the sizes of the registration image and the reference image to construct corresponding multi-scale image sequences, iteratively extract HOPC template features, and then perform matching based on the coarse alignment positions to obtain feature pairs, which are then organized into a multi-scale feature pair set.

[0058] Compress the sizes of the registration image and the reference image to construct the corresponding multi-scale image sequence, including: is the minimum compression ratio, For the scale interval, compress the registration image and the reference image respectively to construct a multi-scale image sequence to form a registration image sequence and reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and Consistent with full-size image.

[0059] The process of acquiring a multi-scale feature pair set includes:

[0060] First, in the scaled image and Window size Extract the HOPC template features, and then search for matching feature pairs centered on the rough alignment position to form a feature pair set , based on this set, run the RANSAC algorithm to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix to correct the registration position, and then based on the new registration position, repeat this process with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

[0061] S5. Based on the multi-scale feature pair set, the transformation matrix between images is estimated through the RANSAC algorithm, and finally the image precise matching is completed.

[0062] Set all feature pairs Integrate into a multi-scale feature pair set , run the RANSAC algorithm based on the multi-scale feature set to estimate the final transformation matrix , complete the precise matching between the registered image and the reference image.

[0063] In order to further illustrate the cross-view heterogeneous image matching method for airborne sensors provided by the present invention, it is described with reference to the accompanying drawings.

[0064] See also Figure 1 As shown in Figure 2, the process of the cross-view heterogeneous image matching method for airborne sensors is as follows:

[0065] Step 1: Input the registration image to be matched and the reference image, and use the following projection transformation matrix to achieve the transformation:

[0066]

[0067] in, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system, respectively. and are the position vectors from the center of the scene to the reference image sensor coordinate system and the registration image coordinate system, respectively. is the internal parameter of the reference image sensor, To align the image sensor internal parameters, the sensor internal parameters can be obtained based on design or calibration.

[0068] The calculation of the projection transformation matrix in the above equation is divided into two steps. The first step is to project the reference image into the world coordinate system, and the second step is to project the image back into the registration image sensor coordinate system. When the reference image is a digital map, the first step can be the identity matrix.

[0069] Step 2: Extract edge features using the grayscale gradient information of the image pair. The operators that can be used include Canny, Sobel, etc. The edge extraction algorithm based on the grayscale gradient information of the image obtains the edge features. The extracted edge features are represented in the form of a point set, which is recorded as and .

[0070] Step 3: Fast coarse alignment of images based on Hausdorff distance between edge features, the objective function is the set and The Hausdorff distance between:

[0071]

[0072] The minimum position of the objective function is taken as the coarse registration position.

[0073] Step 4: Multi-scale feature pair extraction.

[0074] by For the minimum reduction ratio, For the proportional interval, reduce the registration image and the reference image to form a registration image sequence and reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and Consistent with full-size image.

[0075] First, in the scaled image and Window size Extract the HOPC template features, and then search for matching feature pairs centered on the rough alignment position obtained in step 3 to form a feature pair set , based on this set, run the RANSAC algorithm to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix to correct the registration position, and then based on the new registration position, repeat this process with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

[0076] Step 5: Collect all feature pairs Integrate into a multi-scale feature pair set , run the RANSAC algorithm based on the multi-scale feature set to estimate the final transformation matrix , complete the matching between the registered image and the reference image.

[0077] See also Figures 2 and 3 As shown in FIG, the effect diagram of matching the image captured by the near infrared camera with the digital map by the method provided by the present invention. Figures 2 and 3 As can be seen in the figure, the algorithm successfully matches the near-infrared black-and-white image used as the registration image with the color digital map used as the reference image. The details in the image are magnified and displayed on the right in the form of a checkerboard image. The edge structures of scenes such as roads, bridges, and greenhouses in the image are precisely aligned, and the image matching results achieve pixel-level accuracy, which well verifies the effectiveness of the method of the present invention.

[0078] The present invention provides a cross-view heterogeneous image matching system for airborne sensors, comprising:

[0079] The image acquisition module is used to obtain the registration image and the reference image to be matched; the projection transformation method is used to correct the perspective difference between the registration image and the reference image, and then the grayscale gradient information is used to extract the edge features of the registration image and the reference image respectively;

[0080] A coarse alignment module is used to coarsely align the registration image and the reference image based on the Hausdorff distance between the edge features of the registration image and the reference image;

[0081] The feature pair extraction module is used to compress the sizes of the registration image and the reference image to construct corresponding multi-scale image sequences, iteratively extract the HOPC template features, and then match them based on the coarse alignment positions to obtain feature pairs, and organize them into a multi-scale feature pair set;

[0082] The precise alignment module is used to estimate the transformation matrix between images through the RANSAC algorithm based on a set of multi-scale feature pairs, and ultimately complete the precise image matching.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-view heterogeneous image matching method for airborne sensors, characterized in that: include: Obtaining a registration image and a reference image to be matched; The projection transformation method is used to correct the perspective difference between the registered image and the reference image, and then the grayscale gradient information is used to extract the edge features of the registered image and the reference image respectively; The registered image and the reference image are roughly aligned based on the Hausdorff distance between the edge features of the registered image and the reference image; The sizes of the registration image and the reference image are compressed to construct corresponding multi-scale image sequences, and the HOPC template features are extracted iteratively. Then, feature pairs are matched based on the coarse alignment positions and organized into a multi-scale feature pair set. Based on the multi-scale feature pair set, the RANSAC algorithm is used to estimate the transformation matrix between images, and finally complete the image precise matching; The projective transformation method is used to correct the perspective difference between the registered image and the reference image, including the use of the following projective transformation matrix to achieve the transformation: Where, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system respectively; and are the position vectors from the center of the scene to the reference image sensor coordinate system and the registration image coordinate system respectively; is the internal parameter of the reference image sensor; To align the internal parameters of the image sensor; Compress the sizes of the registration image and the reference image to construct the corresponding multi-scale image sequence, including: is the minimum compression ratio, For the scale interval, compress the registration image and the reference image respectively to construct a multi-scale image sequence to form a registration image sequence and reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and Consistent with the original size image pair; The process of acquiring a multi-scale feature pair set includes: First, in the scaled image and Window size Extract the HOPC template features, and then search for matching feature pairs centered on the rough alignment position to form a feature pair set , based on this set, run the RANSAC algorithm to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix to correct the registration position, and then based on the new registration position, repeat this process with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

2. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: The objective function for coarse alignment of the registered image and the reference image using the Hausdorff distance between edge features is: Where, and and are the edge feature point sets of the registration image and the reference image respectively.

3. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: Set all feature pairs Integrate into a multi-scale feature pair set , run the RANSAC algorithm based on the multi-scale feature set to estimate the final transformation matrix , complete the precise matching between the registered image and the reference image.

4. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: Canny or Sobel is used to obtain edge features based on the edge extraction algorithm of image grayscale gradient information.

5. The cross-view heterogeneous image matching method for airborne sensors according to claim 1, characterized in that: The registration image and the reference image to be matched both include airborne photoelectric sensor images, synthetic aperture radar sensor images or digital map images.

6. A cross-view heterogeneous image matching system for airborne sensors, characterized in that: include: The image acquisition module is used to obtain the registration image and the reference image to be matched; the projection transformation method is used to correct the perspective difference between the registration image and the reference image, and then the grayscale gradient information is used to extract the edge features of the registration image and the reference image respectively; A coarse alignment module is used to coarsely align the registration image and the reference image based on the Hausdorff distance between the edge features of the registration image and the reference image; The feature pair extraction module is used to compress the sizes of the registration image and the reference image to construct corresponding multi-scale image sequences, iteratively extract the HOPC template features, and then match them based on the coarse alignment positions to obtain feature pairs, and organize them into a multi-scale feature pair set; The precise alignment module is used to estimate the transformation matrix between images through the RANSAC algorithm based on a set of multi-scale feature pairs, and ultimately complete the image precise matching; The projective transformation method is used to correct the perspective difference between the registered image and the reference image, including the use of the following projective transformation matrix to achieve the transformation: Where, and are the rotation transformation matrices from the world coordinate system to the reference image sensor coordinate system and the registration image sensor coordinate system respectively; and are the position vectors from the center of the scene to the reference image sensor coordinate system and the registration image coordinate system respectively; is the internal parameter of the reference image sensor; To align the internal parameters of the image sensor; Compress the sizes of the registration image and the reference image to construct the corresponding multi-scale image sequence, including: is the minimum compression ratio, For the scale interval, compress the registration image and the reference image respectively to construct a multi-scale image sequence to form a registration image sequence and reference picture sequence ,in and The reduction ratio is , and The reduction ratio is , and Consistent with the original size image pair; The process of acquiring a multi-scale feature pair set includes: First, in the scaled image and Window size Extract the HOPC template features, and then search for matching feature pairs centered on the rough alignment position to form a feature pair set , based on this set, run the RANSAC algorithm to estimate the transformation matrix between the scaled image pairs , based on the transformation matrix to correct the registration position, and then based on the new registration position, repeat this process with the next set of image pairs in the multi-scale image sequence as the object until the feature extraction and matching of all image sequences are completed, and the feature pair set is obtained .

Images