A heterogeneous image matching algorithm based on hog features and edge features

Abstract

The application belongs to the field of multi-source data fusion of avionics system, and particularly relates to an airborne heterogeneous image matching algorithm based on HOG features and edge features. The application comprises the following steps: pre-processing an original SAR image to obtain a first SAR image; performing filtering transformation on the first SAR image to obtain a second SAR image; unifying the resolution of the second SAR image to that of a satellite image to obtain a third SAR image; combining HOG features and edge features to extract features from the third SAR image to obtain a SAR feature image; converting the original satellite image into a Mercator projection first satellite image; pre-processing the first satellite image to obtain a second satellite image; combining HOG features and edge features to extract features from the second satellite image to obtain a satellite feature image; performing feature matching on the SAR feature image and the satellite feature image by using a template matching algorithm and verifying the matching; and if the verification is successful, correcting the longitude and latitude information of a specific point in the original SAR image according to the longitude and latitude information of the high-precision satellite image to improve the target positioning accuracy.

Description

Technical Field

[0001] This invention belongs to the field of multi-source data fusion in avionics systems, specifically involving a heterogeneous image matching algorithm based on HOG features and edge features. Background Technology

[0002] As battlefield environments become increasingly complex and detection conditions more severe, the requirements for target positioning accuracy from various devices and weapons are rising. Currently, the detection performance of a single sensor is insufficient to meet the overall needs of avionics systems. Therefore, fusing target information from multiple data sources to improve accuracy has become a crucial issue for avionics systems. SAR (Special Radiation Array) has advantages such as long range, strong penetration, and all-weather capability, and is widely used in the military field. By fusing SAR images with high-precision satellite images, environmental information can be extracted to the maximum extent, enabling the identification and precise positioning of high-value targets on the ground and at sea.

[0003] The imaging mechanisms of heterogeneous images differ significantly, resulting in substantial variations in the grayscale values ​​of the same pixel, making it difficult to extract corresponding features. Furthermore, SAR images are greatly affected by factors such as imaging pose, occlusion, and imaging parameters, exhibiting severe multiplicative speckle noise. This poses a challenge to heterogeneous image matching between SAR and satellite images. Common heterogeneous image matching algorithms include: feature point matching algorithms, edge support matching algorithms, HOG feature matching algorithms, Delaunay triangulation constraint matching algorithms, curve feature matching algorithms, and deep learning-based matching algorithms.

[0004] The drawbacks of feature point matching algorithms are that the extraction, description, and matching of feature points in SAR images are all affected by speckle noise, and it is difficult to extract corresponding feature points from heterogeneous images. The drawback of edge support matching algorithms is that image edge detection algorithms use fixed detection thresholds to process images under different conditions, leading to unstable edge detection results. HOG features are generally used for matching images from the same source. For heterogeneous images, especially SAR and visible light images, the different imaging mechanisms result in significant differences in pixel histograms within the same region, thus leading to variations in HOG features. The drawback of Delaunay triangulation matching algorithms is the difficulty in selecting the skeleton extraction threshold; an inappropriate threshold selection directly leads to matching failure. The drawback of curve feature matching algorithms is that they are not suitable for matching closed curves and have high computational complexity. The drawback of deep learning-based matching algorithms is the limited number of SAR images, insufficient training samples, and the difficulty in defining the boundaries of deep learning algorithms, resulting in poor testability and unsuitability for high-precision airborne tasks. Therefore, common heterogeneous image matching algorithms each have their shortcomings, lack robustness, and cannot be directly used in engineering practice. No engineering application cases of heterogeneous image matching have been found in academic papers. Summary of the Invention

[0005] The purpose of this invention is to provide a heterogeneous image matching algorithm based on HOG features and edge features, which solves the problems of poor robustness and large influence of algorithm threshold in existing heterogeneous image matching algorithms.

[0006] The technical solution of this invention:

[0007] A heterogeneous image matching algorithm based on HOG features and edge features, comprising:

[0008] Based on the platform's attitude and position during SAR image imaging, the original SAR image is converted into a top-down, due north first SAR image;

[0009] The second SAR image is obtained by sequentially performing Gaussian filtering, median filtering, and gamma transform on the first SAR image.

[0010] The resolution of the second SAR image is unified to be consistent with the resolution of the satellite image to obtain the third SAR image;

[0011] The third SAR image is feature extracted by combining HOG features and edge features to obtain the SAR feature image;

[0012] The original satellite image is converted into a first satellite image, where the first satellite image is a Mercator projection;

[0013] The second satellite image is obtained by sequentially applying Gaussian filtering and median filtering to the first satellite image;

[0014] The second satellite image was used to extract features by combining HOG features and edge features to obtain a satellite feature image;

[0015] The template matching algorithm is used to match features between SAR feature images and satellite feature images to obtain the relationship between the original SAR image and the original satellite image;

[0016] By utilizing the relationship between the original SAR image and the original satellite image, the latitude and longitude information of specific points in the original SAR image is corrected.

[0017] Furthermore, a combination of HOG features and edge features is used to extract features from the third SAR image, resulting in a SAR feature image, specifically including:

[0018] HOG features are extracted from the third SAR image to obtain the SAR image HOG feature map;

[0019] The edge features of the third SAR image are extracted to obtain the SAR image edge feature map;

[0020] The SAR feature image is obtained by superimposing the HOG feature map and the edge feature map of the SAR image at a certain ratio.

[0021] Furthermore, the second satellite image is used to extract features by combining HOG features and edge features to obtain a satellite feature image, specifically including:

[0022] HOG features of the second satellite image are extracted to obtain the HOG feature map of the satellite image;

[0023] The edge features of the second satellite image are extracted to obtain the satellite image edge feature map;

[0024] The satellite feature image is obtained by superimposing the HOG feature map and the edge feature map of the satellite image at a certain ratio.

[0025] Furthermore, a template matching algorithm is used to perform feature matching between the SAR feature image and the satellite feature image to obtain the relationship between the original SAR image and the original satellite image. This process also includes:

[0026] The relationship between the original SAR image and the original satellite image was verified.

[0027] Furthermore, the relationship between the original SAR image and the original satellite image was verified, specifically including:

[0028] The region on the original satellite image that matches the original SAR image is selected as the ROI region;

[0029] Line features are extracted within the ROI region of the original satellite image, and then stitched and filtered to obtain a set of candidate verification lines;

[0030] For each line in the set of lines, select N equal division points and record the coordinates of the points, where N is a natural number greater than or equal to 10;

[0031] By utilizing the relationship between the original SAR image and the original satellite image, the recorded coordinate points are converted into corresponding coordinate points on the original SAR image;

[0032] Count the number of corresponding coordinate points on the original SAR image that lie on the feature edges in the SAR image edge feature map;

[0033] If the number is greater than a predetermined threshold, then the line containing the current coordinate point is considered to be aligned.

[0034] If more than three straight lines can be aligned and these aligned lines have different slopes, then the original SAR image is considered to be a correct match with the original satellite image; otherwise, the match is considered to have failed.

[0035] Furthermore, the predetermined threshold is 3.

[0036] Furthermore, the method also includes:

[0037] If a match fails, readjust the parameters in the feature extraction algorithm.

[0038] Furthermore, N is 20.

[0039] The beneficial effects of this invention are:

[0040] The heterogeneous image matching algorithm proposed in this paper combines edge features and HOG block gradient features of images, making the features more comprehensive and robust. The algorithm uses a combination of two different features, is unaffected by threshold parameters, and can meet the matching requirements of airborne images in different scenarios. Furthermore, the algorithm has a matching verification module, which enhances its fault tolerance and significantly improves the matching accuracy. Through this heterogeneous image matching algorithm, airborne SAR images are fused with the latitude and longitude information of high-precision satellite images, maximizing the extraction of environmental information and helping aircraft to complete the task of accurately locating high-value targets on the ground and sea. Attached Figure Description

[0041] Figure 1 The ratio of HOG feature map to edge feature map is 1:1;

[0042] Figure 2 This is a schematic diagram of the heterogeneous image matching algorithm based on HOG features and edge features;

[0043] Figure 3 It is a matching and verification process;

[0044] Figure 4 It is a feature matching algorithm that obtains the relationship between the original SAR image and the original satellite image. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0046] like Figure 2The heterogeneous image matching of this invention consists of four modules: image preprocessing, feature extraction, feature matching, and linear feature verification. The image preprocessing module includes filtering, denoising, and color correction of SAR and satellite images to enhance image detail. The feature extraction module extracts HOG and Canny edge features from the SAR and satellite images respectively, and then superimposes them at a certain ratio. The feature matching module calculates the matching support of SAR image features on the satellite image feature map and obtains the translation parameters of the SAR image features relative to the satellite image features through global optimization. Finally, linear features are used to verify the correctness of the matching. The main process of this invention is as follows:

[0047] (1) Preparation before matching: Based on the attitude and position of the platform when the SAR image was captured, the original SAR image is converted to a top-down, true-north view. The satellite image is converted to Mercator projection. The resolution of both images must be known.

[0048] (2) Perform the following steps on the SAR image in sequence: Gaussian filtering, median filtering, gamma transformation, and resolution unification (i.e., unify the resolution to be consistent with the satellite image).

[0049] (3) Perform Gaussian filtering and median filtering on the satellite images in sequence.

[0050] (4) Feature extraction was performed on both SAR and satellite images, using a combination of HOG features and edge features. HOG features were overlaid onto the edge feature map at a certain ratio, resulting in a new feature map as follows: Figure 1 As shown, matching can be achieved with a ratio between 0:1 and 1:0 for HOG feature maps and edge feature maps (this invention recommends using a 1:1 ratio).

[0051] (5) A feature support cost function is constructed using a template matching algorithm, with the translation parameters of SAR image features relative to satellite image features as independent variables. The cost function is solved to obtain the matching translation parameters. See details below. Figure 4 .

[0052] The cost function matrix is ​​obtained using the normalized correlation coefficient method.

[0053]

[0054] Where T is the feature map of the SAR image, (xi,yi) are the coordinates of the point on the SAR image, M is the feature map of the satellite image, and (x,y) are the coordinates of the top-left corner of the SAR image on the satellite image. See details. Figure 4 .

[0055] R(x,y) is a cost function matrix. Taking the coordinates (Δx,Δy) of the maximum value in the matrix, the transformation relationship between the coordinates (x',y') on the satellite image and the coordinates (x,y) on the SAR image is x'=x+Δx

[0056] y' = y + Δy:

[0057] (6) A matching verification module is used to determine whether the template matching is correct. This invention uses straight line feature verification. Theoretically, as long as two straight lines with different slopes in the satellite image region coincide with the straight lines in the corresponding region of the SAR image, the image matching can be considered correct.

[0058] The specific process is as follows: Based on the matching results, the region on the satellite image that matches the SAR image is selected as the ROI region; straight line features are extracted within the satellite ROI region and then stitched and filtered to obtain a candidate verification line set; in the verification line set, each line is divided into twenty equal parts, and the coordinates are recorded; on the edge map of the SAR image, it is checked which of the 20 points are on the edge. If more than 3 points are on the edge (considering the poor edge continuity of the SAR image), the line is considered to be aligned on the satellite image and the SAR image; if more than 3 lines are aligned and the slopes of these aligned lines are different, the SAR image and the satellite image are considered to be matched correctly; otherwise, the matching is considered to have failed.

[0059] If the verification module determines that the match has failed, it may be due to two reasons: one is that the parameter settings are incorrect and the parameters should be readjusted to continue the match; the other is that the matching area does not have clear edges (such as large areas of ocean and grassland).

[0060] Example:

[0061] Image preprocessing

[0062] Reading, saving, and displaying SAR and satellite images. This involves directly calling OpenCV's `imread()`, `imwrite()`, and `imshow()` functions.

[0063] Median filtering is used for denoising SAR images. It calls the OpenCV's medianBlur function with the parameter value set to 7.

[0064] Gaussian filtering is used for denoising SAR images, calling the GaussianBlur function in OpenCV. The parameters are set to 7 and 3.

[0065] Gamma transform, used to improve the contrast of SAR images, is implemented by: traversing the rows and columns of the image, reading the value of each pixel, and performing the transformation s = cr. γ The transformation. Based on the results of multiple experiments, c = 1, γ = 0.3 were selected.

[0066] The SAR image is scaled to match the spatial resolution of the satellite image. This is achieved using non-linear interpolation by calling OpenCV's `resize()` function.

[0067] 2. Feature Extraction

[0068] Extract the HOG features from the two images and plot the histogram information of the HOG features in a graphical form.

[0069] Extract the Canny image features from the two images by calling the Canny function in OpenCV with parameters set to 30 and 250, which correspond to the minimum and maximum thresholds for edge extraction, respectively.

[0070] The edge map and HOG feature map are overlaid. The `addWeighted` function of `opencv` is called. This invention recommends setting the overlay ratio to 1:1.

[0071] 3. Template matching

[0072] Considering the maturity of the algorithm, the support calculation module uses the normalized correlation coefficient algorithm, and the cost function solution module uses the exhaustive search algorithm. Assuming the SAR image dimensions are sar.rows, sar.cols; and the satellite image dimensions are color.rows, color.cols, the specific template matching process is as follows:

[0073] First, generate a matrix result of size (color.rows-sar.rows+1, color.cols-sar.cols+1) to store the value of each edge support during template exhaustive search.

[0074] The template SAR image is slid across the satellite image from top to bottom and left to right, traversing every pixel of the satellite image. The edge support between the SAR image and the satellite image is calculated using a normalized correlation coefficient algorithm at each traversal, and the result is written into the matrix `result`. This process can be performed using the `matchTemplate` function in OpenCV.

[0075] Normalize the values ​​in the matrix result.

[0076] Traverse the matrix result to find the coordinates corresponding to the minimum value, which is the vertex on the satellite image that best matches the edge of the SAR image.

[0077] 4. Matching Validation

[0078] The matching verification module includes three sub-steps: line filtering, corresponding point statistics, and matching judgment.

[0079] The edline algorithm was used to extract straight line features from SAR images and satellite images, respectively.

[0080] The line filtering module mainly uses line length as the filtering condition. It iterates through the lines extracted by Edline in the previous step, and considers a line to be qualified when its length is greater than 1 / 40 of the height of the original SAR image.

[0081] The corresponding point statistics and matching judgment are described by a function. The input of the function is the filtered set of lines and the edge map of the SAR image, and the output of the function is whether a match is found. The specific process is to traverse all lines and extract twenty equal division points for each line. The coordinates of each point are recorded; the corresponding pixel value is read from the SAR image. If the value is greater than 180, the point is considered to be on the edge of the SAR image. For a straight line, if more than three points are on the edge of the SAR image, the line is considered to coincide with the SAR image. If more than three lines coincide with the SAR image and their slopes are all different, the match is considered successful. The specific process is as follows: Figure 3 As shown.

[0082] This paper aims to propose a heterogeneous image matching algorithm applicable to airborne engineering. The algorithm is robust and efficient, minimizes the influence of threshold parameters in image processing, and can meet the matching requirements of different scenarios, thereby improving the target detection accuracy.

[0083] This invention uses a combination of HOG features and edge features to perform support matching calculations and then verifies them using linear features, thereby achieving more accurate and robust heterogeneous image matching.

[0084] The method presented in this paper differs from other methods that combine HOG and edge features in that it plots HOG features as a feature map and overlays them onto the edge feature map. If only edge features are extracted, most of the image's gradient features are lost. However, by adding HOG features, more refined gradient features can be combined with each pixel block in the image, resulting in a more comprehensive overall image feature set. Overlaying the HOG feature map onto the edge features significantly reduces the image's dependence on edges. Even if the edge extraction algorithm fails to extract edges (the worst-case scenario), this method still achieves good matching results and improves matching accuracy. Furthermore, a linear feature verification module is added after image feature matching. The overlap of linear features in the matched regions is used to determine whether a correct match has been made, improving the reliability of the matching algorithm.

[0085] The above description is merely a specific embodiment of the present invention, providing a detailed description of the invention. Parts not covered herein are conventional techniques. However, the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A heterogeneous image matching algorithm based on HOG features and edge features, characterized in that, include: Based on the platform's attitude and position during SAR image imaging, the original SAR image is converted into a top-down, due north first SAR image; The second SAR image is obtained by sequentially performing Gaussian filtering, median filtering, and gamma transform on the first SAR image. The resolution of the second SAR image is unified to be consistent with the resolution of the satellite image to obtain the third SAR image; The third SAR image is extracted by combining HOG features and edge features to obtain a SAR feature image. Specifically, this includes: extracting HOG features from the third SAR image to obtain a SAR image HOG feature map; extracting edge features from the third SAR image to obtain a SAR image edge feature map; and superimposing the SAR image HOG feature map and the SAR image edge feature map at a certain ratio to obtain a SAR feature image. The original satellite image is converted into a first satellite image, where the first satellite image is a Mercator projection; The second satellite image is obtained by sequentially applying Gaussian filtering and median filtering to the first satellite image; The second satellite image is used to extract features by combining HOG features and edge features to obtain a satellite feature image. Specifically, this includes: extracting HOG features from the second satellite image to obtain a satellite image HOG feature map; extracting edge features from the second satellite image to obtain a satellite image edge feature map; and superimposing the satellite image HOG feature map and the satellite image edge feature map at a certain ratio to obtain a satellite feature image. The template matching algorithm is used to match features between SAR feature images and satellite feature images to obtain the relationship between the original SAR image and the original satellite image; By utilizing the relationship between the original SAR image and the original satellite image, the latitude and longitude information of specific points in the original SAR image is corrected.

2. The heterogeneous image matching algorithm based on HOG features and edge features according to claim 1, characterized in that, Template matching algorithms are used to perform feature matching between SAR feature images and satellite feature images to obtain the relationship between the original SAR image and the original satellite image. This then includes: The relationship between the original SAR image and the original satellite image was verified.

3. The heterogeneous image matching algorithm based on HOG features and edge features according to claim 1, characterized in that, The relationship between the original SAR image and the original satellite image was verified, specifically including: The region on the original satellite image that matches the original SAR image is selected as the ROI region; Line features are extracted within the ROI region of the original satellite image, and then stitched and filtered to obtain a set of candidate verification lines; For each line in the set of lines, select N equal division points and record the coordinates of the points, where N is a natural number greater than or equal to 10; By utilizing the relationship between the original SAR image and the original satellite image, the recorded coordinate points are converted into corresponding coordinate points on the original SAR image; Count the number of corresponding coordinate points on the original SAR image that lie on the feature edges in the SAR image edge feature map; If the number is greater than a predetermined threshold, then the line containing the current coordinate point is considered to be aligned. If more than three straight lines can be aligned and these aligned lines have different slopes, then the original SAR image is considered to be a correct match with the original satellite image; otherwise, the match is considered to have failed.

4. The heterogeneous image matching algorithm based on HOG features and edge features according to claim 3, characterized in that, The predetermined threshold is 3.

5. The heterogeneous image matching algorithm based on HOG features and edge features according to claim 1, characterized in that, Also includes: If a match fails, readjust the parameters in the feature extraction algorithm.

6. The heterogeneous image matching algorithm based on HOG features and edge features according to claim 1, characterized in that, N is 20.

Images