Method for matching different-source images based on EMD (Empirical Mode Decomposition) measurement

Abstract

The invention discloses a different-source image matching method based on EMD (empirical mode decomposition) measurement, and aims to solve the problems that feature description differences between different-source images are large, and a target corresponding relation is difficult to establish due to translation or rotation transformation of a reference image and an image to be registered. Firstly, preprocessing such as graying is carried out on a visible light image, the color dimension difference is reduced, and a reference image is obtained; then calculating a threshold value based on Tsallis entropy, and extracting significant foreground targets in the reference image and the infrared image; and on the basis, gray histograms of foreground targets in the infrared image and the reference image are extracted, the gray histograms are used as feature description, and the anti-rotation capability and the scale zooming capability in the matching process are improved. And finally, matching is completed by taking an EMD distance as similarity measurement. The EMD measurement-based different-source image matching method solves the problem that different-source images are difficult to match, improves the matching precision of targets in infrared and visible light images, can be used for a terminal guidance stage of an infrared guided weapon, and has a good application prospect.

Description

Technical Field

[0001] The invention belongs to the field of image processing and machine vision, and relates to a target matching method, in particular to a heterogeneous image matching method based on corner points. Background Art

[0002] Image matching is an important research topic in the military field and is widely used in reconnaissance, attack, navigation and other tasks. The image acquired by the image sensor in real time is matched with the pre-bound reference image to obtain the position of the target on the real-time image, and the relative position of the target and the carrier is further obtained by combining the relevant parameters of the sensor.

[0003] With the development of machine vision technology and the improvement of hardware computing power, image matching technology has developed rapidly, and a large number of algorithms with excellent performance have been born. Different types of sensors can complement each other's strengths. For example, infrared images have the characteristics of all-day, which can make up for the weakness of visible light images in dim light. Visible light images have rich details, clear target edges, and are all-weather. Therefore, heterogeneous image matching has a wider range of applications. Heterogeneous image matching methods can be roughly divided into grayscale-based template matching and feature-based matching algorithms. Due to the difference in field of view of the images to be matched and the nonlinear difference in pixel grayscale,

[0004] The template-based matching method mainly uses the grayscale value of the pixel as a statistical indicator, maps the template to a window of the reference image through transformation, and then compares the two images pixel by pixel to evaluate the similarity between the images to select a corresponding window as the matching result. Common algorithms include normalized cross-correlation, mutual information, phase features, etc. This type of method has strong noise resistance, but is easily affected by grayscale distortion and geometric distortion. Feature-based image matching can be described as extracting a unique feature set from the two images to be matched that does not change with external factors such as sensor type, shooting angle, and lighting conditions, and finding the optimal transformation in the transformation space to maximize the similarity of the two feature sets. Feature-based image matching has certain advantages over template matching in terms of calculation speed and adaptation to deformation, but it also has problems such as weak stability of feature points, uneven distribution, and poor matching quality. Summary of the invention

[0005] The problem to be solved by the present invention is: in view of the shortcomings of the prior art, an attempt is made to propose a heterogeneous image matching method based on EMD measurement, so as to improve the matching probability of the template matching algorithm and realize target recognition between heterogeneous images.

[0006] The technical solution of the present invention is: a heterogeneous image matching method based on EMD measurement, comprising:

[0007] Preprocess the infrared real-time image acquired by the infrared detector and the visible light reference image acquired based on satellite images;

[0008] A target to be matched is selected in the preprocessed visible light reference image, and features of the target to be matched are extracted and described based on Tsallis entropy to obtain a template region A and its feature vector α;

[0009] Taking each pixel point P in the infrared real-time image as the center and the template area A as the window, extract the feature vector β;

[0010] The similarity between the feature vector α and the feature vector β extracted based on each pixel point P is calculated; the pixel position corresponding to the highest similarity is taken as the final matching result, that is, the best matching position of the target in the visible light reference image in the infrared real-time image is obtained.

[0011] The preprocessing of the infrared real-time image acquired by the infrared detector and the visible light reference image acquired based on the satellite image includes:

[0012] Convert the downward-looking satellite image containing the target to be matched into a grayscale image as a visible light reference image;

[0013] Gamma correction and filtering are performed on the obtained visible light reference image and the infrared real-time image acquired by the infrared detector to reduce the characteristic differences of the images caused by illumination changes.

[0014] The gamma correction is accomplished by the following power law transformation:

[0015]

[0016] Among them, V out is the corrected pixel value; V in is the original pixel value; γ represents the correction parameter, γ>1 makes the image darker, and γ<1 makes the image brighter.

[0017] The correction parameter γ=2.2.

[0018] The filtering adopts median filtering, specifically:

[0019] Starting from the upper left corner of the image, all pixels are traversed, and a filtering window of size 3*3 is established with each pixel as the center. The pixel values ​​in the window are sorted by size, and the pixel value in the center of the window is replaced by the median value to complete the image filtering. During the traversal process, if the window exceeds the edge of the image, the excess part is filled with the adjacent image pixel values ​​to complete the calculation.

[0020] The method of selecting a target to be matched in the preprocessed visible light reference image and extracting and describing features of the target to be matched based on Tsallis entropy to obtain a template region A and its feature vector α includes:

[0021] The segmentation threshold of the salient target in the visible light reference image area is calculated based on Tsallis entropy;

[0022] A growth threshold is set based on the above segmentation threshold, and a region growing algorithm is used to extract the foreground target in the visible light reference image; a template region A is constructed according to the extreme value points of the foreground target boundary;

[0023] Calculate the normalized grayscale histogram in the template area A as the feature vector α.

[0024] The segmentation threshold of the significant foreground target in the visible light reference image region is obtained based on Tsallis entropy calculation, including:

[0025] The function of Tsallis entropy is defined as follows:

[0026]

[0027] Where: S q is Tsallis entropy; k is a normalization constant; p i is the probability distribution of the i-th gray level of the image; W is the number of gray levels of the image; q is the non-maximum parameter of entropy;

[0028] The segmentation threshold of the significant foreground object in the visible light reference image area is calculated as follows:

[0029] The probability distribution p of each gray level i in the statistical image i ;

[0030] Calculate the entropy value of the current image based on the function definition of Tsallis entropy;

[0031] The image grayscale mean is used as the initial segmentation threshold i0;

[0032] By iteratively finding q The maximum segmentation threshold i.

[0033] The method of extracting the foreground target in the visible light reference image by using a region growing algorithm and constructing a template region A according to the boundary extreme points of the foreground target includes:

[0034] Select one or more pixels in the ROI region of the visible light reference image as initial seed points;

[0035] Define the growth criteria: First, set the initial value of the grayscale benchmark based on the segmentation threshold i. When the growth starts, update the grayscale benchmark with the average grayscale of the current growing area. Subtract the grayscale benchmark from the pixel to be checked. If it is less than the growth threshold, it is judged as the same type of pixel.

[0036] Define the growth process: starting from the seed pixel, recursively check the neighboring pixels of the seed point one by one. If they meet the growth criteria, they are added to the growth area, and the neighboring pixels of the pixel are added to the list to be checked, and continue to judge;

[0037] Define the condition for stopping growth: when the difference between all pixels to be checked and the grayscale reference is greater than the growth threshold, stop growing, and the growth area obtained at this time is the foreground target;

[0038] The four extreme boundary points of the target are used to construct a rectangular frame as the template area A.

[0039] The grayscale reference initial value=0.8*i, and the growth threshold is 20.

[0040] The calculation of the normalized grayscale histogram in the template area A as a feature vector α includes:

[0041] The grayscale histogram of the reference image in the template area is obtained by statistics and normalized. The horizontal axis of this normalized grayscale histogram is the grayscale level, and the vertical axis is the distribution probability of this grayscale level in the image. Since the reference image is an 8-bit bitmap, that is, the image is divided into 256 grayscale levels, a 256-dimensional vector is defined, corresponding to the 256 horizontal axes of the histogram, and the elements of each dimension correspond to the vertical axis, and the feature vector α is obtained.

[0042] The similarity calculation is performed on the feature vector α and the feature vector β extracted based on each pixel point P; the pixel point position corresponding to the highest similarity is used as the final matching result, that is, the best matching position of the target in the visible light reference image in the infrared real-time image is obtained, including: using the EMD measurement algorithm to calculate the similarity of the feature vector α and the feature vector β extracted based on each pixel point P, wherein the smaller the EMD distance, the higher the similarity; the pixel point position corresponding to the highest similarity is used as the final matching result, that is, the best matching position of the target in the visible light reference image in the infrared real-time image is obtained, and the process is as follows:

[0043] (31) establishing a sliding window of the same size as the template, with the upper left corner of the window initially aligned with the upper left corner of the infrared real-time image;

[0044] (32) extracting the feature vector β in the window area, calculating the EMD distance between the current window feature vector β and the feature vector template α, and recording it in the coarse matching distance sequence;

[0045] (33) Thereafter, the sliding window is moved 2 pixels to the right each time, and the new EMD distance is calculated and recorded in the coarse matching distance sequence until the window reaches the edge of the image; then it is moved down 2 pixels and the sliding calculation process is repeated from the left until the window sliding range covers the entire image;

[0046] (34) sorting the rough matching distance sequence, and taking the window center coordinate (x', y') corresponding to the minimum distance value D' as the rough matching result;

[0047] (35) Using the 8 neighborhood coordinate points of the rough matching result as the sliding window center, repeat step (32) to obtain a precise matching distance sequence based on the 8 neighborhood coordinate points, add D' to the precise matching distance sequence and sort it, and find the window center coordinate (x, y) corresponding to the minimum value D as the best matching position of the target in the visible light reference image in the infrared real-time image.

[0048] The advantages of the present invention compared with the prior art are:

[0049] (1) The present invention automatically extracts foreground salient targets based on Tsallis entropy, avoiding the subjectivity of manual selection and making the algorithm more objective.

[0050] (2) The present invention converts the template matching between the real-time image and the reference image into the normalized grayscale histogram matching of the template area, and utilizes the rotation invariance, translation invariance and scaling invariance of the grayscale histogram to solve the problem of low matching accuracy of pixel-level template matching under scale and rotation changes.

[0051] (3) The present invention adopts EMD distance as the similarity metric. The EMD distance calculation process is relatively simple and can efficiently calculate the distances between a large number of data points. It has high reliability and strong robustness, is not affected by local changes in data points, can better reflect the overall distance relationship between data points, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flow chart for realizing the method of the present invention; DETAILED DESCRIPTION

[0053] like Figure 1 As shown, the specific implementation steps of the present invention are as follows:

[0054] Step 1: Preprocess the infrared real-time image and the visible light reference image

[0055] (11) Select the ROI area on the downward visible light map as the reference image, and convert the reference image into a grayscale image.

[0056] (12) Perform gamma correction on the infrared real-time image with a width and height of 720*576 and the visible light reference image, and then use a 3*3 filter to perform median filtering;

[0057] Gamma correction is a technique used to adjust the brightness of an image by performing nonlinear mapping of the pixel values ​​in the image to change the appearance of the image. The main purpose of gamma correction is to correct the nonlinear response of the display device so that the image looks more natural when viewed. At the same time, it adjusts the contrast of the image, improves the color performance of the image, and makes the details in the image more prominent. In digital image processing, gamma correction is accomplished through the following power law transformation:

[0058]

[0059] in:

[0060] V out is the corrected pixel value.

[0061] V in is the original pixel value.

[0062] γ is a correction parameter and usually takes a positive value.

[0063] The effect of gamma correction depends on the value of the correction parameter γ. Generally, γ>1 will make the image darker, and γ<1 will make the image brighter. Here, based on common experience, γ=2.2 is selected.

[0064] Median filtering is a nonlinear filtering method, which is often used to remove discrete impulse noise such as salt and pepper noise in images. Compared with mean filtering, it can better preserve edge details. Its basic idea is to replace the current pixel value with the median of the neighboring pixel values ​​to reduce the impact of noise. The main steps are:

[0065] (1) Select a filter window of odd size (here 3*3) with the center point of the window being the pixel currently being filtered.

[0066] (2) Slide the window from the upper left corner of the image across the entire image. For each window, sort the pixel values ​​in the window by size, and then replace the pixel value at the center of the window with the median value. When sliding to the edge of the image, fill in the calculation with the pixel values ​​of the edge row / column.

[0067] Step 2: Target extraction and feature description of the preprocessed infrared real-time image and visible light reference image

[0068] (21) Calculate the foreground target segmentation threshold of the benchmark image based on Tsallis entropy

[0069] Tsallis entropy is an entropy measure based on the non-maximum entropy principle, proposed by physicist Constantino Tsallis in 1988. It is a generalized form of Shannon entropy and is often used to describe systems with long-range interactions and non-equilibrium properties.

[0070] The Tsallis entropy is defined as follows:

[0071]

[0072] in:

[0073] S q is the Tsallis entropy.

[0074] k is a normalization constant.

[0075] p i is the probability distribution function of the system, which is represented here as the probability distribution of the i-th gray level of the image.

[0076] W is the number of states of the system, which is the number of gray levels of the image here.

[0077] q is a non-maximum parameter of entropy, usually a real number.

[0078] The segmentation threshold of the significant foreground object in the visible light reference image area is calculated as follows:

[0079] The probability distribution p of each gray level i in the statistical image i ;

[0080] Calculate the entropy value of the current image based on the function definition of Tsallis entropy;

[0081] The image grayscale mean is used as the initial segmentation threshold i0;

[0082] By iteratively finding q The maximum segmentation threshold i.

[0083] (22) Based on the above segmentation threshold, a growth threshold is set, and the foreground target in the reference image is extracted based on the region growing algorithm to construct a template

[0084] The region growing algorithm is a pixel-level grouping method for image segmentation. Its basic idea is to start with a group of seed pixels and gradually add pixels with similar properties to the same region through certain criteria to form a connected region.

[0085] A brief description of the region growing algorithm:

[0086] Select seed pixels: Select one or more from the image as initial seed pixels. These seed pixels are usually pixels with interesting characteristics, and are manually selected on the target.

[0087] Define the growth criterion: Determine the criterion for growing pixels. This is usually a grayscale or color similarity criterion. Subtract the pixel to be checked from the current grayscale reference. If it is less than the growth threshold, it is determined to be a similar pixel. Here, the segmentation threshold i*0.8 calculated in step (21) is used as the initial grayscale reference. When growth begins, the grayscale reference is updated with the average grayscale of the current growth area.

[0088] Define the growth process: starting from the seed pixel, recursively check the neighboring pixels of the seed point one by one. If they meet the growth criteria, they are added to the growth area, and the neighboring pixels of the pixel are added to the list to be checked, and continue to judge;

[0089] Define the condition for stopping growth: When the difference between all pixels to be checked and the grayscale reference is greater than the growth threshold, stop growing. The growth area obtained at this time is the foreground target. The four boundary extreme points of the target are used to construct a rectangular frame as the template area.

[0090] (23) Calculate the grayscale feature vector within the template area

[0091] The grayscale histogram of the reference image in the template area is obtained by statistics and normalized. The horizontal axis of this normalized grayscale histogram is the grayscale level, and the vertical axis is the distribution probability of this grayscale level in the image. Since the reference image is an 8-bit bitmap, that is, the image is divided into 256 grayscale levels, a 256-dimensional vector is defined, corresponding to the 256 horizontal axes of the histogram, and the elements of each dimension correspond to the vertical axis, and the feature vector template is obtained.

[0092] Step 3: Feature matching and result output based on EMD distance

[0093] (31)(31) Establish a sliding window with the same size as the template, and align the upper left corner of the window with the upper left corner of the infrared real-time image at the initial position;

[0094] (32) extracting the feature vector in the window area by the method (23), calculating the EMD distance between the feature vector of the current window and the feature vector template, and recording it in the coarse matching distance sequence;

[0095] EMD (Earth Mover's Distance), also known as Wasserstein distance, is a distance metric used to measure the similarity between two probability distributions. The name of EMD comes from its intuitive explanation: the minimum cost required to transform one distribution into another by "moving".

[0096] For two distributions P and Q, their EMD distance can be calculated as follows:

[0097]

[0098] in:

[0099] C i,j represents the distance cost from i to j.

[0100] γ i,j Represents the amount of material transported from i to j.

[0101] In addition, there are certain constraints:

[0102] γ i,j ≥0, where 1≤i≤M, 1≤j≤N (M, N are the dimensions of P, Q respectively). This constraint indicates that the transformation can only be made from distribution P to distribution Q, but not the reverse.

[0103] ∑γ i,j ≤W p ,∑γ i,j ≤W q , where W p , W q are the total distribution of P and Q respectively. This constraint indicates that the total transportation amount cannot exceed the total distribution of P and Q during the transportation process.

[0104] Specifically, if we consider two N-dimensional feature vectors as two mounds of N cells, the scalar in each dimension of the vector is the amount of earth in each cell of the mound, and the dimension difference is the transportation distance. Then the EMD is the minimum total work required to transform one mound into another. The work is defined as the amount of earth transferred per cell multiplied by its transportation distance.

[0105] The total workload is minimal, indicating that the feature vector P only needs a small change to be converted into the distribution of the feature vector Q, that is, the similarity between the two is the highest.

[0106] (33) Each time thereafter, the sliding window is moved 2 pixels to the right, the new EMD distance is calculated, and recorded in the coarse matching distance sequence until the window reaches the edge of the image. Then it is moved down 2 pixels and the above steps are repeated until the sliding window covers the entire image.

[0107] (34) Sort the coarse matching distance sequence, and take the window center coordinate (x', y') corresponding to the minimum distance value D' as the coarse matching result.

[0108] (35) Using the 8 neighborhood coordinate points of the rough matching result as the sliding window center, repeat step (32) to obtain a precise matching distance sequence based on the 8 neighborhood coordinate points, add D' to the precise matching distance sequence and sort it, and find the window center coordinate (x, y) corresponding to the minimum value D as the precise matching result. At this time, the best matching position of the reference image target in the infrared real-time image is (x, y).

Claims

1. A heterogeneous image matching method based on EMD measurement, characterized in that: include: Preprocess the infrared real-time image acquired by the infrared detector and the visible light reference image acquired based on satellite images; A target to be matched is selected in the preprocessed visible light reference image, and features of the target to be matched are extracted and described based on Tsallis entropy to obtain a template region A and its feature vector α; Taking each pixel point P in the infrared real-time image as the center and the template area A as the window, extract the feature vector β; The similarity between the feature vector α and the feature vector β extracted based on each pixel point P is calculated; the pixel position corresponding to the highest similarity is taken as the final matching result, that is, the best matching position of the target in the visible light reference image in the infrared real-time image is obtained.

2. The heterogeneous image matching method based on EMD metric according to claim 1, characterized in that: The preprocessing of the infrared real-time image acquired by the infrared detector and the visible light reference image acquired based on the satellite image includes: Convert the downward-looking satellite image containing the target to be matched into a grayscale image as a visible light reference image; Gamma correction and filtering are performed on the obtained visible light reference image and the infrared real-time image acquired by the infrared detector to reduce the characteristic differences of the images caused by illumination changes.

3. The heterogeneous image matching method based on EMD measurement according to claim 2 is characterized in that: The gamma correction is accomplished by the following power law transformation: Among them, V out is the corrected pixel value; V in is the original pixel value; γ represents the correction parameter, γ>1 makes the image darker, and γ<1 makes the image brighter.

4. The heterogeneous image matching method based on EMD measurement according to claim 1, characterized in that: The filtering adopts median filtering, specifically: Starting from the upper left corner of the image, all pixels are traversed, and a filtering window of size 3*3 is established with each pixel as the center. The pixel values ​​in the window are sorted by size, and the pixel value in the center of the window is replaced by the median value to complete the image filtering. During the traversal process, if the window exceeds the edge of the image, the excess part is filled with the adjacent image pixel values ​​to complete the calculation.

5. The heterogeneous image matching method based on EMD measurement according to claim 1, characterized in that: The method of selecting a target to be matched in the preprocessed visible light reference image and extracting and describing features of the target to be matched based on Tsallis entropy to obtain a template region A and its feature vector α includes: The segmentation threshold of the salient target in the visible light reference image area is calculated based on Tsallis entropy; A growth threshold is set based on the above segmentation threshold, and a region growing algorithm is used to extract the foreground target in the visible light reference image; a template region A is constructed according to the extreme value points of the foreground target boundary; Calculate the normalized grayscale histogram in the template area A as the feature vector α.

6. The heterogeneous image matching method based on EMD measurement according to claim 5, characterized in that: The segmentation threshold of the significant foreground target in the visible light reference image region is obtained based on Tsallis entropy calculation, including: The function of Tsallis entropy is defined as follows: Where: S q is Tsallis entropy; k is a normalization constant; p i is the probability distribution of the i-th gray level of the image; W is the number of gray levels of the image; q is the non-maximum parameter of entropy; The segmentation threshold of the significant foreground object in the visible light reference image area is calculated as follows: The probability distribution p of each gray level i in the statistical image i ; Calculate the entropy value of the current image based on the function definition of Tsallis entropy; The image grayscale mean is used as the initial segmentation threshold i0; By iteratively finding q The maximum segmentation threshold i.

7. The heterogeneous image matching method based on EMD measurement according to claim 6, characterized in that: The foreground target in the visible light reference image is extracted by using a region growing algorithm; The template area A is constructed according to the extreme points of the foreground target boundary, including: Select one or more pixels in the ROI region of the visible light reference image as initial seed points; Define the growth criteria: First, set the initial value of the grayscale benchmark based on the segmentation threshold i. When the growth starts, update the grayscale benchmark with the average grayscale of the current growth area. Subtract the grayscale benchmark from the pixel to be checked. If it is less than the growth threshold, it is considered to be the same type of pixel. Define the growth process: starting from the seed pixel, recursively check the neighboring pixels of the seed point one by one. If they meet the growth criteria, they are added to the growth area, and the neighboring pixels of the pixel are added to the list to be checked, and continue to judge; Define the condition for stopping growth: when the difference between all pixels to be checked and the grayscale reference is greater than the growth threshold, stop growing, and the growth area obtained at this time is the foreground target; The four extreme boundary points of the target are used to construct a rectangular frame as the template area A.

8. The heterogeneous image matching method based on EMD measurement according to claim 7, characterized in that: The grayscale reference initial value=0.8*i, and the growth threshold is 20.

9. The heterogeneous image matching method based on EMD measurement according to claim 7, characterized in that: The calculation of the normalized grayscale histogram in the template area A as a feature vector α includes: The grayscale histogram of the reference image in the template area is obtained by statistics and normalized. The horizontal axis of this normalized grayscale histogram is the grayscale level, and the vertical axis is the distribution probability of this grayscale level in the image. Since the reference image is an 8-bit bitmap, that is, the image is divided into 256 grayscale levels, a 256-dimensional vector is defined, corresponding to the 256 horizontal axes of the histogram, and the elements of each dimension correspond to the vertical axis, and the feature vector α is obtained.

10. The heterogeneous image matching method based on EMD measurement according to claim 9, characterized in that: The similarity calculation is performed on the feature vector α and the feature vector β extracted based on each pixel point P; the pixel point position corresponding to the highest similarity is used as the final matching result, that is, the best matching position of the target in the visible light reference image in the infrared real-time image is obtained, including: using the EMD measurement algorithm to calculate the similarity of the feature vector α and the feature vector β extracted based on each pixel point P, wherein the smaller the EMD distance, the higher the similarity; the pixel point position corresponding to the highest similarity is used as the final matching result, that is, the best matching position of the target in the visible light reference image in the infrared real-time image is obtained, and the process is as follows: (31) establishing a sliding window of the same size as the template, with the upper left corner of the window initially aligned with the upper left corner of the infrared real-time image; (32) extracting the feature vector β in the window area, calculating the EMD distance between the current window feature vector β and the feature vector template α, and recording it in the coarse matching distance sequence; (33) Thereafter, the sliding window is moved 2 pixels to the right each time, and the new EMD distance is calculated and recorded in the coarse matching distance sequence until the window reaches the edge of the image; then it is moved down 2 pixels and the sliding calculation process is repeated from the left until the window sliding range covers the entire image; (34) sorting the rough matching distance sequence, and taking the window center coordinate (x', y') corresponding to the minimum distance value D' as the rough matching result; (35) Using the 8 neighborhood coordinate points of the rough matching result as the sliding window center, repeat step (32) to obtain a precise matching distance sequence based on the 8 neighborhood coordinate points, add D' to the precise matching distance sequence and sort it, and find the window center coordinate (x, y) corresponding to the minimum value D as the best matching position of the target in the visible light reference image in the infrared real-time image.

Images