Method for eliminating by double thresholds after fitting surface of matching point position difference

Abstract

A method for eliminating double thresholds by fitting a surface of the position difference of matching points belongs to the technical field of aviation image processing. The purpose of the present invention is a method for eliminating double thresholds by fitting a surface of the position difference of matching points, which uses the fitted surface to eliminate points from each matching point. The present invention uses the upper limit threshold surface obtained from the position distance difference of the matching points obtained by superimposing the matching points of the multi-spectral image to be registered on the reference image. All matching points above the upper limit threshold surface are mis-matching points. The smoothed surface is translated downward to obtain the lower limit threshold surface. The upper limit threshold surface and the lower limit threshold surface form a three-dimensional threshold space. The upper limit threshold surface is used to eliminate mis-matching points in flat terrain areas, mis-matching points in hilly terrain areas, and mis-matching points on terrain slopes. The lower limit threshold surface is used to eliminate matching points. The present invention eliminates a large number of image matching points in hilly terrain areas, and the registered images can meet the requirements of post-processing and application of the images.

Description

Technical Field

[0001] The present invention belongs to the technical field of aviation image processing. Background Art

[0002] Currently, the commonly used matching point screening method adopts a strategy from coarse to fine. In the coarse matching stage, mainly based on the RANSAC algorithm, the mismatched point pairs in the feature point pairs are removed to obtain the coarse matching point pairs; in the fine matching stage, mainly the least squares method is used for the adjustment of the matching points to obtain the fine matching point pairs (Yang et al., 2019).

[0003] For example: The steps of adjusting the matching points by the least squares method are: (1) calculating the homography transformation matrix; (2) adjusting the matching points; by removing the point pairs with a root mean square greater than the threshold, the correct fine matching point pairs are obtained. For a multi-spectral camera with the imaging centers of each band image being the same point (i.e., the images of each band are imaged simultaneously), the geometric positions of its image matching points are roughly the same, and the corresponding image points basically satisfy a single matrix transformation relationship, and the effect of removing mismatched points or low-precision matching points is better. However, for the filter array multi-spectral image, due to the large pixel displacement between its band images, the positions of the matching points are different, the Euclidean distance is large in the terrain undulation area, while the coordinates of the matching points in the flat area are roughly the same. Therefore, the filter array multi-spectral image with pixel displacement cannot accurately describe the position coordinate conversion relationship between the matching points using a single matrix.

[0004] In addition, some scholars have proposed: directly superimposing the matching points in the to-be-registered multi-spectral image onto the reference image, and using an arithmetic formula to calculate the position distance difference between the matching points of the to-be-registered image and the reference image, and then adopting a global threshold to eliminate the matching points with a large position distance error (Liang et al., 2015). This method solves the inherent defects of screening matching points using the homography transformation matrix and the matching point adjustment method to a certain extent, and effectively retains the matching points in the flat area, but it is easy to eliminate all the matching points in the area with large terrain undulation, making it impossible to perform image registration in the area with large terrain undulation.

[0005] Some scholars have proposed to remove the matching points with a large difference from the surrounding ones through the geometric positions of the matching points (Li et al., 2009). This method well solves the shortcomings of screening matching points using the global threshold and can effectively retain the matching points in the high terrain area and the flat area on the image, but the problem of screening the matching points at the terrain slope is still not well solved. For example Figure 10 in, point A and point D are in the terrain undulation area, point B is in the flat area, and there are obvious differences in the position differences between points A, B, and D and the surrounding matching points, and they should belong to the mismatched points. According to the method of removing the matching points with a large difference from the surrounding ones through the geometric positions of the matching points, the three mismatched points A, B, and D can be effectively eliminated. Figure 10Points C and E are located on the terrain slope. Judging from the change trend of the position difference between the matching points, C and E should also be mis-matched points. However, it is very difficult to eliminate them by using the method of removing the matching points with large differences around based on the geometric positions of the matching points. Summary of the Invention

[0006] The object of the present invention is to propose a surface fitting double-threshold matching point screening algorithm. Different regions in the image can fit different surfaces according to the distance differences of the matching points, and a method for removing points from the matching points using the surface after fitting, that is, a surface fitting double-threshold removal method based on the position difference of the matching points.

[0007] The steps of the present invention are as follows:

[0008] S1. Using the position distance differences of the matching points obtained by superimposing the matching points of the multi-spectral image to be registered on the reference image, fitting and interpolating to obtain a distance difference surface, and performing smoothing processing on the fitted surface. Translating the smoothed surface upward to obtain an upper limit threshold surface. All matching points higher than the upper limit threshold surface are mis-matched points. Translating the smoothed surface downward to obtain a lower limit threshold surface. The upper limit threshold surface and the lower limit threshold surface constitute a three-dimensional threshold space. Using the upper limit threshold surface to eliminate the mis-matched points B in the flat terrain area, the mis-matched points A in the hilly terrain area, and the mis-matched points C on the terrain slope. Using the lower limit threshold surface to eliminate the matching points D and point E;

[0009] S2. The surface fitting double-threshold matching includes four aspects: surface interpolation fitting, surface smoothing, threshold space determination, and threshold space judgment:

[0010] a. Surface interpolation fitting

[0011] Construct a triangular mesh using the position of the matching points on the reference image, and then fill it using the inverse distance weighted interpolation method within each triangle. Points A, B, and C are the vertices of a certain triangular mesh among the matching points on the reference image, and points A′, B′, and C′ are the matching points of the image to be registered. By calculating the position distance differences between A and A′, B and B′, and C and C′ respectively, m 1 、m 2 、m 3 are obtained. The distances of a certain point in the triangle from vertices A, B, and C are d 1 、d 2 、d 3 respectively. Then the value of the distance difference surface at this point is:

[0012]

[0013] A distance difference surface generated by using inverse distance weighted interpolation;

[0014] b. Surface smoothing

[0015] Select a 3×3 mean filter for smoothing, convert the perspective below the conversion surface, and obtain D and E mismatched points;

[0016] c. Threshold space determination

[0017] The smoothed distance difference surface plus ±0.5 constitutes a three-dimensional threshold space. That is, in a flat area, adding -0.5 makes the lower threshold surface lower than 0, but it does not affect the subsequent threshold judgment;

[0018] d. Threshold space judgment

[0019] The distance difference d of the matching points i , the blue dot is the upper threshold U i , the cyan dot is the lower threshold D i , calculate the upper threshold U of each matching point by interpolation according to the upper threshold surface i , calculate the lower threshold D of each matching point by interpolation according to the lower threshold surface i , and then use Equation (8) to judge whether the distance difference Y i is a correct matching point

[0020]

[0021] S3. Gray resampling

[0022] For a pair of matching triangles A 1 B 1 C 1 and A 2 B 2 C 2 , first, according to the vertex coordinates (x 1 B 1 C 1 ) of triangle A in the reference image 11 ,y 11 ), (x 12 ,y 12 ), (x 13 ,y 13 ), and the vertex coordinates (x 2 B 2 C 2 ) of triangle A in the image to be registered 21 ,y 21 ), (x 22 ,y 22 ), (x 23 ,y 23 ), use Equation (3) to obtain the geometric position transformation matrix H between the reference image and the image to be registered,

[0023]

[0024] Then, for a point P within the triangle of the reference image 1 (x 1 , y 1 ), using Equation (4), its corresponding position P 2 (x 2 , y 2 )

[0025]

[0026] Finally, in triangle A 2 B 2 C 2 of the image to be registered, the gray value of point P 2 is obtained using the inverse distance weighted interpolation method and assigned to point P 1 . If the interpolation is performed for each point in each triangle in sequence, the gray value interpolation work is completed.

[0027] The present invention eliminates a large number of image matching points in hilly areas. The eliminated points cannot meet the registration accuracy requirements. When screening the matching points, both the surrounding point pairs are considered, and the points that do not conform to the pixel displacement change in the surrounding area of the point pairs are eliminated, effectively solving the problem of accurately screening matching points based on a local area. The present invention reduces the influence of mis-matching points and low-precision matching points in the filter array multi-spectral image on the image registration result. Through experimental comparison, it shows that the algorithm proposed by the present invention has a better effect of removing mis-matching points than the currently commonly used algorithms, and preferably solves the problem of accurate registration of the filter array multi-spectral images in complex terrains. The registered image can meet the requirements of the subsequent processing and application of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of the selection of double thresholds for surface fitting;

[0029] Figure 2 is a surface fitting diagram using the bicubic interpolation method;

[0030] Figure 3 is a surface fitting diagram using the triangular network inverse distance weighted interpolation method;

[0031] Figure 4 is a schematic diagram of the distance difference surface effect;

[0032] Figure 5 is a schematic diagram of the smoothing effect of the distance difference surface;

[0033] Figure 6 is a schematic diagram of the corresponding points of the matching points on the smoothed fitting surface;

[0034] Figure 7 is a schematic diagram of the threshold space;

[0035] Figure 8 It is a schematic diagram for judging the threshold of feature matching points;

[0036] Figure 9 It is a schematic diagram of gray resampling;

[0037] Figure 10 It is a schematic diagram of the position difference of matching points on an existing row / column image;

[0038] Figure 11 It is a filter array multispectral image;

[0039] Figure 12 It is a schematic diagram of the fine matching points after algorithm processing;

[0040] Figure 13 It is the comparison of the gray histograms of the difference images. Specific implementation manners

[0041] The present invention describes the specific steps in detail with reference to the accompanying drawings:

[0042] S1. Using the position distance difference of the matching points obtained by superimposing the matching points of the multi-spectral image to be registered on the reference image, fitting and interpolating to obtain a distance difference surface, and smoothing the fitted surface, as Figure 1 shown by the double-dashed line in. Then, the smoothed surface is translated upward by a certain tolerance unit to obtain an upper limit threshold surface, as Figure 1 shown by the solid line in. All matching points above the upper limit threshold surface can be considered as mis-matching points. The upper limit threshold surface can be used to well eliminate the mis-matching point B in the flat terrain area, the mis-matching point A in the undulating terrain area, and the mis-matching point C on the terrain slope. However, the mis-matching point D in the undulating terrain area and the slope mis-matching point E cannot be eliminated.

[0043] Therefore, the smoothed surface is translated downward by a certain tolerance unit to obtain a lower limit threshold surface, as Figure 1 shown by the dashed line in. Similarly, the matching points below the lower limit threshold surface can be considered as mis-matching points. Therefore, the lower limit threshold surface can be used to eliminate the matching points D and E.

[0044] The upper limit threshold surface and the lower limit threshold surface form a three-dimensional threshold space. Simply put, all the position differences of the matching points falling inside the three-dimensional threshold space can be determined as correct matching points within a reasonable range, while the matching points falling outside the three-dimensional threshold space are determined as mis-matching points.

[0045] S2. The key technologies involved in the surface fitting double-threshold matching mainly include four aspects: surface interpolation fitting, surface smoothing, threshold space determination, and threshold space judgment:

[0046] a. Surface interpolation fitting

[0047] There are many surface interpolation fitting algorithms, such as bicubic interpolation, bicubic spline interpolation, inverse distance weighted interpolation, etc. As Figure 2 shown, the distance difference surface generated by the bicubic interpolation method for the multi-spectral local image of the filter array, and the color of the chromaticity bar represents the displacement of the image points on the fitting surface.

[0048] From Figure 2 it can be seen that there is a large change in the displacement of the image points of adjacent points in the local area of the distance difference interpolation fitting result, showing the phenomenon of "severe up and down jitter". As in the Figure 2 dotted circle area in, there are even negative values, and the fitting surface obviously does not conform to the change trend of the image point displacement, which is not conducive to determining the three-dimensional threshold space.

[0049] Therefore, the inverse distance weighted interpolation method is used to generate the distance difference surface, so that the interpolation result is between the maximum value and the minimum value. First, a triangular network is constructed using the position of the matching points of the reference image, and then the inverse distance weighted interpolation method is used to fill in each triangle, as Figure 3 shown.

[0050] Figure 3 In, points A, B, and C are the vertices of a certain triangular network among the matching points of the reference image, while points A′, B′, and C′ are the matching points of the image to be registered. By calculating the position distance differences between A and A′, B and B′, and C and C′ respectively, m 1 , m 2 , m 3 are obtained. If the distances of a certain point in the triangle from vertices A, B, and C are d 1 , d 2 , d 3 respectively, then the value of the distance difference surface at this point is:

[0051] The distance difference surface generated by using the inverse distance weighted interpolation is as Figure 4 shown, and different colors in the chromaticity bar represent the displacement of the image points on the fitting surface.

[0052] b. Surface smoothing

[0053] Generally, the distribution of mismatched points in the multi-spectral image is relatively sparse and the number is small. Therefore, in order to prevent the mismatched points from affecting other matching points during the smoothing process, the present invention selects a 3×3 mean filter for smoothing. The smoothing effect diagram is as Figure 5 shown.

[0054] Convert the perspective below the conversion surface, as Figure 5 shown, then it can also be clearly seen from Figure 6 similar mismatched points D and E in Figure 1 (marked by circles in the figure).

[0055] c. Threshold space determination

[0056] As Figure 1 shown, taking the smoothed distance difference surface as the reference, floating up and down by a certain tolerance range respectively to form a three-dimensional threshold space. Among them, the selection of the threshold range determines to a certain extent the quality of the elimination of mis-matched points in the image. If the selected threshold range is too large, the mis-matched points in the image matching points cannot be effectively eliminated. On the contrary, the remaining correct matching points cannot meet the requirements of the multi-spectral image registration accuracy.

[0057] The extraction accuracy of the feature points of the multi-spectral image extracted by the present invention is ±0.5 pixels. Add ±0.5 to the smoothed distance difference surface to form a three-dimensional threshold space. Note: If in a flat area, adding -0.5 may make the lower limit threshold surface lower than 0, but it does not affect the subsequent threshold judgment, as Figure 7 shown.

[0058] d. Threshold space judgment

[0059] The coordinates of the feature matching points are generally not integers. For example, Figure 8 the matching points A, B, and C shown. The dotted round dots in the figure are the distance differences d i of the matching points, the solid round dot at the top is the upper limit threshold U i and the dotted curved round dot at the bottom is the lower limit threshold D i .

[0060] Interpolate and calculate the upper limit threshold U of each matching point according to the upper limit threshold surface i , interpolate and calculate the lower limit threshold D of each matching point according to the lower limit threshold surface i , and then use Equation (2) to judge whether the distance difference Y i is a correct matching point.

[0061]

[0062] S3. Gray resampling

[0063] First, establish triangular meshes for the reference image and the image to be registered respectively, and then perform gray resampling on each triangle in turn, as Figure 9 shown.

[0064] For a certain pair of matching triangles A 1 B 1 C 1 and A 2 B 2 C 2 , first, according to the vertex coordinates (x of triangle A 1 B 1 C 1 in the reference image11 ,y 11 )、(x 12 ,y 12 )、(x 13 ,y 13 ), triangle A in the image to be registered 2 B 2 C 2 The vertex coordinates (x 21 ,y 21 )、(x 22 ,y 22 )、(x 23 ,y 23 ), use formula (3) to obtain the geometric position transformation matrix H between the reference image and the image to be registered,

[0065]

[0066] Then, for a point P in the reference image triangle 1 (x 1 ,y 1 ), using formula (4), we can find its corresponding position P in the image to be registered. 2 (x 2 ,y 2 )

[0067]

[0068] Finally, in the image triangle A to be registered 2 B 2 C 2 The inverse distance weighted interpolation method is used to obtain P 2 The gray level of the point is assigned to P 1 If each point in each triangle is interpolated in turn, the grayscale interpolation work is completed.

[0069] Experiment and analysis

[0070] In order to verify the feasibility of the algorithm proposed in this paper, this paper uses an airborne filter array multispectral camera to capture multispectral images of a mountainous area. The detector of the filter array multispectral camera is a full frame transfer (Full Frame Transfer) CCD detector from DALSA, Canada. Two sets of images at three wavelengths of 580nm, 610nm and 650nm are selected for verification experiments. The experimental image size is 1977 pixels × 2034 pixels, as shown in Figure 1. Figure 11 shown.

[0071] Due to the terrain undulation in the imaging area of the multispectral camera, there is a large pixel displacement between the multispectral images in the 580nm, 610nm, and 650nm bands. If the multispectral images of the three bands are directly subjected to false color synthesis, the image targets in the synthesized false color image are severely ghosted (pseudo-edges), which is not conducive to the subsequent application and processing of the multispectral images.

[0072] As Figure 12 can be seen, when Algorithm 1 (RANSAC algorithm) processes the multispectral images of the filter array, while removing the mismatched points, a large number of correctly matched points are also removed, resulting in no matching points in some areas of the image.

[0073] Algorithm 2 (the algorithm in (Liang et al., 2015)) removes a large number of image matching points in the areas with terrain undulation, and the remaining points cannot meet the registration accuracy requirements. The comparison of the number of matching points of the three algorithms (Algorithm 1, Algorithm 2, and the present invention) is shown in Table 1.

[0074] Table 1 Comparison of the number of matching points

[0075]

[0076] Through data comparison, it is found that Algorithm 2 removes a large number of points, and there is a phenomenon of no matching points in some areas, which is relatively obvious; the number of points removed by Algorithm 1 is between the two, but there are also problems of no matching points in some areas. The present invention retains the largest number of matching points and the matching points are distributed throughout the image, which can ensure the accuracy requirements of image registration.

[0077] When screening the matching points, the present invention not only considers the surrounding point pairs but also removes the points that do not conform to the pixel displacement change in the surrounding area of the point pairs, effectively solving the problem of accurately screening the matching points based on the local area.

[0078] Taking the registration of the 610nm and 580nm spectral bands of Data 1 as an example, the removed point pairs are as Figure 13 shown. Among them, the upper right figure is the area with terrain undulation, and the lower right figure is the flat area.

[0079] Generally, the greater the degree of left-lower shift of the gray histogram of the difference image, the smaller the gray value of the difference image, and the more accurate the image registration effect; the more the histogram shifts to the upper right, the worse the registration effect. Therefore, in this paper, the shift of the difference image histogram is used to judge the quality of the image registration result, and further to illustrate the advantages and disadvantages of the matching point screening algorithm.

[0080] As Figure 13It can be seen from (a), (b), (c) and (d) that in the registration experiment of the two groups of images, the position offset between the original images is relatively large, and the histograms of the difference images are all located in the upper rightmost side. However, the gray histograms of the difference images of Algorithm 1 and Algorithm 2 are significantly shifted to the lower left relative to the original image data, indicating that Algorithm 1 and Algorithm 2 can achieve good registration effects.

[0081] Compared with the histograms of the difference images of Algorithm 1 and Algorithm 2, the histogram of the present invention is significantly shifted to the lower left, especially in the image registration of 610nm - 650nm, showing that the registration effect of the present invention is significantly better than that of Algorithm 1 and Algorithm 2, and further indicating that the matching point screening effect of the present invention is better than that of Algorithm 1 and Algorithm 2.

[0082] In the synthetic image processed by the present invention, the ground objects are clear, the details are more distinct, and there is no obvious pseudo - edge phenomenon, indicating that the present invention can better handle the registration problem of the filter array multi - spectral images.

[0083] In summary, aiming at the problem of accurately selecting matching points for airborne filter array multi - spectral images, the method of the present invention is proposed, that is, the control points outside the threshold range are removed according to the surface fitted by the image point displacement of the matching points, reducing the influence of mis - matching points and low - precision matching points in the filter array multi - spectral images on the image registration result. Through experimental comparison, it shows that the method proposed by the present invention has a better effect of removing mis - matching points than the currently commonly used methods, better solves the problem of accurate registration of filter array multi - spectral images in complex terrains, and the registered images can meet the requirements of the subsequent processing and application of the images.

[0084] By analyzing and verifying the algorithms currently used for screening matching points in remote sensing images, it is found that using the global matrix cannot effectively remove mis - matching points and low - precision matching points in images with large image point displacements, or a large number of correct matching points are removed while removing mis - matching points, which cannot meet the requirements of image registration accuracy. The position difference fitting surface constructed by the present invention can reflect the change trend of the image point displacement of the whole image. According to the characteristic that the image point displacements of the matching points in the same area are approximately the same, mis - matching points around the correct matching points can be effectively removed.

[0085] The selection of the threshold determines to a certain extent the quality of the removal effect of mis - matching points in the image. In this paper, a threshold of 0.5 is selected to achieve good experimental results, but it cannot guarantee full applicability to all images. For different types of image data, how to quickly select a more appropriate threshold is the focus of the next research. In addition, the fitting surface part takes a long time, and when the image data is large, the running speed is more obvious. Accelerating through GPU and optimizing the program are also the research work in the next step.

Claims

1. A method for double-threshold elimination by fitting the surface of the position difference of matching points, characterized in that: The steps are as follows: S1. Using the position distance difference of the matching points obtained by superimposing the matching points of the multi-spectral images to be registered on the reference image, fitting and interpolating to obtain the distance difference surface, smoothing the fitted surface, translating the smoothed surface upward to obtain the upper threshold surface. All matching points higher than the upper threshold surface are mis-matching points. Translating the smoothed surface downward to obtain the lower threshold surface. The upper threshold surface and the lower threshold surface form a three-dimensional threshold space. Using the upper threshold surface to eliminate the mis-matching points in the flat terrain area, the mis-matching points in the hilly terrain area, and the mis-matching points on the terrain slope. Using the lower threshold surface to eliminate the mis-matching points D in the hilly terrain area and the mis-matching points E on the slope; S2. The double-threshold matching of surface fitting includes four aspects: surface interpolation fitting, surface smoothing, threshold space determination, and threshold space judgment: a. Surface interpolation fitting Construct a triangular mesh using the positions of the matching points in the reference image, and then fill it using the inverse distance weighted interpolation method within each triangle. Points A, B, and C are the vertices of a certain triangular mesh among the matching points in the reference image, and points A′, B′, and C′ are the matching points in the image to be registered. By calculating the position distance differences between A and A′, B and B′, and C and C′ respectively, m 1 , m 2 , m 3 are obtained. The distances of a certain point in the triangle from vertices A, B, and C are d 1 , d 2 , d 3 respectively. Then the value of the distance difference surface at this point is: Generating the distance difference surface by inverse distance weighted interpolation; b. Surface smoothing Selecting a 3×3 mean filter for smoothing processing, converting the perspective below the surface to obtain the mis-matching points D and E; c. Threshold space determination Adding ±0.5 to the smoothed distance difference surface to form a three-dimensional threshold space, that is, in the flat area, adding -0.5 to make the lower threshold surface lower than 0, but it does not affect the subsequent threshold judgment; d. Threshold space judgment Interpolate and calculate the upper limit threshold U of each matching point according to the upper limit threshold surface i Interpolate and calculate the lower limit threshold D of each matching point according to the lower limit threshold surface i Then use Equation (2) to judge the distance difference d i Whether it is a correct matching point where d i is the matching point distance difference; S3. Gray resampling For a pair of matching triangles A 1 B 1 C 1 and A 2 B 2 C 2 , first, according to the vertex coordinates (x 1 B 1 C 1 , y 11 , y 11 )、(x 12 , y 12 )、(x 13 , y 13 ) of triangle A 2 B 2 C 2 in the reference image, and the vertex coordinates (x 21 , y 21 )、(x 22 , y 22 )、(x 23 , y 23 ) of triangle A 2 B 2 C 2 in the image to be registered, use Equation (3) to obtain the geometric position transformation matrix H between the reference image and the image to be registered. Then, for a point P inside the reference image triangle 1 (x 1 , y 1 ), using Equation (4), its corresponding position P 2 (x 2 , y 2 ) in the image to be registered is obtained Finally, in the triangle A 2 B 2 C 2 of the image to be registered, the gray value of point P 2 is obtained by using the inverse distance weighted interpolation method and assigned to point P 1 . If the interpolation is performed for each point in each triangle in turn, the gray value interpolation work is completed.

Images