Improved census and confidence guided lunar surface image matching method

Abstract

The application relates to a lunar surface image matching method based on improved census and confidence guidance, which comprises the following steps: acquiring binocular images of a lunar surface and performing pretreatment, and obtaining a core line image through core line correction processing; constructing an image pyramid based on the core line image, and calculating a matching cost by using a census transformation algorithm based on a bit string; for each layer of the image pyramid, calculating a path cost of an SGM algorithm based on the matching cost, calculating the confidence of each path, and selecting a path for cost aggregation based on the confidence, and calculating a disparity by using the SGM algorithm; and realizing the matching of the lunar surface image based on the disparity. Compared with the prior art, the application has the advantages of strong robustness, good matching effective pixel effect and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of planetary photogrammetry image matching, and particularly to a lunar surface image matching method based on improved census and confidence guidance. BACKGROUND

[0002] Lunar surface three-dimensional reconstruction is an important part of lunar exploration, and the lunar surface digital elevation model has been widely used in various studies of the moon, such as lunar surface geology, landing site selection, exploration path planning, and impact crater extraction. Dense image matching is a key step in three-dimensional reconstruction, which generates a disparity map by finding corresponding points in binocular or multi-view images.

[0003] Dense matching methods can be divided into local matching methods and global matching methods according to whether global constraints are considered. Local matching methods are fast but ineffective in areas with occlusions and insufficient texture information; global matching methods consider global constraints and have better matching results than local matching methods, but are time-consuming. In contrast, semi-global matching algorithms achieve a good balance between efficiency and accuracy, and the principle is to use multi-directional dynamic programming to approximate the global energy optimum. The SGM algorithm (binocular stereo matching algorithm) is currently used in the field of mapping.

[0004] The census transform is commonly used in cost calculation in stereo matching. This algorithm measures the cost by comparing the relative relationship between pixels within a local window, which is robust to light differences, but the cost is not reliable in weak texture areas and areas with discontinuous disparity. There are many impact craters on the lunar surface, and the discontinuous disparity at the impact craters poses a great challenge to dense matching of lunar surface images. Therefore, it is necessary to improve the cost calculation step to enhance the robustness of the matching cost. On the other hand, SGM accumulates the cost through multi-path cost aggregation, and finally obtains the disparity through the winner-takes-all strategy. However, in difficult matching areas, the disparities obtained by different paths are not consistent, and the disparity obtained by simple accumulation and WTA is not reliable. It is necessary to determine which path is more reliable and reduce the spread of unreliable information. At the same time, the lunar surface texture is weak and the light difference is significant, which poses a great challenge to dense matching. It is necessary to modify the SGM algorithm to improve the matching effect in these difficult matching areas. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a lunar surface image matching method based on improved census and confidence guidance to solve or partially solve the problem that existing methods cannot effectively match the lunar surface.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] In one aspect of the present application, a lunar surface image matching method based on improved census and confidence guidance is provided, comprising the following steps:

[0008] Obtaining binocular images of the lunar surface and pre-processing, and obtaining epipolar images through epipolar correction processing;

[0009] Constructing an image pyramid based on the epipolar images, and calculating a matching cost by using a census transformation algorithm based on a bit string;

[0010] For each layer of the image pyramid, calculating a path cost of an SGM algorithm based on the matching cost, calculating the confidence of each path, and selecting a path for cost aggregation based on the confidence, and calculating a disparity by using the SGM algorithm;

[0011] Realizing the matching of the lunar surface images based on the disparity.

[0012] As a preferred technical solution, the obtaining of the epipolar images comprises the following steps:

[0013] Extracting homonymous matching points from the binocular images by using a feature matching algorithm;

[0014] Calculating a fundamental matrix parameter by using a gold standard algorithm based on the homonymous matching points;

[0015] Calculating a rotation matrix for horizontal transformation of the epipolar lines, and solving a scale and a transformation matrix by using least squares based on the homonymous matching points;

[0016] Outputting the epipolar images.

[0017] As a preferred technical solution, the calculation of the matching cost comprises the following steps:

[0018] Obtaining the image pyramid by performing low-pass filtering and down-sampling on the epipolar images;

[0019] For a single layer image in the image pyramid, calculating an overall pixel mean value in a preset size window and pixel mean values in multiple directions, and selecting a threshold value therefrom;

[0020] Comparing each pixel in the window based on the threshold value to obtain a binary bit string;

[0021] Calculating the matching cost based on the bit strings corresponding to the two view angles in the binocular images.

[0022] As a preferred technical solution, the calculation process of the path cost comprises the following steps:

[0023] For any path of any pixel, a smooth term and a minimum cost value are calculated, and a path cost corresponding to the current path is calculated based on the smooth term, the minimum cost value and the matching cost;

[0024] For any pixel, path costs of each path are accumulated to obtain a path cost of the pixel.

[0025] As a preferred technical solution, the calculation of the confidence of each path comprises the following steps:

[0026] A disparity feature is calculated based on the path cost of each path;

[0027] For any path, a plurality of sub-confidence values are calculated based on the path cost and the disparity feature, and a total confidence is obtained by accumulating each sub-confidence, thereby obtaining a confidence map,

[0028] The sub-confidence comprises at least one of U p , MM determined based on a difference between the minimum cost and the second minimum cost, NLM obtained based on a two-exponential form of the confidence, PKR determined based on a ratio between the second local minimum cost and the minimum cost, APKR obtained based on a mean value of PKR of all pixels within a certain neighborhood window of the pixel, VAR obtained based on a variance within a certain neighborhood of the pixel, MSM represented by a negative form of the minimum cost, CUR obtained based on a local shape of a cost curve of the minimum cost.

[0029] As a preferred technical solution, the process of selecting a path for cost aggregation based on the confidence comprises the following steps:

[0030] Each path under any pixel is sorted according to the confidence, and a path with the lowest confidence is determined;

[0031] For the path with the lowest confidence, the pixels under the path are sorted from high to low according to the confidence, and pixels with high confidence are selected at a preset proportion for cost aggregation.

[0032] As a preferred technical solution, the calculation of the disparity comprises the following steps:

[0033] Based on the result after cost aggregation, a winner-takes-all strategy is used to calculate the disparity, thereby obtaining a disparity map.

[0034] As a preferred technical solution, after the disparity is calculated, the following steps are further included:

[0035] The disparity map is subjected to left-right consistency detection to remove false disparity in an occluded area;

[0036] The disparity map is smoothed by using median filtering and texture filtering.

[0037] In another aspect of the present application, an electronic device is provided, comprising: one or more processors and a memory having stored therein one or more programs, the one or more programs including instructions for performing the improved census and confidence guided lunar surface image matching method described above.

[0038] In another aspect of the present application, a computer-readable storage medium is provided, comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for performing the improved census and confidence guided lunar surface image matching method described above.

[0039] Compared with the prior art, the present application has the following advantages:

[0040] (1) Effective matching of lunar surface images: the present application modifies the matching cost of semi-global matching, uses a census transformation algorithm based on a bit string to calculate the matching cost, making it more robust, and combines confidence information to reduce the spread of false information, effectively improving the matching effect in areas with discontinuous depth and occluded edges.

[0041] (2) Robustness of matching cost: based on the census transformation, the present application uses a binary bit string to encode the relationship between pixels, and adaptively determines the judgment threshold using pixel information within the local window, effectively reducing the sensitivity of the original census transformation to the center pixel, while capturing subtle changes in the relationship between pixels, effectively enhancing the robustness of the matching cost.

[0042] (3) Good effective pixel matching effect: the semi-global matching method based on confidence guidance in the present application combines multiple confidence levels to obtain effective decision indicators, and the pixels on unreliable paths do not participate in the cost aggregation process, effectively reducing the spread of false information, allowing more effective pixels to be matched in occluded edge and weak texture areas. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 Flowchart of the improved census and confidence guided lunar surface image matching method in the embodiment;

[0044] Figure 2 Original LROC NAC image in the embodiment;

[0045] Figure 3 Radial image obtained after radial correction in the embodiment;

[0046] Figure 4 Combined confidence map in the embodiment;

[0047] Figure 5 a disparity map obtained after consistency check in the embodiment;

[0048] Figure 6 a disparity map obtained after filtering in the embodiment;

[0049] Figure 7 a disparity map generated by semi-global matching and the application in the embodiment. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the application will be apparently and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work should belong to the protection scope of the application.

[0051] Embodiment 1

[0052] In order to obtain more accurate image disparity information, especially in difficult matching areas such as lunar surface impact craters, weak texture areas of images and edges of occluded areas, see Figure 1 The embodiment provides a lunar surface image matching method based on improved census and confidence guidance, and the method comprises the following steps.

[0053] Step S1, pre-processing left and right images to obtain epipolar images through epipolar rectification.

[0054] Step S1 comprises the following sub-steps.

[0055] Step S101, extracting homonymous matching points from left and right images by using a feature matching algorithm.

[0056] Step S102, solving five unknown parameters in an affine fundamental matrix F by using a gold standard algorithm according to the homonymous points obtained by feature matching.

[0057] Step S103, calculating a rotation matrix to make the epipolar line horizontal.

[0058] Step S104, least square solving scale and shear transformation matrices H1 and Hr of left and right image matching points.

[0059] Step S105, outputting left and right epipolar images.

[0060] See Figure 2 LROC NAC images, Figure 3 epipolar images obtained after epipolar rectification.

[0061] Step S2, constructing image pyramid layers for dense matching, improving the census transform algorithm to calculate the matching cost.

[0062] The image pyramid construction process includes: low-pass filtering and smoothing the image, and down-sampling the smoothed image.

[0063] The improved census transform algorithm includes:

[0064] Obtain left and right epipolar images, define an initial window of size n*n, calculate the pixel mean Imean of all pixels in the window, and calculate the pixel mean Imup, Imdown, Imleft, Imright of the center pixel and its four directions above and below and left and right respectively. From the five values, select the maximum and minimum values as the high threshold hiqh and the low threshold low for subsequent determination. thresh thres .

[0065] Compare each pixel in the window with the low threshold and the high threshold in turn, and store the comparison result in binary form. The formula is as follows:

[0066]

[0067] Where I(p) represents the pixel point in the window, low thres , high thres are the low threshold and the high threshold for determination. After transforming all the pixels in the window, concatenate the obtained bit string to represent the transform result of the window. The formula is as follows:

[0068]

[0069] Obtain the bit string of the left image window and the bit string of the right image window respectively, calculate the Hamming distance of the two bit strings to obtain the matching cost. The formula is as follows:

[0070] C census (u, v, d) = Hamming[c cenl (u, v), c cenr (u-d, v)]

[0071] d∈[d min , d max ]#(3)

[0072] Where c cenl (u, v) represents the census transform result of pixel (u, v) in the left image, c cenr (u-d, v) represents the census transform result of pixel (u-d, v) in the right image, d represents the disparity, d min and d​max The search range for the parallax.

[0073] The pyramid is constructed with 5 layers, and the Census transform uses a 5*5 window for calculation.

[0074] Step S3, dense matching is performed using a semi-global matching (SGM) algorithm, based on the parallax map obtained for each layer, eight kinds of confidence are calculated according to the parallax and cost features, and the confidence of each path of SGM is obtained by fusion.

[0075] Step S3 includes the following sub-steps:

[0076] Step S301, dense matching is performed using the SGM algorithm. The SGM algorithm mainly consists of four parts, which are: cost calculation, cost aggregation, parallax calculation, and parallax optimization. The cost calculation uses the improved census transform described in step S2. The cost aggregation part uses a method similar to dynamic programming, using one-dimensional path aggregation to approximate two-dimensional optimization, which is comparable in effect but more efficient than other solutions.

[0077] The energy function formula of SGM is as follows:

[0078]

[0079] Among them, the first term is the data term, which represents the cumulative matching cost of all pixels when the parallax map is D. The second and third terms are smoothing terms, which represent the penalty for all pixels q in the Np neighborhood of pixel p. P1 is small, which is a penalty for the case where the parallax of adjacent pixels changes very little (1 pixel). P2 is much larger than P1, which is a penalty for the case where the parallax of adjacent pixels changes very much (more than 1 pixel).

[0080] Among them, P2 is generally dynamically adjusted according to the gray difference of adjacent pixels, and the formula for adjusting P2 is as follows:

[0081]

[0082] Among them, P2' is the initial value of P2, which is generally set to a number much larger than P1.

[0083] The cost aggregation of SGM is performed along eight paths, and the path cost calculation method of pixel p along a certain path r is as follows:

[0084]

[0085] Wherein the first term C is the matching cost of the data item; the second term is a smoothing term, which means no penalty if the previous pixel in the direction r has a disparity d, P1 penalty if the previous pixel in the direction r has a disparity d with a difference of 1, and the fourth term in the second term means that the previous pixel in the direction r has the minimum cost in all the disparity range, and P2 penalty is applied; the third term is the minimum cost of the previous pixel in the direction, which is mainly to prevent the value from being too large.

[0086] After the cost of each path is obtained, the cost curve of each pixel under the current path is obtained, and the minimum cost and the second minimum cost on the cost curve are obtained to calculate the cost curve features.

[0087] The total path cost of the pixel is accumulated by the cost of each path, and the formula is as follows:

[0088]

[0089] Wherein, S(p, d) represents the matching cost corresponding to the disparity d obtained after the aggregation of multiple paths.

[0090] After the cost aggregation is completed, the winner takes all strategy is used to obtain the disparity of each pixel, so as to obtain the disparity map, and the variance and other disparity map features can be calculated from the disparity map.

[0091] In step S302, eight kinds of confidence are calculated according to the cost curve features and the disparity map features, and are fused to obtain the confidence of each path of SGM. The cost curve of each pixel of each path can be obtained in step S301, and the disparity map features can be calculated from the disparity map obtained in step S301.

[0092] In step S302, eight kinds of confidence are calculated according to the cost curve features and the disparity map features, and are fused to obtain the confidence of each path of SGM. The cost curve of each pixel of each path can be obtained in step S301, and the disparity map features can be calculated from the disparity map obtained in step S301.

[0093] The calculation formula of each confidence is as follows:

[0094]

[0095] The first confidence is the U obtained based on the SGM uncertainty p where L r (p, d) represents the minimum cost of each path, ∑ r min d L r (p, d) represents the sum of the minimum cost of all 8 paths, min d ∑ r L r (p, d) represents the minimum total cost.

[0096] MM(p) = c d2m (p) - c d1 (p) # (9)

[0097] The second confidence is the MM determined by the difference between the minimum cost and the second smallest cost, where c d2m (p) represents the second local minimum of the curve, c d1 (p) represents the minimum cost on the cost curve.

[0098]

[0099] The third confidence is the NLM obtained in the form of confidence two exponentials, where c d2m (p) represents the second local minimum of the curve, c d1 (p) represents the minimum cost on the cost curve.

[0100]

[0101] The fourth confidence is the PKR determined by the ratio between the second local minimum cost and the minimum cost, where c d2m (p) represents the second local minimum of the curve, c d1 (p) represents the minimum cost on the cost curve.

[0102]

[0103] The fifth confidence is the APKR obtained by the average of the PKR of all pixels within a certain neighborhood window range for each pixel, where c d2m(p) (q) represents the second local minimum of the q-pixel cost curve, c d1(p) (q) represents the minimum cost on the q-pixel cost curve.

[0104]

[0105] The sixth confidence is VAR obtained from the variance in a certain neighborhood of the pixel in the disparity map, where d1(q) represents the disparity of pixel q, d1(p) represents the disparity of x pixel p, and μ() represents the calculation of the mean in the local window.

[0106] MSM(p) = -c d1 (p) # (14)

[0107] The seventh confidence is MSM represented by the negative form of the minimum cost, where c d1 (p) represents the minimum cost on the cost curve.

[0108] CUR(p) = -2c d1 (p) + c d1+1 (p) + c d1-1 (p) # (15)

[0109] The eighth confidence is CUR obtained from the local shape of the cost curve corresponding to the minimum cost, where c d1 (p) represents the minimum cost on the cost curve, c d1+1 (p) and c d1-1 (p) represents the cost value corresponding to the adjacent disparity on the left and right sides of d1.

[0110] For the eight confidences obtained, the values of each are normalized, and then added to obtain the combined confidence. Each path is calculated in this way to obtain the confidence of each path.

[0111] The eight confidences are calculated for each path, the values of each confidence are normalized, and the eight normalized confidences are added to obtain the combined confidence map. The confidence value of each pixel can be obtained from the confidence map. Figure 4 The combined confidence image is shown.

[0112] S4, according to the confidence of each path, the pixels located in the top 20% reliable pixels in the most unreliable path are selected to participate in cost aggregation, and other pixels do not participate in cost aggregation.

[0113] Step S4 is specifically: the confidence images obtained from the eight paths are respectively counted for the total sum of confidences, the eight total sum values are sorted, and the corresponding most unreliable path is selected. Referring to the confidence value of each pixel of the path, the pixels with confidence values in the top 20% reliable pixels normally participate in cost aggregation, and the cost of other pixels is set to a constant value, thereby reducing the propagation of information of these unreliable pixels.

[0114] The total confidence value is calculated from the confidence image of each path, and the most unreliable path is determined by comparing the confidence values of the eight paths. For this path, the confidence of the pixels is determined, and the top 20% reliable pixels are selected. The costs of these pixels on the path are retained for the subsequent cost aggregation process. For the unreliable pixels, the matching cost is modified to 0, so that they will not be added to the final cost in the final cost aggregation.

[0115] S5, a winner-takes-all strategy is used to obtain the disparity, and the disparity map is optimized by left-right consistency detection, median filtering and texture filtering.

[0116] Step S5 includes the following sub-steps:

[0117] Step S501, a winner-takes-all strategy is used to obtain the disparity;

[0118] Step S502, left-right consistency detection is performed on the disparity map to remove false disparity in the occluded area;

[0119] Step S503, median filtering and texture filtering are used to smooth the disparity map;

[0120] Figure 5 The disparity map obtained after left-right consistency check is shown, Figure 6 The disparity map obtained after filtering is shown.

[0121] In order to verify the effectiveness of the method proposed in this paper, the disparity map obtained in this paper is compared with the disparity map obtained by the original SGM algorithm (matching cost using census) for quantitative comparison and analysis.

[0122] By Figure 7 The comparison of the disparity maps shows that the disparity map obtained in this paper can match more effective pixels at the edge of the occluded area and the weak texture area at the top, and the disparity does not have sudden changes, is continuous and consistent with the surrounding disparity, and is effective. Through statistical analysis, it can be seen that the matching success rate of the original SGM algorithm is 80.11%, and the effective matching pixels are 836033, while the matching success rate of the method in this paper is 81.71%, and the effective matching pixels are 852714, which improves the matching success rate by 1.6%. It can be seen that the method in this paper improves the matching effect of the lunar surface image.

[0123] Embodiment 2

[0124] The embodiment provides an electronic device, which comprises one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs comprise instructions for executing the improved census and confidence guided lunar surface image matching method.

[0125] Example 3

[0126] This embodiment provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs comprising instructions for performing the above-described method of improved census and confidence-guided lunar surface image matching.

[0127] The above describes only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be encompassed within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A lunar surface image matching method based on improved census and confidence-guided methods, characterized in that, Includes the following steps: Binocular images of the lunar surface were acquired and preprocessed, and epipolar images were obtained through epipolar line correction. An image pyramid is constructed based on the epipolar image, and the matching cost is calculated using a bit string-based census transform algorithm. For each layer of the image pyramid, the path cost of the SGM algorithm is calculated based on the matching cost, the confidence of each path is calculated, and the path for cost aggregation is selected based on the confidence. The disparity is then calculated using the SGM algorithm. Matching of lunar surface images is achieved based on the parallax. The calculation of the matching cost includes the following steps: The image pyramid is obtained by low-pass filtering, smoothing, and downsampling the epipolar image. For a single-layer image in the image pyramid, calculate the overall pixel mean and the pixel mean in multiple directions within a window of a preset size, and select a threshold from them; Based on the threshold, each pixel within the window is compared to obtain a binary bit string; The matching cost is calculated based on the bit strings corresponding to the two viewpoints in the binocular images. The calculation of the confidence score for each path includes the following steps: Calculate disparity features based on the path cost of each path; For any path, multiple sub-confidence values ​​are calculated based on the path cost and the disparity feature. These sub-confidence values ​​are then summed to obtain the total confidence score, resulting in a confidence map. The sub-confidence level includes those obtained based on SGM uncertainty. At least one of the following: MM determined based on the difference between the minimum cost and the second minimum cost; NLM obtained based on the confidence level in a bi-exponential form; PKR determined based on the ratio between the second local minimum cost and the minimum cost; APKR obtained based on the mean PKR of all pixels within a certain neighborhood window of a pixel; VAR obtained based on the variance within a certain neighborhood of a pixel; MSM represented by the negative form of the minimum cost; and CUR obtained based on the local shape of the cost curve of the minimum cost.

2. The lunar surface image matching method based on improved census and confidence-guided methods according to claim 1, characterized in that, The acquisition of the epipolar image includes the following steps: Feature matching algorithms are used to extract corresponding matching points from binocular images; Based on the corresponding matching points, the fundamental parameters of the radiation matrix are calculated using the gold standard algorithm. Calculate the rotation matrix used to transform the epipolar lines horizontally, and solve for the scale and transformation matrix using least squares based on the corresponding matching points; Output the epipolar image.

3. The lunar surface image matching method based on improved census and confidence-guided methods according to claim 1, characterized in that, The path cost calculation process includes the following steps: For any path of any pixel, calculate the smoothing term and the minimum cost value. Based on the smoothing term, the minimum cost value, and the matching cost, calculate the path cost corresponding to the current path. For any pixel, the path cost of each path is summed to obtain the path cost of that pixel.

4. The lunar surface image matching method based on improved census and confidence-guided methods according to claim 1, characterized in that, The process of selecting a path for cost aggregation based on the confidence level includes the following steps: Sort the paths under any pixel according to their confidence level, and determine the path with the lowest confidence level; For the path with the lowest confidence, the pixels under the path are sorted from highest to lowest confidence, and the pixels with the highest confidence are selected for cost aggregation at a preset ratio.

5. The lunar surface image matching method based on improved census and confidence-guided methods according to claim 1, characterized in that, The calculation of disparity includes the following steps: Based on the results of cost aggregation, a winner-take-all strategy is used to calculate disparity, resulting in a disparity map.

6. The lunar surface image matching method based on improved census and confidence-guided methods according to claim 1, characterized in that, After calculating the disparity, the following is also included: Perform left-right consistency detection on the disparity map to remove erroneous disparities in occluded areas; The disparity map is smoothed using median filtering and texture filtering.

7. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the lunar surface image matching method based on the improved census and confidence-guided method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, Includes one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the lunar surface image matching method based on the improved census and confidence-guided method as described in any one of claims 1-6.

Images