Lunar surface stereo matching method based on multi-cost fusion and superpixel optimization
Through multi-cost fusion and superpixel optimization methods, the accuracy problem of the stereo matching algorithm under high light and shadow occlusion in the lunar scene is solved, and the generation of high-precision disparity maps is achieved, which enhances the accuracy of stereo matching in the lunar surface.
Patent Information
- Application Number
- CN202411570829.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the lunar scene, due to the high-contrast lighting differences, lack of texture information and shadow occlusion, the existing stereo matching algorithm is difficult to obtain high-precision parallax maps, especially the local method is not robust enough and the global method calculation complexity is High and mismatch is prone to occur in areas with missing textures.
The cost calculation method of improved Census changes and color intensity information based on multi-cost fusion is adopted, combined with multi-level superpixel disparity optimization, and high-precision disparity maps are generated through steps such as the winner-oriented strategy and graph cutting algorithm.
It effectively reduces the impact of unique lighting conditions on the moon surface, enhances the three-dimensional matching accuracy of weak texture areas, solves the shadow occlusion problem, and improves the accuracy of three-dimensional matching in the moon surface scene.
Smart Images

Figure CN119068036A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a lunar surface stereo matching method based on multi-cost fusion and super-pixel optimization. Background Art
[0002] High-precision three-dimensional data of lunar surface topography is crucial for lunar exploration and scientific research station construction. Currently, most of the three-dimensional data of lunar surface topography is obtained based on orbiters and has low resolution. The lunar surface binocular stereo vision system can obtain high-resolution images and estimate disparity through stereo matching technology, and finally generate accurate disparity maps to achieve high-precision three-dimensional reconstruction of lunar scenes. However, due to the unique environment of the moon, such as the lack of atmosphere, low albedo of the lunar surface resulting in high-contrast lighting differences, lack of lunar surface texture information, and common shadow occlusion caused by low solar altitude angle, traditional stereo matching algorithms will have serious errors and are no longer applicable in special lunar scenes.
[0003] Stereo matching algorithms can be divided into two categories according to different constraints and search strategies: local stereo matching algorithms and global stereo matching algorithms. Local algorithms use information such as image intensity or gradient to perform cost calculation and aggregation within a set search range, and finally use the winner-take-all (WTA) algorithm to calculate the disparity map. However, this type of algorithm is not robust enough when faced with sparse textures and extreme lighting conditions of lunar scenes, resulting in low disparity estimation accuracy. Later, the Census variation stereo matching algorithm proposed by Zali alleviated the impact of lighting to a certain extent, but it performed poorly in weak texture areas and was also easily disturbed by noise.
[0004] Compared with the local method, the global method is aimed at the entire image domain, by constructing an energy function with a data term and a smooth term, and by optimizing this energy function to obtain a smooth and accurate disparity map. This type of method can more effectively solve the problem of image matching ambiguity, especially for discontinuous and weak texture areas. The literature "Peng Man, Di Kaichang, Liu Zhaoqin. Dense matching of deep space exploration images based on adaptive Markov random fields [J]. Journal of Remote Sensing, 2014, 18(01): 77-89." proposed an adaptive Markov stereo matching model for deep space exploration images. By adaptively determining the disparity range and combining it with the window matching algorithm, a relatively accurate disparity estimation was achieved. However, this method has high computational complexity and low efficiency.
[0005] The paper "Heiko H. Stereo processing by semiglobal matching and mutual information. [J]. IEEE transactions on pattern analysis and machine intelligence, 2008, 30 (2): 328-41." proposed a semi-global stereo matching (SGM) algorithm, which combines the advantages of local and global methods and has a good balance between accuracy and computational complexity. The paper "Li H, Chen L, Li F. An Efficient Dense Stereo Matching Method for Planetary Rover. [J]. IEEE Access, 2019, 748551-48564." established a coarse-to-fine pyramid framework based on the SGM algorithm to improve the accuracy of stereo matching in lunar scenes. However, when processing texture-missing areas, both papers are prone to false matching, resulting in inaccurate disparity maps. In semi-global stereo matching (SGM), the traditional two-dimensional Markov random field (MRF) optimization problem is approximated as a collection of one-dimensional linear optimizations along all directions, which improves efficiency while ensuring a certain degree of accuracy. However, each path independently summarizes the matching cost and does not share information with each other. Once the data item information is insufficient, it will lead to matching holes, resulting in inaccurate final matching results, and it is difficult to cope with the special lighting environment and weak texture areas on the lunar surface. Therefore, it is still of great significance to explore a high-precision stereo matching method that is suitable for lunar scenes. Summary of the invention
[0006] In order to solve the above technical problems, the present invention proposes a lunar stereo matching method based on multi-cost fusion and superpixel optimization. Multi-cost fusion adopts a cost calculation method that improves the fusion of census change and color intensity information and a disparity optimization method based on multi-level superpixels. It can effectively deal with the high-contrast lighting differences, lack of texture information and shadow occlusion problems in lunar scenes, thereby obtaining a high-precision disparity map.
[0007] To achieve the above object, the technical solution adopted by the present invention is as follows: A lunar surface stereo matching method based on multi-cost fusion and superpixel optimization includes the following steps: Step 1: Acquire a pair of lunar scene stereo images, and the lunar scene stereo image pair has been epipolar corrected; Step 2: Perform multi-cost fusion calculation on the lunar scene stereo image pair obtained in step 1, and obtain the fusion matching cost by fusion calculation based on the cost of improved Census change and the cost of image color intensity information; Step 3: Based on the fusion matching cost, the initial disparity map is obtained through the winner-takes-all strategy; Step 4: Perform multi-level super-pixel disparity optimization on the input image and the initial disparity map to obtain a high-precision disparity map.
[0008] The beneficial effects of the present invention compared with the prior art are: The present invention uses a matching cost calculation method that combines multi-cost fusion and improved census changes and color intensity information to reduce the impact of the unique lighting conditions on the lunar surface and enhance the stereo matching accuracy in weak texture areas. The present invention uses multi-level superpixel parallax optimization to solve the problem of large-area shadow occlusion caused by the extremely low sun angle in the lunar South Pole region. Ultimately, the present invention can effectively improve the accuracy of stereo matching in lunar scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 A flow chart of a lunar surface stereo matching method based on multi-cost fusion and superpixel optimization according to the present invention; Figure 2 This is a comparison example between the improved Census change method of the present invention and the original Census change method; Figure 3 : The experimental result diagram of the present invention, where (a) is the left camera image of the input lunar scene image, (b) is the initial disparity map obtained after improved cost calculation, (c) is the precise disparity map after multi-level disparity optimization, and (d) is the real disparity map of the lunar scene image. DETAILED DESCRIPTION
[0010] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0011] The present invention is further described in detail below in conjunction with specific embodiments.
[0012] like Figure 1 As shown, a lunar surface stereo matching method based on multi-cost fusion and superpixel optimization of the present invention comprises the following steps: Step 1: Acquire a pair of lunar scene stereo images, and the lunar scene stereo image pair has been epipolar corrected; Step 2: Perform multi-cost fusion calculation on the lunar scene stereo image pair obtained in step 1, and obtain the fusion matching cost by fusion calculation based on the cost of improved Census change and the cost of image color intensity information; Step 3: Based on the fusion matching cost, the initial disparity map is obtained through the winner-takes-all (WTA) strategy; Step 4: Perform multi-level super-pixel disparity optimization on the input image and the initial disparity map to obtain a high-precision disparity map.
[0013] Specifically, the step 2 includes: Step 21: Use the left camera and the right camera to obtain a left camera image and a right camera image respectively, that is, obtain a binocular image; For the obtained left and right camera images, the cost of the improved Census change is first calculated as follows: ; ; ; in, is the comparison function, is the pixel at the center of the window, In pixels Other pixels within the window centered at are the intensity values of the respective pixels, represents the neighborhood of pixel p, is the intensity difference threshold, It represents the weighted neighborhood intensity of pixels in the window. It is calculated by removing the pixel with the maximum and minimum intensity in the window and averaging the remaining pixel intensities. . Represents pixels Census change value, Represents the bitwise concatenation operator. Represents pixels In Parallax The cost value of Indicates the calculation of Hamming distance, is the pixel in the left camera image Census change value, It's pixels In Parallax The Census change value of the corresponding pixel point in the right camera image at the time, is the weight of the matching cost that controls the change of Census, and exp represents the exponential function.
[0014] Figure 2The example of the original Census transformation and the improved Census transformation of the present invention is shown. First, a 3×3 window is randomly selected from a lunar scene image, and noise is added to the window, replacing the central pixel I=195 with the noise pixel 100. The results show that the original Census transformation has failed, while the improved Census transformation result of the present invention has not changed.
[0015] Step 22: Calculate the cost of the image color intensity information using the following formula: ; in, Indicates the center pixel of the window In Parallax The matching cost when is the pixel in the left image The color intensity value, yes In Parallax The color intensity value of the corresponding pixel in the right camera image at the time i is the index value; R, G, B represent red, green, and blue. This method calculates the color matching cost of each pixel in the RGB color space of the image and obtains the final cost value by averaging the differences of the three color channels.
[0016] Step 23: The cost based on the improved Census change and the cost of the image color intensity information are fused to obtain the matching cost, and the result is limited to the range of 0-1 in exponential form.
[0017] The final matching cost is obtained by combining these two costs: ; in, It is a function of the image matching fusion cost. Both matching costs are exponentially normalized, and the final result is controlled within 0-2. is the matching cost weight of the color intensity scale.
[0018] Specifically, the step 3 includes: After completing the matching cost calculation, the optimal disparity is selected through the winner-takes-all (WTA) strategy to obtain the initial disparity map : ; in, The initial disparity map is obtained by selecting the minimum fusion cost of each pixel.
[0019] Specifically, step 4 includes: Step 41: Use a graph cut algorithm to perform superpixel segmentation on the input left and right camera images. These superpixels are a collection of pixels with high similarity. At the same time, the superpixels obtained by segmentation are used as basic units for subsequent processing to perform parallax optimization of the overall optimization layer and the detail optimization layer. Step 42: In the overall optimization layer, based on the Markov random field (MRF) principle, the superpixels obtained in the previous step are converted into graph nodes by minimizing the following energy function Finally, the disparity of each superpixel is determined: ; in, and is a pair of superpixels, and Superpixel and The average disparity of is a data item, is the smoothing term, is the balanced smoothing term image parameter, Superpixel A collection of is the set of adjacent superpixels. Since superpixels have irregular shapes, The parallax distribution within is modeled as a normal distribution : ; in, and Superpixels The average disparity and variance of superpixels are calculated, and adjacent superpixels with the same disparity are merged into new superpixels.
[0020] Then, by evaluating the disparity differences between different superpixels, the 3D spatial relationship between superpixels is calculated to determine the 3D neighborhood system of superpixels with discontinuous disparity, and the calculation formula is as follows: ; in, is a three-dimensional neighborhood system, is a three-dimensional neighborhood, is a pair of superpixels if their average disparity is similar, i.e. , then they belong to the three-dimensional neighborhood. Otherwise, . is the width of the disparity bins of the disparity histogram.
[0021] Step 43: In the detail optimization layer, based on the disparity map of the overall optimization layer, the RANSAC (random sampling consensus) algorithm is used to establish a tilted plane for each superpixel. .
[0022] Assuming that the disparity distribution within each superpixel conforms to the normal distribution, the effective distribution should be a unimodal distribution with a continuous disparity range. The disparity density distribution is used instead of the disparity distribution. Disparity density within a superpixel The definition is as follows: ; in, is the width of the disparity segment of the disparity histogram, Superpixel For each pixel within is the disparity value of the pixel in the initial disparity map, is a conditional judgment function, defined as follows: .
[0023] Since the initial plane It is generated by fitting each superpixel independently, which may cause the disparity quality to degrade in weak texture and occluded areas. In the above example, the local prior information of superpixels is used to optimize in a probabilistic way to effectively deal with the occlusion problem. The posterior probability It can be expressed as: ; in, is the observation value of the superpixel, Superpixel The number of pixels in . It is super pixel Under the observation data of possibility. Represents the prior probability of the disparity plane of the neighborhood superpixel t, which is obtained by the similarity of color and inlier rate. It is super pixel Superpixels in the 3D field, so Superpixel Under the observation data of possibility. Represents the neighborhood superpixel The prior probability of the disparity plane. Index value , is the number of pixels in the superpixel, and then the weighted least squares method is used to estimate the disparity plane of each superpixel to generate the detail optimization layer disparity plane. Finally, the adaptive mean filtering method is used to further refine it to obtain the final optimized high-precision lunar scene disparity map.
[0024] Example: In order to demonstrate the contribution of the present invention, the method of the present invention is verified using binocular stereo vision lunar scene dataset images, and a stereo matching result diagram of the lunar scene is given here. Figure 3 is the experimental result diagram, where Figure 3 (a) is the left camera image of the input lunar scene image. Figure 3 (b) is the initial disparity map obtained after the improved cost calculation. Figure 3 (c) is the accurate disparity map after multi-level disparity optimization. Figure 3 (d) is the true disparity map of the lunar scene image.
[0025] The parts not described in detail in the present invention belong to the known technologies in the technical field. The above description is only a specific example of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A lunar surface stereo matching method based on multi-cost fusion and superpixel optimization. It is characterized in that The steps include: Step 1: Acquire a pair of lunar scene stereo images, and the lunar scene stereo image pair has been epipolar corrected; Step 2: Perform multi-cost fusion calculation on the stereo image pair of the lunar scene obtained in step 1, and obtain the fusion matching cost by fusion calculation based on the cost of improved Census change and the cost of image color intensity information; Step 3: Based on the fusion matching cost, the initial disparity map is obtained through the winner-takes-all strategy; Step 4: Perform multi-level super-pixel disparity optimization on the input image and the initial disparity map to obtain a high-precision disparity map.
2. The lunar surface stereo matching method based on multi-cost fusion and superpixel optimization according to claim 1, It is characterized in that The step 2 comprises: Calculating the cost of improved census changes , calculated as follows: ; ; ; in, is the comparison function, is the pixel at the center of the window, In pixels Other pixels within the window centered at are the intensity values of the respective pixels, Represents pixels Neighborhood of is the intensity difference threshold, represents the weighted neighborhood intensity of pixels within the window, Represents pixels Census change value, Represents the bitwise concatenation operator; Represents pixels In Parallax The cost value of Indicates the calculation of Hamming distance, is the pixel in the left camera image Census change value, It's pixels In Parallax The Census change value of the corresponding pixel point in the right camera image at the time, is the weight of the matching cost that controls the change of Census, and exp represents the exponential function.
3. The lunar surface stereo matching method based on multi-cost fusion and superpixel optimization according to claim 2, It is characterized in that The step 2 comprises: The cost of calculating the image color intensity information is calculated as follows: ; in, Indicates the center pixel of the window In Parallax The matching cost when is the pixel in the left image The color intensity value, It's pixels In Parallax The color intensity value of the corresponding pixel in the right camera image at that time; i is the index value; R, G, B represent red, green, and blue; The result of the cost of the image color intensity information is limited to 0-1; The improved Census change cost and the image color intensity information cost are fused using the following formula to obtain the fused matching cost: : ; Among them, the cost of the improved Census change and the cost of the image color intensity information are both exponentially normalized, and the results are controlled within 0-2; is the matching cost weight of the color intensity scale.
4. The lunar surface stereo matching method based on multi-cost fusion and superpixel optimization according to claim 3, It is characterized in that The step 3 comprises: After obtaining the fusion matching cost, the optimal disparity value is selected through the winner-takes-all strategy to obtain the initial disparity map : ; in, It means that the minimum fusion cost of each pixel is selected to obtain the initial disparity map.
5. The lunar surface stereo matching method based on multi-cost fusion and superpixel optimization according to claim 1, It is characterized in that The step 3 comprises: Firstly, the input image is segmented into superpixels using the graph cut method, and superpixels are used as the basic processing unit for subsequent processing. Subsequently, two layers of parallax optimization are performed on the overall optimization layer and the detail optimization layer, including: For the overall optimization layer, the Markov random field is used to infer the average disparity of superpixels and establish an energy function as follows: Get the optimal disparity map: ; in, and is a pair of superpixels, and Superpixel Average disparity and superpixels The average disparity of is a data item, is the smoothing term, is the balanced smoothing term image parameter, Superpixel A collection of is a set of adjacent superpixels; For the detail optimization layer, a random sampling consistency method is used to fit the disparity plane of each superpixel. Then, the 3D neighborhood information from the superpixel is used to further refine the disparity map using a probability-based approach to obtain the final accurate disparity map.
Citation Information
Patent Citations
Local stereo matching method based on fusion cost calculation and weighted guide filtering
CN115601406A
Anti-noise stereo matching method and system for improving Census transformation
CN118015059A
Stereophotogrammetric method based on binocular vision
US11959744B1
