A method for registering lunar orbiter images with rover navigation camera images
By reconstructing the lunar surface point cloud model and image orthographic projection transformation, combined with feature extraction and affine transformation, the problem of registration between the lunar orbiter image and the rover navigation camera image under cross-scale, large inclination and point ambiguity conditions is solved, high-precision image registration and navigation camera positioning are achieved, and a multi-source image registration framework and accuracy evaluation method are provided.
Patent Information
- Application Number
- CN202211079236.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-05
- Publication Date
- 2025-09-09
- Estimated Expiration
- 2042-09-05
AI Technical Summary
Existing technologies make it difficult to effectively align lunar orbiter images with rover navigation camera images under cross-scale, large-angle, and point-ambiguity conditions, especially when the field of view of a single rover navigation camera is narrow, and it is impossible to effectively utilize multi-image information for alignment.
The lunar orbiter image and rover navigation camera image registration method is adopted. By reconstructing the lunar surface point cloud model, the lunar surface plane equation and projection transformation model parameters are calculated, and the image forward projection transformation and stitching are performed. The impact crater features are extracted by combining mean shift filtering, threshold segmentation and morphological processing. The affine transformation matrix is used for registration, and the registration accuracy is indirectly verified.
High-precision image registration is achieved under cross-scale, large-angle and point-ambiguity conditions, and the accuracy of the positioning navigation camera is controlled within a few pixels, which solves the problem that conventional registration methods fail under such conditions and provides a multi-source image registration framework and accuracy evaluation index.
Smart Images
Figure CN115439519B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of cross-scale, large-angle, point-blur, and multi-source image registration under lunar surface conditions, and specifically relates to a method for registering lunar orbiter images with rover navigation camera images. Background Art
[0002] Image registration is defined as the process of combining information from multiple images by superimposing images taken at different times, angles, and with different sensors. The main problem in image registration research is to compare the similarities between different images with overlapping areas using features describing different images or the images themselves, thereby finding the corresponding relationships between overlapping areas and different images, and aligning each pixel of the moving image to the reference image. Image registration is a comprehensive task. Different methods are designed for different problems. The common problem is to find a transformation to achieve matching between two or more images. Most image registration processes generally include four steps: the first step is feature extraction, the second step is feature matching. These two steps are the key to image registration, the third step is to estimate the parameters of the transformation model, and finally, image resampling and transformation operations are performed.
[0003] Traditional image registration methods can be divided into two categories: one based on regions, and the other based on feature points, which then further determine matching relationships. Region-based registration generally involves selecting a reference image and then maximizing a number of relevant metrics in the hope of finding the optimal matching position. The normalized cross-correlation algorithm is a classic approach, but it is computationally intensive and its maximum value is not particularly pronounced, making it suitable for homologous image registration. The phase correlation algorithm is suitable for heterologous images but is significantly affected by noise. A more widely used approach for heterologous image registration is to use mutual information to measure statistical correlation. These region-based matching algorithms are generally simple to implement but have a limited range of applications. Feature point-based matching is more widely used in practice. Currently, four algorithms with widespread application in feature extraction are Harris corner detection-based registration, ORB (Oriented FAST and Rotated BRIEF)-based image registration, SIFT (Scale Invariant Feature Transform)-based image registration, and SURF (Speeded Up Robust Features)-based image registration. The Harris corner algorithm has good illumination invariance and is insensitive to translation and rotation, but it does not adapt well to changes in scale. The ORB algorithm has the fastest computational speed among these methods, but is not scalable, requiring the camera to be nearly straight-on in practical applications. The SIFT descriptor is robust and invariant to translation, rotation, and scale. However, due to its nature as an extreme point in scale space, the distances between feature descriptors can be very small in weakly textured scenes, leading to a high incidence of false matches. Compared to the SIFT algorithm, SURF offers good stability under affine transformations and noise, outperforming SIFT in computational speed and adaptability to illumination changes. However, the SIFT operator performs better in adapting to scale, rotation, and blur. Feature matching generally uses nearest neighbor matching. After matching, the RANSAC algorithm is used to eliminate false matches. A transformation model is then selected and calculated based on the matching relationships, and finally, the floating image is transformed and resampled.
[0004] With the development of deep learning, numerous deep learning methods have been introduced in the field of computer vision. Deep learning can be applied to image registration tasks in two general ways. The first approach involves training networks using deep learning methods to replace the feature extraction, matching, and transformation parameter estimation steps in the traditional registration process. For example, the MatchNet network, developed in early research, uses a feature network composed of CNNs to generate feature descriptors and a metric network consisting of three fully connected layers to learn the distance (similarity) between feature descriptors. MC-CNN uses a deep Siamese network composed of several convolutional and fully connected layers to calculate similarity. This method achieved state-of-the-art results on the KITTI stereo dataset at the time, demonstrating that deep learning methods, including convolutional neural network-based feature extraction, are more effective than traditional hand-crafted features. Furthermore, networks such as PN-Net, which introduces a new loss function; LF-Net, which consists of a detector network and a feature descriptor network; RF-Net, which uses receptive fields; and ASLFeat, which incorporates variable convolution, are also examples of the first integration of deep learning and image registration. Recent advances in stereo matching technology have also been applied to the feature extraction and matching steps of registration. A representative example is the SuperPoint+SuperGlue method, a feature point matching network based on a graph neural network and an attention mechanism, proposed by Magicleap. In 2021, Ufuk et al. proposed a novel image matching method, DFM, inspired by the mental rotation paradigm. This method surpassed previous state-of-the-art methods tested on the Hpatches dataset. Using deep learning to simulate feature extraction, matching, and parameter estimation relies on the traditional feature point-based image registration framework, improving feature extraction speed and matching accuracy in practical applications. A second approach is to directly learn the geometric relationship between the reference image and the moving image to align the two images. Using deep learning to directly estimate transformation parameters is known as direct registration based on deep learning. This approach can be further categorized into supervised and unsupervised direct registration methods. The HomographyNet regression network can learn the homography between two images and simultaneously learn the CNN model parameters in an end-to-end manner, but requires labeled data and is relatively expensive. Unsupervised image registration offers greater adaptability than supervised methods.
[0005] Due to the complex nature of image registration and the diversity of the images to be registered and the application scenarios, it is impossible to design a universal method for all registration tasks. Instead, a series of methods, tailored to the specific problem at hand, are needed. Regarding the registration of lunar scene images, unlike the simulated and real-world datasets commonly used for image registration, lunar images lack texture information and exhibit numerous repetitive patterns. This poses the challenge of matching key points under point ambiguity. Currently, a common approach to registering images captured by lunar rovers on the lunar surface is graph matching with local affine invariance constraints. This approach achieves high accuracy in feature matching and image registration, resolving the key point correspondence problem under point ambiguity, is invariant to scale and rotation, and is more robust to outliers. Regarding the registration of navigation camera images with orbiter images, the Yutu series of lunar rovers, in my country's lunar exploration missions, employ a semi-automated teleoperation system, manually registering orbiter images with lunar surface images for positioning. There are no publicly available references to methods for localizing the navigation camera by registering orbiter images with those captured by the navigation camera. The current difficulties are as follows: (1) How to solve the registration problem under the conditions of cross-scale, large inclination, point ambiguity, and multiple sources; (2) How to comprehensively utilize multi-image information to complete the registration when the field of view of a single patrol navigation camera is narrow. Summary of the Invention
[0006] To overcome the shortcomings of existing technologies, addressing the issues of cross-scale, high-angle, and point-ambiguity between two types of heterogeneous images, as well as the limited field of view of a single rover navigation camera image, this paper provides a method for registering lunar orbiter images with rover navigation camera images, addressing the image registration requirements for specific tasks. This method employs an initial warping followed by the extraction of macroscopic features from multiple images, addressing the inability of conventional registration methods to register multi-source images across scales, high-angle, and point-ambiguity conditions.
[0007] In order to achieve the above object, the technical solution adopted by the present invention is:
[0008] A method for registering lunar orbiter images with rover navigation camera images comprises the following steps:
[0009] Step (1) using the binocular image of the rover navigation camera to reconstruct the lunar surface point cloud model, calculate the lunar surface plane equation and projection transformation model parameters, and realize the lunar surface orthographic projection transformation of a single rover navigation camera image;
[0010] Step (2) stitching the sequential lunar surface orthographic projection images based on the single rover navigation camera lunar surface orthographic projection image and the DFM registration algorithm with the preselected area, to obtain the rover navigation camera lunar surface orthographic projection stitching image;
[0011] Step (3) automatically extracting lunar impact craters based on the lunar orbiter image and the lunar surface orthographic projection splicing image of the rover navigation camera through mean shift filtering, threshold segmentation, and morphological processing, and obtaining the matching relationship of the lunar impact craters by finding the affine transformation with the minimum error to obtain the affine transformation matrix;
[0012] Step (4) finally realizes the registration of the lunar orbiter image and the rover navigation camera image according to the affine transformation matrix, locates the navigation camera, and compares it with the actual position to indirectly evaluate the registration accuracy.
[0013] Furthermore, in step (1), the method for implementing the lunar surface orthographic projection transformation of a single rover navigation camera image includes:
[0014] First, the coordinate system of the left-eye camera of the rover navigation camera is defined as the world coordinate system. In the case that no prior camera parameters are provided for the lunar scene, the camera intrinsic parameters are initialized using the COLMAP sparse reconstruction method, and the final camera intrinsic parameters are fitted in combination with the BundleAdjustment optimization process. Based on the camera intrinsic parameters and the defined world coordinate system, the lunar plane equation is calculated using 3D reconstruction. Then, the coordinates of the four vertices of a single rover navigation camera image in the lunar plane coordinate system are calculated. Based on the coordinate correspondence between the four vertices of the single rover navigation camera image in the single rover navigation camera image coordinate system and the lunar plane coordinate system, the single rover navigation camera image is orthographically projected.
[0015] The three-dimensional reconstruction is performed on each set of binocular images of the rover navigation camera, and the lunar surface plane is segmented from the sparse point cloud using the RANSAC method. The lunar surface plane equation is expressed as:
[0016] π1x c +π2y c +π3z c +π4w=0
[0017] Among them, X c =(x c ,y c ,z c ,w) T is the homogeneous coordinate of the space point in the reference coordinate system, π=(π1,π2,π3,π4) T are the homogeneous coordinates of the plane;
[0018] Calculate the position coordinates of the four vertices of a single patrol navigation camera image in the coordinate system of the patrol navigation camera's left eye camera, and the coordinates X of each point m in a single patrol navigation camera image in the camera coordinate system. c The conversion relationship is:
[0019]
[0020] Among them, z c is the depth, m=(u,v,1) T is the homogeneous coordinate of the pixel in the image coordinate system, is the camera internal parameter; f x With f y is the scale factor of the CCD camera in the u-axis and v-axis directions, (u o ,v o ) T is the principal point of the CCD camera, Denoted as P;
[0021] The corresponding relationship between the lunar plane equation and the image coordinates of the point in a single rover navigation camera image is established, and the following equation is obtained:
[0022]
[0023] z c Move to the left side of the equation. The right side of the equation after the shift is all known quantities, and the calculation results are:
[0024]
[0025] Finally, we get the point X=(x,y,z) in the coordinate system of the left eye camera of the patrol navigation. T The coordinates are:
[0026]
[0027] Where w is the parameter of the homogeneous coordinate.
[0028] After obtaining the positions of the four vertices of a single rover navigation camera image corresponding to the lunar plane, the point at the upper left corner of the single rover navigation camera image when the vertices correspond to the lunar plane is used as the new image pixel coordinate origin, and the point at the upper right corner of the single rover navigation camera image when the vertices correspond to the lunar plane is fixed as the width of the desired lunar orthographic projection image. After obtaining the lengths of the four sides of the shooting range of the single rover navigation camera image on the lunar plane from the spatial coordinates of the four vertices of the single rover navigation camera image, the pixel coordinates of the four vertices of the single rover navigation camera image corresponding to the lunar plane can be obtained according to the Pythagorean theorem, thereby calculating a projection transformation model for transforming from the single rover navigation camera image coordinate system to the lunar orthographic projection image coordinate system; the single rover navigation camera image can obtain the lunar orthographic projection image of the rover navigation camera image through the projection transformation model.
[0029] Furthermore, in step (2), stitching the sequential lunar orthographic projection images based on the DFM registration algorithm introducing the preselected area includes:
[0030] The true value of the pre-selected area is introduced to screen the initial matching; in the matching process of each stitching, the newly added image is used as the reference image to match the image used as the reference image in the previous stitching in the stitching image; in the stitching process, each image that needs to extract key points has not undergone a second projection transformation and is in a state of minimal deformation. The difference between the two images is minimal, which is most conducive to feature extraction and matching. The error caused by each stitching is not directly accumulated in the subsequent stitching process, and a more reliable lunar surface orthographic projection stitching image is finally obtained.
[0031] Furthermore, in step (3), lunar impact craters are automatically extracted by mean shift filtering, threshold segmentation, and morphological processing, and the matching relationship of the impact craters is obtained by finding an affine transformation with the minimum error, thereby obtaining an affine transformation matrix, including:
[0032] First, mean shift filtering is performed on the rover navigation camera lunar surface orthographic projection mosaic image and the lunar orbiter image respectively, and pixels with similar grayscale values in the image are clustered; secondly, the image after mean shift filtering is threshold segmented in local areas, and noise is removed by corrosion dilation. The lunar surface impact crater contours are found on the binary image after noise removal; then, the center coordinates of the lunar surface impact craters are obtained, and an ellipse is fitted to each contour according to the extracted lunar surface impact crater contours, and the center of the ellipse is regarded as the center of the lunar surface impact crater; after the projection transformation in step (1), the relationship between the rover navigation camera lunar surface orthographic projection mosaic image and the lunar orbiter image is only an affine transformation within the plane; based on the similarity of geometric shapes, the lunar surface impact craters in the rover navigation camera lunar surface orthographic projection mosaic image and the lunar surface impact craters in the orbiter image are randomly matched, and the affine transformation matrix is calculated. The affine transformation with the smallest error is found as the affine transformation matrix for aligning the rover navigation camera lunar surface orthographic projection mosaic image and the lunar orbiter image.
[0033] Furthermore, the step (4) specifically includes:
[0034] According to the affine transformation matrix obtained in step (3), the lunar orbiter image is enlarged to a scale similar to the rover navigation camera's lunar surface orthographic projection mosaic image, and the orthographic projection image is aligned to the lunar orbiter image through affine transformation, and the image data of the original lunar orbiter image in the overlapping area is directly replaced to obtain the alignment result; before the transformation, the preselected area of the rover navigation camera's lunar surface orthographic projection mosaic image is threshold segmented, and the center of the hollow circle in the middle part of the image is fitted as the position of the rover navigation camera; in the image coordinate system of the rover navigation camera's lunar surface orthographic projection mosaic image before affine transformation, a line segment starting from the center and pointing directly upward is taken as the initial orientation of the rover navigation camera, and after affine transformation, the aligned navigation camera position and orientation are marked in the lunar orbiter image; by comparing the image released by NASA with the lunar orbiter image, the real position coordinates of the rover navigation camera are marked in the lunar orbiter image, and the rover navigation camera coordinates located in the alignment process are compared with the real position coordinates to indirectly evaluate the alignment accuracy.
[0035] The advantages of the present invention compared with the prior art are:
[0036] (1) This paper designs a multi-source image registration framework for the registration task of lunar orbiter images and rover navigation camera images. To address the registration difficulties caused by cross-scale and large inclination between lunar orbiter images and rover navigation camera images, the paper uses techniques such as image orthographic projection, image stitching, and registration based on the distribution relationship of impact crater landmarks to reconstruct the lunar surface point cloud using triangulation and output an orthographic image of the lunar surface. It also uses a DFM with pre-selected regions to perform feature extraction and matching of adjacent orthographic images of the lunar surface, and uses a recursive stitching strategy to output an orthographic image of the lunar surface taken by the rover navigation camera. It also extracts and locates impact landmarks based on mean shift filtering, and outputs the registered image and the position of the navigation camera.
[0037] (2) This paper designs an evaluation metric for indirectly verifying the registration accuracy of lunar orbiter images and navigation camera images. The registration accuracy is qualitatively and quantitatively evaluated by calculating the position coordinates of the navigation camera. Experimental results show that the registration of lunar orbiter images and navigation camera images can be achieved effectively when the positioning accuracy of the navigation camera is controlled to a few pixels.
[0038] In summary, under the premise that conventional registration methods fail, the method adopted in the present invention effectively solves the registration problems caused by cross-scale, large inclination, point blur, etc. between the lunar orbiter image and the rover navigation camera image. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a flow chart of a method for registering lunar orbiter images with rover navigation camera images according to the present invention. DETAILED DESCRIPTION
[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] like Figure 1 As shown, the specific implementation steps of the method for aligning lunar orbiter images with rover navigation camera images of the present invention are as follows:
[0042] Step 1: Use the binocular image of the rover navigation camera to reconstruct the lunar surface point cloud model, calculate the lunar surface plane equation and projection transformation model parameters, and realize the lunar surface orthographic projection transformation of a single rover navigation camera image.
[0043] First, the coordinate system of the rover navigation camera's left eye is defined as the world coordinate system. The coordinates in this coordinate system are independent of their real-world dimensions. In the absence of prior camera parameters for the lunar scene, the COLMAP sparse reconstruction method is used to initialize the camera's intrinsic parameters, and the final camera intrinsic parameters are fitted using the Bundle Adjustment optimization process. Based on the camera's intrinsic parameters and the defined world coordinate system, the lunar plane equation is calculated using 3D reconstruction. The coordinates of the four vertices of the rover navigation camera image in the lunar plane coordinate system are then calculated. The rover navigation camera image is orthographically projected based on the corresponding coordinate relationship between the four vertices of the rover navigation camera image in the coordinate system of a single rover navigation camera image and the lunar plane coordinate system.
[0044] The three-dimensional reconstruction is performed on each set of binocular images of the rover navigation camera, and the lunar surface plane is segmented from the sparse point cloud using the RANSAC method. The lunar surface plane equation is expressed as:
[0045] π1x c +π2y c +π3z c +π4w=0
[0046] Among them, X c =(x c ,y c ,z c ,w) T is the homogeneous coordinate of the space point in the reference coordinate system, π=(π1,π2,π3,π4) T are the homogeneous coordinates of the plane.
[0047] Calculate the position coordinates of the four vertices of a single patrol navigation camera image in the coordinate system of the patrol navigation camera's left eye camera, and the coordinates X of each point m in a single patrol navigation camera image in the camera coordinate system. c The conversion relationship is:
[0048]
[0049] Among them, z c is the depth, m=(u,v,1) T is the homogeneous coordinate of the pixel in the image coordinate system, is the camera intrinsic parameter. x With f y is the scale factor of the CCD camera in the u-axis and v-axis directions, (u0,v0) T is the principal point of the CCD camera, Denoted as P.
[0050] Traditionally, when the camera intrinsic parameters and image coordinate positions are known, calculating the spatial coordinates of the four vertices of a single rover navigation camera image in the coordinate system of the rover navigation left-eye camera also requires the depth information of the vertices. Here, the depth information is unknown, but by constraining it with the lunar plane equation in the coordinate system of the rover navigation left-eye camera, the coordinate position in the coordinate system of the rover navigation left-eye camera can still be calculated. A correspondence is established between the lunar plane equation and the image coordinates of the point in a single rover navigation camera image, and the following equation is obtained jointly:
[0051]
[0052] z c Move to the left side of the equation. The right side of the equation after the shift is all known quantities, which can be calculated:
[0053]
[0054] Finally, we get the point X=(x,y,z) in the coordinate system of the left eye camera of the patrol navigation. T The coordinates are:
[0055]
[0056] Where w is the parameter of the homogeneous coordinate.
[0057] After determining the positions of the four vertices of a single rover navigation camera image on the lunar plane, the point at the upper left corner of the image's vertices when it lies on the lunar plane is used as the new image pixel coordinate origin. The point at the upper right corner of the image's vertices when it lies on the lunar plane is fixed as the desired width of the lunar orthographic projection image. Using the spatial coordinates of the four vertices of the single rover navigation camera image, the lengths of the four sides of the image's shooting range on the lunar plane are determined. The Pythagorean theorem is then used to determine the pixel coordinates of the four vertices of the single rover navigation camera image on the lunar plane, thereby calculating a projection transformation model from the single rover navigation camera image coordinate system to the lunar orthographic projection image coordinate system. Using this projection transformation model, the lunar orthographic projection image of the rover navigation camera image can be obtained.
[0058] Step 2: Based on the single rover navigation camera lunar orthographic projection image, a DFM registration algorithm with a preselected area is used to stitch the lunar orthographic projection images to obtain a rover navigation camera lunar orthographic projection stitching image.
[0059] Due to a single projection transformation, the lunar orthographic image has irregular boundaries. Black pixels are used to fill the areas outside these boundaries. The black-filled areas and the pixels where they meet the lunar orthographic image boundaries form large gradients, significantly impacting nearly all feature extraction and matching methods. During stitching, the DFM registration method is used for feature extraction and matching. Due to the influence of the black regions, the first stage of the two-stage DFM registration algorithm is often not initialized correctly, resulting in incorrect initial distortions as the matching of the connecting areas progresses. To address this issue, a preselected region ground truth is introduced to filter the initial matches. During each stitching process, the newly added image serves as the reference image and is matched against the image used as the reference image in the previous stitching. Under this premise, each image requiring keypoint extraction during the stitching process has not yet undergone the second projection transformation, is in a state of minimal deformation, and the difference between the two images is minimized, which is optimal for feature extraction and matching. Furthermore, errors introduced during each stitching process are not directly accumulated in subsequent stitching processes. The result is a more reliable lunar orthographic image.
[0060] Step 3: Based on the lunar orbiter image and the lunar surface orthographic projection stitching image of the rover navigation camera, lunar impact craters are automatically extracted through mean shift filtering, threshold segmentation, and morphological processing. The matching relationship of the lunar impact craters is obtained by finding the affine transformation with the minimum error, and the affine transformation matrix is obtained.
[0061] First, mean-shift filtering is performed on the rover navigation camera's orthographic projection mosaic image and the lunar orbiter image, clustering pixels with similar grayscale values. Next, threshold segmentation is performed on the mean-shift filtered image, and noise is removed using erosion and dilation. The lunar crater outlines are then found on the de-noised binary image. Next, the center coordinates of the lunar craters are determined. Based on the extracted crater outlines, an ellipse is fitted to each outline, with the center of the ellipse being considered the crater center. After the projective transformation in step 1, the relationship between the rover navigation camera's orthographic projection mosaic image and the lunar orbiter image is solely an in-plane affine transformation. Based on geometric similarity, the lunar craters in the rover navigation camera's orthographic projection mosaic image are randomly matched with those in the orbiter image. The affine transformation matrix is calculated, and the affine transformation with the minimum error is used as the affine transformation matrix for registering the rover navigation camera's orthographic projection mosaic image with the lunar orbiter image.
[0062] Step 4: Based on the affine transformation matrix, the lunar orbiter image and the rover navigation camera image are finally registered, the navigation camera is positioned, and the actual position is compared with the image released by NASA to indirectly evaluate the registration accuracy.
[0063] Using the affine transformation matrix calculated in step 3, the lunar orbiter image is enlarged to a scale similar to the rover navigation camera's orthographic projection mosaic image. The orthographic projection image is then registered to the lunar orbiter image via affine transformation, directly replacing the original lunar orbiter image data in the overlapping region to obtain the registration result. Before the transformation, a threshold segmentation is performed on a preselected region of the rover navigation camera's orthographic projection mosaic image. The center of the hollow circle in the center of the image is fitted as the position of the rover navigation camera. In the image coordinate system of the rover navigation camera's orthographic projection mosaic image before the affine transformation, a line segment extending from the center to the top is taken as the initial orientation of the rover navigation camera. After the affine transformation, the registered navigation camera position and orientation are marked in the lunar orbiter image. By comparing the NASA-released image with the lunar orbiter image, the true position coordinates of the rover navigation camera are annotated in the lunar orbiter image. The rover navigation camera coordinates determined by the registration process are compared with the true position coordinates to indirectly evaluate the registration accuracy.
[0064] As shown in Table 1, the proposed method is quantitatively compared with existing image registration methods on real lunar orbiter imagery and rover navigation camera imagery data. Four representative algorithms were selected: SIFT, R2D2, SuperPoint+SuperGlue, and DFM. Existing image registration methods fail when directly applied to the current task. However, the proposed method effectively achieves registration of lunar orbiter and navigation camera images while maintaining the navigation camera's positioning accuracy within a few pixels.
[0065] Table 1
[0066]
[0067] Among them, the reason why some registration accuracies are empty in the table is that the four existing image registration methods cannot match correctly and therefore the registration accuracy cannot be calculated.
[0068] The present invention addresses the issues of multi-scale, high-angle, and point-ambiguous multi-source image registration between two types of heterogeneous images, as well as the limited field of view of a single rover navigation camera image. This method provides a method for aligning lunar orbiter images with rover navigation camera images, thereby meeting the image registration requirements for specific missions. Experiments have shown that the present invention effectively addresses image registration problems under specific mission conditions where conventional image registration methods are difficult to apply. By indirectly evaluating the registration accuracy through positioning the navigation camera, the positioning accuracy of the navigation camera can be controlled to within an error of a few pixels. The present invention addresses the complex multi-source image registration problem of aligning lunar orbiter images with rover navigation camera images, which involves multi-scale, high-angle, and point-ambiguous multi-source image registration. Research on this problem will help achieve autonomous positioning, navigation, and global path planning for lunar rovers in future lunar exploration missions, and has very important scientific significance for manned lunar landings.
[0069] The contents not described in detail in the specification of the present invention belong to the common knowledge of professionals in this field.
[0070] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, and it should be clear that the present invention is not limited to the scope of the specific embodiments, it is obvious to those skilled in the art that as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.
Claims
1. A method for registering lunar orbiter images with rover navigation camera images, characterized in that: The following steps are involved: Step (1) using the binocular image of the rover navigation camera, reconstructing the lunar surface point cloud model, calculating the lunar surface plane equation and projection transformation model parameters, and realizing the lunar surface orthographic projection transformation of a single rover navigation camera image; Step (2) performing sequential lunar surface orthographic projection image stitching based on the single rover navigation camera image and the DFM registration algorithm with the preselected area, to obtain a rover navigation camera lunar surface orthographic projection stitching image; Step (3) automatically extracting lunar impact craters based on the lunar orbiter image and the lunar surface orthographic projection splicing image of the rover navigation camera through mean shift filtering, threshold segmentation, and morphological processing, and obtaining the matching relationship of the lunar impact craters by finding the affine transformation with the minimum error to obtain the affine transformation matrix; Step (4) finally realizes the registration of the lunar orbiter image and the rover navigation camera image based on the affine transformation matrix, locates the navigation camera, and compares it with the actual position to indirectly evaluate the registration accuracy.
2. The method for registering lunar orbiter images with rover navigation camera images according to claim 1, characterized in that: In step (1), the method for implementing the lunar surface orthographic projection transformation of a single rover navigation camera image includes: First, the coordinate system of the left-eye camera of the rover navigation camera is defined as the world coordinate system. In the case that no prior camera parameters are provided for the lunar scene, the camera intrinsic parameters are initialized using the COLMAP sparse reconstruction method, and the final camera intrinsic parameters are fitted in combination with the Bundle Adjustment optimization process. Based on the camera intrinsic parameters and the defined world coordinate system, the lunar plane equation is calculated using 3D reconstruction. Then, the coordinates of the four vertices of a single rover navigation camera image in the lunar plane coordinate system are calculated. Based on the coordinate correspondence between the four vertices of the single rover navigation camera image in the single rover navigation camera image coordinate system and the lunar plane coordinate system, the single rover navigation camera image is orthographically projected. The three-dimensional reconstruction is performed on each set of binocular images of the rover navigation camera, and the lunar surface plane is segmented from the sparse point cloud using the RANSAC method. The lunar surface plane equation is expressed as: in, are the homogeneous coordinates of the space point in the reference coordinate system, are the homogeneous coordinates of the plane; Calculate the position coordinates of the four vertices of a single patrol navigation camera image in the coordinate system of the patrol navigation camera's left eye camera. Each point in a single patrol navigation camera image Coordinates in the camera coordinate system The conversion relationship is: in, is the depth, is the homogeneous coordinate of the pixel in the image coordinate system, Is the camera internal parameter and Is the CCD camera in Axis and The scale factor in the axial direction, is the principal point of the CCD camera, Denoted as P; The corresponding relationship between the lunar plane equation and the image coordinates of the point in a single rover navigation camera image is established, and the following equation is obtained: Will Move to the left side of the equation. The right side of the equation after the shift is all known quantities, and the calculation results are: Finally, the point in the left eye camera coordinate system of the patrol navigation is obtained The coordinates are: in, are the parameters of homogeneous coordinates; After obtaining the positions of the four vertices of a single rover navigation camera image corresponding to the lunar plane, the point at the upper left corner of the single rover navigation camera image when the vertices correspond to the lunar plane is used as the new image pixel coordinate origin, and the point at the upper right corner of the single rover navigation camera image when the vertices correspond to the lunar plane is fixed as the width of the desired lunar orthographic projection image. After obtaining the lengths of the four sides of the shooting range of the single rover navigation camera image on the lunar plane from the spatial coordinates of the four vertices of the single rover navigation camera image, the pixel coordinates of the four vertices of the single rover navigation camera image corresponding to the lunar plane can be obtained according to the Pythagorean theorem, thereby calculating a projection transformation model for transforming from the single rover navigation camera image coordinate system to the lunar orthographic projection image coordinate system; the single rover navigation camera image can obtain the lunar orthographic projection image of the rover navigation camera image through the projection transformation model.
3. The method for registering lunar orbiter images with rover navigation camera images according to claim 2, characterized in that: In step (2), the sequential lunar orthographic projection image stitching based on the DFM registration algorithm introducing the pre-selected area includes: The true value of the pre-selected area is introduced to screen the initial matching; in the matching process of each stitching, the newly added image is used as the reference image to match the image used as the reference image in the previous stitching in the stitching image; in the stitching process, each image that needs to extract key points has not undergone a second projection transformation and is in a state of minimal deformation. The difference between the two images is minimal, which is most conducive to feature extraction and matching. The error caused by each stitching is not directly accumulated in the subsequent stitching process, and a more reliable lunar surface orthographic projection stitching image is finally obtained.
4. The method for registering lunar orbiter images with rover navigation camera images according to claim 3, characterized in that: In step (3), lunar impact craters are automatically extracted by mean shift filtering, threshold segmentation, and morphological processing, and the matching relationship of the impact craters is obtained by finding the affine transformation with the minimum error, and the affine transformation matrix is obtained, including: First, mean shift filtering is performed on the rover navigation camera lunar surface orthographic projection mosaic image and the lunar orbiter image respectively, and pixels with similar grayscale values in the image are clustered; secondly, the image after mean shift filtering is threshold segmented in local areas, and noise is removed by corrosion dilation. The lunar surface impact crater contours are found on the binary image after noise removal; then, the center coordinates of the lunar surface impact craters are obtained, and an ellipse is fitted to each contour according to the extracted lunar surface impact crater contours, and the center of the ellipse is regarded as the center of the lunar surface impact crater; after the projection transformation in step (1), the relationship between the rover navigation camera lunar surface orthographic projection mosaic image and the lunar orbiter image is only an affine transformation within the plane; based on the similarity of geometric shapes, the lunar surface impact craters in the rover navigation camera lunar surface orthographic projection mosaic image and the lunar surface impact craters in the orbiter image are randomly matched, and the affine transformation matrix is calculated. The affine transformation with the smallest error is found as the affine transformation matrix for aligning the rover navigation camera lunar surface orthographic projection mosaic image and the lunar orbiter image.
5. The method for registering lunar orbiter images with rover navigation camera images according to claim 4, characterized in that: The step (4) specifically includes: According to the affine transformation matrix obtained in step (3), the lunar orbiter image is enlarged to a scale similar to the rover navigation camera's lunar surface orthographic projection mosaic image, and the orthographic projection image is aligned to the lunar orbiter image through affine transformation, directly replacing the image data of the original lunar orbiter image in the overlapping area to obtain the alignment result; before the transformation, the pre-selected area of the rover navigation camera's lunar surface orthographic projection mosaic image is threshold segmented, and the center of the hollow circle in the middle part of the image is fitted as the position of the rover navigation camera; in the image coordinate system of the rover navigation camera's lunar surface orthographic projection mosaic image before affine transformation, a line segment starting from the center and pointing directly upward is taken as the initial orientation of the rover navigation camera, and after affine transformation, the aligned navigation camera position and orientation are marked in the lunar orbiter image; by comparing the image released by NASA with the lunar orbiter image, the real position coordinates of the rover navigation camera are marked in the lunar orbiter image, and the rover navigation camera coordinates located in the alignment process are compared with the real position coordinates to indirectly evaluate the alignment accuracy.