A lunar surface cross-view absolute positioning method based on multi-source images

Abstract

The application provides a lunar surface cross-view absolute positioning method based on multi-source images, comprising: (1) respectively extracting orbit imager and lunar rover navigation camera image features, projecting the lunar rover navigation camera image features to a top view angle through geometric correspondence; (2) for the features extracted and transformed in (1), combining the orbit imager semantic map to enhance the edges, and outputting the synthesized top view features by a cross-view feature transfer module; (3) for the orbit imager features and the synthesized top view features with edge enhancement in (2), calculating the spatial correlation to solve the positioning result; (4) for the positioning result in (3), applying a triplet loss to supervise the position and a perception loss to supervise the feature similarity in the optimization process; (5) for the result optimized in (4), the absolute position solution in the orbit imager can be realized for any lunar rover navigation camera image. The application is suitable for the lunar surface cross-view absolute positioning with large view angle, scale and illumination change.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of lunar surface visual positioning, and particularly relates to a lunar surface cross-view absolute positioning method based on multi-source images. BACKGROUND

[0002] Lunar surface visual positioning is a navigation positioning technology and computer vision technology, aiming to input camera sensor data combined with terrain and map information to determine the accurate position of a lunar rover on the lunar surface. Precise lunar rover positioning enables the device to accurately navigate on the lunar surface, perform scientific tasks, and effectively interact with scientific target points, providing the necessary positioning basis for scientific research and task execution.

[0003] In previous exploration missions, lunar surface positioning mainly adopts relative positioning and absolute positioning strategies. Relative positioning methods estimate the displacement of the lunar rover relative to the starting position through dead reckoning, visual odometry, laser positioning, etc. However, these methods are usually limited by cumulative errors or require high-precision equipment support, increasing the difficulty of implementation. Absolute positioning methods include celestial navigation, radio navigation, and map matching, etc., which obtain the absolute position of the probe through celestial observation, signal reception, or map information, facing the challenges of improving accuracy and resisting interference. In future exploration missions, the lunar rover's lunar surface roaming pursues higher autonomy and safety, which puts forward urgent needs for more portable and accurate lunar surface positioning.

[0004] Vision-based positioning technology takes airborne camera image data as the input of the algorithm, and the configuration is relatively simple. However, in vision-based lunar absolute positioning, strict restrictions on imaging angles, ground feature conditions, etc. are required, and there are technical difficulties in matching steps between orbital imager images and lunar rover navigation camera images. Currently, the matching positioning is mainly completed by manual marking of feature correspondence by ground operators, and the positioning performance is highly dependent on complete surface markers and high-resolution orbital imager images. Therefore, this method still faces challenges for future exploration missions, and researching a lunar cross-view absolute positioning method based on multi-source images is of great significance for further optimization of lunar surface visual positioning technology.

[0005] For the research of lunar cross-view absolute positioning, the deficiencies of existing technologies mainly manifest in the following aspects: (1) There are large viewing angles, scales, and illumination changes between lunar rover navigation camera images and orbital imager images, and existing technologies are difficult to cope with the feature descriptor similarity failure problem, making it difficult to achieve absolute positioning of the lunar rover in the orbital imager images; (2) Existing technologies have high requirements for visual input, relying on complete impact crater markers and high-resolution orbital imager images, which may lead to autonomous positioning failure, increasing the complexity of positioning under artificial intervention and reducing the efficiency of the algorithm. SUMMARY

[0006] In order to overcome the prior art, the present application provides a lunar surface cross-view absolute positioning method based on multi-source images, designs a feature extraction and description method with more similarity between multi-source images, weakens the input image restriction condition, reduces the artificial participation degree in the positioning process, and realizes more robust high-precision lunar surface cross-view absolute positioning under the conditions of large viewing angle, scale and illumination change.

[0007] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0008] A lunar surface cross-view absolute positioning method based on multi-source images comprises the following steps:

[0009] Step (1) extracts the features of the orbiter image and the lunar rover navigation camera image respectively, and projects the features of the lunar rover navigation camera image to the overhead view angle through the geometric correspondence relationship;

[0010] Step (2) for the features extracted and transformed in step (1), the edge of the orbiter semantic map is enhanced, the edge-enhanced orbiter image features are obtained, and the synthesized overhead view features are output by the cross-view feature transfer module;

[0011] Step (3) for the edge-enhanced orbiter image features and the synthesized overhead view features of step (2), the spatial correlation is calculated, and the positioning result is solved;

[0012] Step (4) for the positioning result of step (3), the ternary loss supervision position and the perception loss supervision feature similarity are applied in the optimization process;

[0013] Step (5) for the result optimized in step (4), the absolute position solution in the orbiter image is realized for any lunar rover navigation camera image, and the lunar surface cross-view absolute positioning under the conditions of large viewing angle, scale and illumination change is obtained.

[0014] Compared with the prior art, the present application has the following beneficial effects:

[0015] (1) The present application proposes a neural map matching framework for end-to-end training of lunar surface scenes, and the goal is to estimate the absolute position of the lunar rover navigation camera image in the orbiter image. A single lunar rover navigation camera image and an orbiter image are input, and only the appearance of the part of the terrain landmark such as the impact crater in the field of view is required, the restriction conditions such as resolution, number of landmarks contained and completeness of the input data of the existing lunar surface absolute positioning method are weakened, and the algorithm application range is effectively expanded without relying on radar, point cloud or sequence image and other complex 3D information prompts, and the artificial participation degree in the data screening and map matching link is reduced.

[0016] (2) This invention improves the feature point matching process in previous studies by extracting the true contour of the impact crater through the semantic map of the orbiter, suppressing the false contour of the shadow area of ​​the impact crater, and transferring cross-view texture information in the two feature spaces to improve feature similarity. This effectively solves the problem of feature descriptor similarity failure caused by changes in viewpoint, scale and illumination, and significantly improves positioning accuracy and algorithm stability.

[0017] In summary, the method employed in this invention is simple in principle and can achieve high-precision absolute positioning of the lunar surface across views under multi-source image input. Attached Figure Description

[0018] Figure 1 This is a flowchart of a lunar surface cross-view absolute positioning method based on multi-source images according to the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0020] like Figure 1 As shown, the present invention provides a lunar surface cross-view absolute localization method based on multi-source images, comprising the following steps:

[0021] Step 1: Construct a feature extraction network to extract features and project the lunar view onto a top-down view based on geometric relationships, including:

[0022] The multi-source image input for lunar surface cross-view absolute localization includes orbiter images and rover navigation camera images. To effectively process cross-view images, a feature extraction network is constructed to perform detailed feature analysis on the acquired images. The network backbone adopts a U-Net architecture. The encoder part uses VGG16 pre-trained on the ImageNet dataset to ensure strong feature recognition capabilities, while the decoder part uses randomly initialized weights to adapt to specific feature reconstruction needs and promote joint feature learning from multiple source images, thereby capturing more accurate and similar features.

[0023] The feature extraction network extracts feature maps at scales of 1 / 2, 1 / 4, and 1 / 8 from the orbiter images and the lunar rover navigation camera images, respectively. These feature maps from the orbiter images are denoted as sat2, sat4, and sat8, and those from the lunar rover navigation camera images are denoted as moon2, moon4, and moon8. This multi-scale feature extraction method ensures that the network captures different levels of detail in the images, thus providing richer and more accurate information for subsequent cross-view absolute localization.

[0024] After extracting the features from the lunar rover's navigation camera image, the features are projected from the lunar rover's navigation camera viewpoint to the top-down viewpoint using the geometric correspondence module with relative rotation and translation operations.

[0025] The origin of the world coordinate system is defined as being located at the center of the orbiter image, and the Z-axis of the world coordinate system is defined as follows: w Axis and X w The axes are parallel to the u coordinate system of the orbiter image. s axis, v s The Y-axis of the world coordinate system w The axis is along the direction of gravity; the coordinate system X of the lunar rover's navigation camera is defined. c Y c Z c Its initial position coincides with the world coordinate system.

[0026] Point P in the world coordinate system w =(x w ,y w ,z w ) and orbiter image coordinates P usvs =[u s ,v s ,1] T There are:

[0027]

[0028] Where p is the spatial resolution of the orbiter image pixels, set to 1.6m for practical reference, (u s0 ,v s0 ) is the center of the orbiter image, and the superscript T indicates the transpose of the matrix.

[0029] Further, based on relative rotation and translation, as well as the intrinsic parameter K of the lunar rover's navigation camera, the image coordinates P of the lunar rover's navigation camera are obtained. umvm =[u m ,v m ,1] T and orbiter image coordinates P usvs =[u s ,v s ,1]T The geometric correspondence between them is:

[0030]

[0031] where z is the depth in the rover navigation camera coordinate system, and is obtained by zP umvm The last dimension value of P umvm is normalized to obtain P cw R cw is the rotation matrix for transforming the vector in the world coordinate system to the rover navigation camera coordinate system, and t cw is the value of the vector from the origin of the rover navigation camera coordinate system to the origin of the world coordinate system in the rover navigation camera coordinate system:

[0032]

[0033] where θ is the rotation angle of the rover navigation camera around the Y c axis, φ is the rotation angle around the X c axis, and h is the height of the rover navigation camera from the lunar surface.

[0034] A top-view feature grid is constructed, and each coordinate value can be indexed to the coordinate in the rover navigation camera image coordinate system based on the cross-view geometric correspondence relationship derived earlier. Through bilinear interpolation, the projection feature maps moon2sat2, moon2sat4, and moon2sat8 of the top-view perspective are generated, providing key information for high-precision positioning.

[0035] Step 2, combine the orbiter semantic map to enhance the edge, and output the synthesized top-view feature using the cross-view feature transfer module, including:

[0036] First, the orbiter semantic map is made according to the orbiter image, and after labeling the real outline of the impact crater, it is also sent to the backbone network to extract multi-scale features edge2, edge4, and edge8. Then, using a 1x1 convolution kernel Conv 1×1 adjust the number of feature channels to 1, and use the sigmoid function σ to normalize to obtain the edge weight. The orbiter image features extracted in step (1) are weighted using the edge weight, and then connected with the original orbiter image feature residual to obtain the edge-enhanced orbiter image feature maps edge_sat2, edge_sat4, and edge_sat8 as follows:

[0037]

[0038] where, represents the addition of corresponding position elements of the feature map, represents the multiplication of corresponding position elements of the feature map, Conv 1×1denotes a 1x1 convolution kernel, k denotes the index, k = 2, 4, 8;

[0039] To further reduce the feature domain difference, a cross-view feature transfer module containing relevance embedding (RE), hard attention (HA) for feature transfer and soft attention (SA) for feature synthesis is designed to transfer low-level texture features from the edge-enhanced orbiter image features to the overhead perspective projection feature map, guiding the model to learn a more powerful feature representation in the overhead view. The input of the cross-view feature transfer module is the projection feature map moon2sat8 in the overhead view and the edge-enhanced orbiter image feature map edge_sat8:

[0040] Q = moon2sat8, K = edge_sat8, V = edge_sat8

[0041] Wherein, the extracted texture features Q (query), K (key) and V (value) represent the three basic elements of the transformer attention mechanism and will be further used in the relevance embedding. The texture relevance of the overhead perspective projection feature map and the edge-enhanced orbiter image feature map is modeled, which estimates the similarity between Q and K to embed the relevance between the two view features. Q and K are expanded into patches q i and k j The relevance between patches is calculated by normalized inner product:

[0042]

[0043] Wherein, r i,j represents the relevance embedding between the two feature maps, ||·|| represents the Euclidean norm of the vector, is the square root of the sum of the squares of all elements in the vector, and i and j represent the patch indices of Q and K expansion respectively.

[0044] For each query position, the hard attention mechanism only searches for the most relevant position from V for feature transfer:

[0045]

[0046] Wherein, h i The value can be regarded as an index, which represents the position in the edge-enhanced orbiter image feature map that is most relevant to the i-th position feature in the overhead perspective projection feature map, and argmax represents the index of selecting the maximum item from a set of numerical values.

[0047] In order to obtain the transferred texture feature T from the edge-enhanced orbiter image feature map, the hard attention map index V is used to expand the block, which is denoted as:

[0048]

[0049] where t i denotes the value of the texture feature T at the i-th position, which is selected from the h-th i position of the texture feature V, denotes the feature value of the h-th i position selected from V. Soft attention mechanism is used to fuse the transferred texture feature T and the overhead perspective projection feature map. First, the transferred texture feature T is combined with the overhead perspective projection feature map to make full use of the information in the original overhead perspective projection feature map, and the soft attention map is used to measure the confidence of transferring features from this position:

[0050]

[0051] where s i denotes the i-th position of the soft attention map S, and the final output F out8 denotes the synthesized overhead view feature:

[0052]

[0053] where Concat denotes the feature fusion by concatenating two features.

[0054] In order to improve memory efficiency, the cross-view feature transfer is only applied to the coarsest feature level (i.e. 1 / 8 of the original image size). This also ensures a large receptive field of cross-attention, and a decoder is used to recover finer details of the overhead view at higher resolution, obtaining multi-scale synthesized overhead view features F out2 , F out4 , and F out8 .

[0055] Step 3, calculate the spatial correlation and analyze the position possibility, including:

[0056] When processing multi-scale feature maps, each layer of feature maps is analyzed, and the spatial correlation between the synthesized overhead view feature and the edge-enhanced orbiter image feature is utilized. By sliding the synthesized overhead view on the edge-enhanced orbiter image feature map and performing convolution and normalized cross-correlation operation, the correlation score of each position is calculated:

[0057] score = F outi * edge_sat i

[0058] where * represents normalized cross-correlation, score represents spatial correlation score. Higher score in different positions of the grid indicates that the query camera is more likely to be in that position, so the pixel position with the highest correlation score is recorded as the positioning result, which is actually the absolute positioning result in the current cropped orbiter image. In order to facilitate model training, it is not necessary to convert its coordinates to latitude and longitude during training.

[0059] Step 4, training the model using the positioning result and the synthesized feature similarity, including:

[0060] The loss function of the algorithm is constructed, the first term is the spatial correlation loss function L1, the algorithm adopts the triplet loss, the goal is to maximize the possibility of the lunar rover navigation camera position true value, and at the same time minimize the possibility of other positions:

[0061]

[0062] where, is the total number of pixels, (u sgt ,v sgt ) represents the camera position true value, (u s ,v s ) represents other pixels, and γ is a hyperparameter determined by experiment adjustment to establish the best model performance;

[0063] The second term is the perception loss function L2, which limits the synthesized overhead view features to have similar texture features to the transmitted texture features T, effectively promoting the network to learn joint features of multi-source images:

[0064]

[0065] where C, H, and W represent the channel number, height, and width of the 1 / 8 scale feature map, respectively, and represents the square of the average Euclidean distance.

[0066] Step 5, realizing high-precision lunar surface cross-view absolute positioning under conditions of large viewing angle, scale, and illumination change, including:

[0067] After the model is trained, input the images collected by the lunar rover navigation camera in the scene and the orbiter images within a certain range of the current region, to synthesize as many multi-scale feature maps as possible that are aligned and reliable for cross-view image synthesis. Finally, through correlation analysis, the absolute positioning result in the current cropped orbiter image is obtained, and then it is registered to the lunar fixed coordinate system according to the actual task requirements, realizing the complete lunar surface cross-view absolute positioning.

[0068] Therefore, the application can establish the coordinate geometric correspondence between the multi-source images in view of the lunar rover navigation camera images and the orbital imager images with large viewing angle, illumination and scale variation, combine the orbital semantic map and the cross-view feature transfer module to guide the top view feature synthesis, and realize high-precision lunar surface cross-view absolute positioning.

[0069] The contents not described in detail in the specification of the present application belong to the common knowledge of the skilled in the art. Although the above describes the specific embodiments of the present application in a demonstrative manner, so as to facilitate the understanding of the present application by the skilled in the art, and it should be clear that the present application is not limited to the scope of the specific embodiments, and for the skilled in the art, as long as the various changes are within the spirit and scope of the present application defined and determined by the appended claims, all the application creations utilizing the concept of the present application are within the protection.

Claims

1. A method for lunar surface cross-view absolute localization based on multi-source images, characterized in that, The method comprises the following steps: Step (1) extracts features of the rover navigation camera image and the orbiter image respectively, and projects the features of the rover navigation camera image to the overhead view through geometric correspondence; Step (2) in view of the features extracted and transformed in step (1), combines the orbiter semantic map to enhance the edges, obtains the edge-enhanced orbiter image features, and outputs the synthesized overhead view features through the cross-view feature transfer module; Step (3) in view of the edge-enhanced orbiter image features and the synthesized overhead view features in step (2), calculates the spatial correlation, and solves the positioning result; Step (4) in view of the positioning result in step (3), applies the triplet loss to supervise the position and the perception loss to supervise the feature similarity in the optimization process, including: The loss function of the algorithm is constructed, with the first term being a spatial correlation loss function The algorithm employs a triplet loss, with the goal of maximizing the likelihood of the true value of the rover navigation camera position while minimizing the likelihood of other positions: ； wherein, is the total number of pixels, represents the ground truth camera position, represents other pixels, is a hyperparameter determined by experimental adjustment to build the best model performance; The second term is the perceptual loss function The algorithm limits the synthesized top-view features to have similar texture features as the transmitted texture features T through the perceptual loss, effectively promoting the network to perform joint feature learning of multi-source images: ； where C, H, W represent the channel number, height, and width of the 1 / 8 scale feature map, respectively, denotes the squared average Euclidean distance, the total loss function is interpreted as: ； Step (5) in view of the result optimized in step (4), realizes the absolute position solution in the orbiter image for any rover navigation camera image, and obtains the lunar surface cross-view absolute positioning with large view angle, scale and illumination change.

2. The method of claim 1, wherein, The step (1) comprises: The multi-source image input of the lunar surface cross-view absolute positioning includes an orbiter image and a lunar rover navigation camera image, a feature extraction network is constructed, and detailed feature analysis is performed on the collected images; the network backbone adopts a U-Net architecture, the encoder uses a VGG16 pre-trained on an ImageNet dataset, and the decoder uses randomly initialized weights; the feature extraction network extracts 1 / 2, 1 / 4 and 1 / 8 scale feature maps from the orbiter image and the lunar rover navigation camera image respectively, and the 1 / 2, 1 / 4 and 1 / 8 scale feature maps from the orbiter image are denoted as , , , and the 1 / 2, 1 / 4 and 1 / 8 scale feature maps from the lunar rover navigation camera image are denoted as , , . After extracting the features of the rover navigation camera image, the geometric correspondence module is used to project the features from the rover navigation camera view to the overhead view through relative rotation and translation operation; The origin of the world coordinate system is located at the center of the imager image, and the axis, axis is parallel to the axis, axis of the imager image coordinate system, and the axis of the world coordinate system is along the direction of gravity; the initial position of the lunar rover navigation camera coordinate system , , is coincident with the world coordinate system. Point in world coordinate system And orbiter image coordinates The following relationship exists between them: ； wherein, is the spatial resolution of the orbiter imagery pixels, set to 1.6 m by reference to actual conditions, is the center of the orbiter imagery; Further in accordance with the relative rotation and translation and the lunar rover navigation camera intrinsic parameters , the geometric correspondence between the lunar rover navigation camera image coordinates and the orbiter image coordinates is obtained as: ； wherein, is the depth in the lunar rover navigation camera coordinate system, obtained by normalizing the last dimension value of , denotes the rotation matrix that transforms the vector in the world coordinate system into the lunar rover navigation camera coordinate system, is the value of the vector from the origin of the lunar rover navigation camera coordinate system to the origin of the world coordinate system in the lunar rover navigation camera coordinate system, and is specifically represented as:​ ， ； wherein, the rotation angle of the lunar rover navigation camera around , the rotation angle of the lunar rover navigation camera around , the height of the lunar rover navigation camera from the lunar surface; A top-view perspective feature grid is constructed, each coordinate value can be indexed to the coordinate in the lunar rover navigation camera image coordinate system based on the cross-view geometry correspondence relationship deduced in the foregoing, and a top-view perspective projection feature map is generated through bilinear interpolation 、 、 .

3. The method of claim 1, wherein, The step (2) comprises: Firstly, the orbiter semantic map is made according to the orbiter image, and after the real profile of the impact crater is labeled, it is also sent into the main network to extract multi-scale features 、 、 ; then, the feature channel number is adjusted to 1 by using convolution kernel , and the edge weight is obtained by using sigmoid function normalization, the orbiter image features extracted in step (1) are weighted by using the edge weight, and then connected with the original orbiter image feature residual to obtain the edge-enhanced orbiter image feature map 、 、 as follows: ； wherein, represents the addition of corresponding position elements of the feature maps, represents the multiplication of corresponding position elements of the feature maps, represents convolution kernel, k represents the serial number, k = 2, 4, 8; To further reduce the feature domain difference between cross-view images, a cross-view feature transfer module containing correlation embedding, hard attention for feature transfer and soft attention for feature synthesis is designed to transfer low-level texture features from the edge-enhanced orbiter image features to the overhead perspective projection feature map, guiding the model to learn a more powerful feature representation under the overhead view; the input of the cross-view feature transfer module is the projection feature map of the overhead view and the edge-enhanced orbiter image feature map : ； where the extracted texture features Q, K and V represent the three basic elements of the transformer attention mechanism; texture correlation between the overhead perspective projection feature map and the edge-enhanced trackside image feature map is modeled, which is intended to embed the correlation between the two view features by estimating the similarity between Q and K; Q and K are unfolded into patches and the correlation between patches is calculated by normalized inner product: ； wherein, represents a correlation embedding between two feature maps, represents the Euclidean norm of a vector, which is the square root of the sum of the squares of all elements of the vector, i and j represent the patch indices of Q, K expansion, respectively; For each query position, the hard attention mechanism only searches the most relevant position from V to transfer the features: ； wherein, The value of argmax can be regarded as an index representing the position in the edge-enhanced orbital imager feature map that is most relevant to the i-th position feature in the overhead perspective projection feature map. In order to obtain the transferred texture features T from the edge-enhanced orbiter image feature map, the hard attention map index V is used to expand the block, which is expressed as: ； wherein, denotes the value of the feature at the th position in V, denotes the feature value of the th position selected from V; a soft attention mechanism is used to fuse the transferred texture features T and the projection feature map of the top-down view; first, the transferred texture features T are combined with the projection feature map of the top-down view to make full use of the information in the original top-down view projection feature map, and a soft attention map is combined to measure the confidence of the transferred features from this position: ； wherein, represents the i-th position of the soft attention map S, the final output represents the synthesized overhead view features: ； wherein Concat denotes feature fusion by concatenating two features; a decoder is adopted to recover finer details of the top view at a higher resolution to obtain multi-scale synthesized top view features 、 、 .

4. The method of claim 1, wherein, The step (3) comprises: By sliding the synthesized overhead view on the edge-enhanced orbiter image feature map to perform convolution and normalized cross-correlation operation, the correlation score of each position is calculated: ； wherein, represents the normalized cross-correlation, represents the spatial correlation score.

5. The method of claim 1, wherein, The step (5) comprises: After the model training is completed, the image collected by the rover navigation camera in the scene and the orbiter image in a certain range of the current region are input, so as to synthesize the aligned and reliable multi-scale feature map for the cross-view image, and finally the absolute positioning result in the current cropped orbiter image is obtained through correlation analysis, and then it is registered to the lunar fixed coordinate system according to the actual task requirement, so that the lunar surface cross-view absolute positioning is completely realized.

Images