Asteroid landing segment location recognition method based on 3D-NetVLAD

Abstract

A kind of asteroid landing section location identification method based on 3D-NetVLAD, including generating containing multiple asteroid image data sets;Based on the asteroid image data set, 3D-NetVLAD network carries out multi-scale sampling, extracts feature map, obtains global feature vector;3D-NetVLAD network constructs triad data set and carries out weak supervision training;The navigation image to be queried is input into the trained 3D-NetVLAD network, and is input into the database generated in asteroid fly-around section, and 3D-NetVLAD outputs k images similar to the image to be queried in the database, determines the position of the navigation image to be queried.This application mainly solves the problem of autonomous navigation positioning of asteroid probe in landing process, by comparing known database, 3D-NetVLAD can realize the identification of the location of the position of probe, while overcoming the influence of illumination and scale on the identification result.

Description

Technical Field

[0001] This invention relates to a method for identifying asteroid landing sites based on 3D-NetVLAD, belonging to the field of computer vision technology. Background Technology

[0002] During the flyby observation phase before asteroid landing, the probe needs to simultaneously acquire images of the asteroid's surface from different orbital altitudes. Local features, such as corners, lines, edges, and craters, are then extracted manually or automatically from these images. During the landing phase, these reference image features are matched to achieve position estimation and navigation. This method of using known two-dimensional images or three-dimensional model features to assist the probe in precise landing has been widely used in asteroid exploration missions. Common methods include template matching, pattern matching, and descriptor matching. However, long-term flyby observations of asteroids generate a massive number of images, each with hundreds or thousands of local features, posing a challenge to the efficiency and accuracy of database retrieval. Compared to local features, global features can represent all local features of an image as a single feature vector, improving retrieval efficiency. Therefore, researchers have proposed a two-step strategy: first, constructing global features for each image to quickly identify the location of the navigation image; then, performing fine-grained local feature matching.

[0003] The most widely used method for location identification based on global features is the NetVLAD network. Its main idea is to perform weakly supervised training using the Euclidean distance between two images (such as GPS data), essentially assuming that images closer together have higher feature similarity. However, unlike terrestrial scenes, asteroid imagery orbits are not confined to a single plane but rather controlled in three-dimensional space near the asteroid. The probe's motion and observations at different orbital altitudes result in significant differences in image features between images that are close together but have different fields of view. Conversely, images at different orbital altitudes but at greater distances but with similar fields of view exhibit very similar features (only scale variations exist). These images are highly valuable for autonomous landing missions. Figure 1 As shown, due to the different camera perspectives, the image taken by the probe at point A differs significantly from the nearby image at point C, but is more similar to the image taken at the more distant image at point B. Therefore, methods like NetVLAD, which only use Euclidean distance for weakly supervised network training, are no longer applicable. This invention proposes a novel location identification method suitable for asteroid scenarios to achieve autonomous navigation and positioning of the probe during its descent and landing phase. Summary of the Invention

[0004] The technical problem to be solved by this invention is to overcome the shortcomings of the prior art and solve the problem of location identification in asteroid scenarios, so as to realize autonomous navigation and positioning of the probe during the landing phase.

[0005] The objective of this invention is achieved through the following technical solutions:

[0006] A method for identifying asteroid landing sites based on 3D-NetVLAD includes:

[0007] Generate a dataset containing images of multiple asteroids;

[0008] Based on asteroid image datasets, the 3D-NetVLAD network performs multi-scale sampling, extracts feature maps, and obtains global feature vectors.

[0009] The 3D-NetVLAD network is used to construct a triplet dataset for weakly supervised training.

[0010] The navigation image to be queried is input into the trained 3D-NetVLAD network, along with the database generated from the asteroid's flyby segment. 3D-NetVLAD outputs k images from the database that are similar to the image to be queried, thus determining the location of the navigation image to be queried.

[0011] In one embodiment of the present invention, Blender software is used to generate a dataset containing multiple asteroid images.

[0012] In one embodiment of the present invention, the process of generating a dataset containing multiple asteroid images using Blender software includes: adding an asteroid model, using the asteroid's center of mass as the origin of the world coordinate system, placing four light sources near the asteroid, and simulating camera imaging under different light intensities and directions.

[0013] In one embodiment of the present invention, in order to generate images of different scales, the camera is placed at different orbital altitudes to take pictures of the target asteroid. The optical axis of the camera always points to the center of mass of the asteroid to ensure that the asteroid is at the center of gravity of the image.

[0014] In one embodiment of the present invention, the 3D-NetVLAD network first performs multi-scale sampling on the input image through a feature pyramid, then extracts feature maps from images at different scales, and finally encodes feature maps at different levels to form feature vectors.

[0015] In one embodiment of the present invention, an L-level feature map {F} is obtained through a multi-scale feature extraction network. 1 ,F 2 ,...,F L}, where F l ∈W l ×H l ×D,W l and H l Represents the feature map F of the l-th layer lThe size of the feature map is given by D, which represents the number of feature channels or dimension. By setting v cluster centers and clustering all feature maps at level L, each level will generate v global feature vectors, describing the residual from each feature vector to the cluster center. in It is a soft assignment function, representing the weight of each feature assigned to each cluster center; the residuals of each layer are summed to obtain the final residual m. v ∈V×D, the final residual is represented as a global feature vector, which describes the distance from all local features on the image to the cluster center.

[0016] In one embodiment of the present invention, during the training of the 3D-NetVLAD network, positive and negative samples are defined by the following two conditions:

[0017] (1) Define images whose distance from the query image is less than dmin as positive samples and those whose distance is greater than dmax as negative samples. Positive samples that meet the requirements are contained in a sphere with the query image q as the center and the radius dmin.

[0018] (2) The viewpoint was restricted, and positive samples were defined as the set of images whose viewpoints with the query image q were less than θ.

[0019] The final set of positive sample images is the intersection of the two conditions above.

[0020] In one embodiment of the present invention, the 3D-NetVLAD network constructs a triple dataset {q,p,n} for weakly supervised training. For a query image q, it finds a set of positive sample images {p} that satisfy the above range with q, and a set of negative sample images {n} that are far away from q. The purpose is to make the distance d(V(q),V(p)) between the query image features and the positive sample image features much smaller than the distance d(V(q),V(n)) between the query image features and the negative sample image features, where V() represents the global features extracted from each image.

[0021] In one embodiment of the present invention, during the weakly supervised training of the 3D-NetVLAD network, the triplet loss function is set as follows:

[0022]

[0023] Where δ is the threshold for controlling the similarity of features and thus determining whether a loss occurs.

[0024] A computer-readable storage medium having stored thereon computer program instructions, which, when loaded and run by a processor, cause the processor to execute the aforementioned asteroid landing site identification method based on 3D-NetVLAD.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] (1) This invention proposes a method for identifying the landing site of an asteroid based on 3D-NetVLAD, which mainly solves the problem of autonomous navigation and positioning of an asteroid probe during landing. By comparing with known databases, 3D-NetVLAD can identify the location of the probe and overcome the influence of illumination and scale on the identification results.

[0027] (2) In the feature extraction stage, feature pyramids are used to generate L-layer feature maps. Unlike previous methods that directly fuse L-layer features and then encode them, this invention first sets cluster centers to calculate the residuals of each layer of feature maps, and then fuses the residuals of multiple layers, so that the algorithm still maintains good performance when the image resolution difference is large.

[0028] (3) Unlike previous NetVLAD, 3D-NetVLAD fully considers the characteristics of the asteroid probe landing process, increases the limitation of the viewpoint, and raises the definition of positive samples from 2D to 3D during the training process, which can effectively improve the algorithm performance.

[0029] (4) The present invention constructs a dataset of asteroid scenarios, which facilitates the training and verification of algorithms. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the asteroid exploration process.

[0031] Figure 2 This is a flowchart of the method.

[0032] Figure 3 This is a diagram of the triplet training process.

[0033] Figure 4 This is a schematic diagram for generating a simulated image.

[0034] Figure 5 'a' represents the actual image. Figure 5 b is a simulated image.

[0035] Figure 6 This is a schematic diagram of feature extraction. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0037] A method for identifying asteroid landing sites based on 3D-NetVLAD is proposed, exhibiting adaptability to changes in illumination and scale. This method can identify the top k most similar images from a reference database when given an input image taken during the landing phase, providing support for autonomous navigation of the probe. The main idea of ​​this invention is to compare images taken during the landing phase with a database formed during the flyby phase, using a 3D-NetVLAD network to identify the top-k images most similar to the landing phase images. The location of the probe can then be determined using these images. A flowchart of the specific method is shown below. Figure 2 As shown.

[0038] 1. Dataset

[0039] Training 3D-NetVLAD requires relevant data on asteroid scenes. This invention uses a self-made asteroid image dataset for training. Currently, only real-world images of asteroids are available, and there is a lack of publicly available datasets suitable for location recognition training. Therefore, this invention uses Blender software to generate a dataset containing multiple asteroid images. The generation process includes: adding an asteroid model, using its centroid as the origin of the world coordinate system, and placing four light sources near the asteroid to simulate camera imaging under different light intensities and directions.

[0040] To generate images at different scales, the study chose to place the camera at different orbital altitudes to orbit and photograph the target asteroid, with the camera resolution set to 1024*1024 pixels. Figure 4 As shown, the camera's optical axis (the camera's z-axis) always points towards the asteroid's center of mass to ensure that the asteroid is at the center of the image.

[0041] 2.3D-NetVLAD Network Feature Extraction

[0042] Because the image resolution varies greatly across different parts of the asteroid's surface during the landing process, the 3D-NetVLAD network first performs multi-scale sampling of the input image using a feature pyramid, then extracts feature maps from images at different scales, and finally encodes these feature maps at different levels to form feature vectors. This multi-layer feature approach effectively improves the network's adaptability to scale. The input asteroid surface image, such as... Figure 6 As shown, an L-level feature map {F} is obtained through a multi-scale feature extraction network. 1 ,F 2 ,...,F L}, where F l ∈W l ×H l ×D,W l and H l Represents the feature map F of the l-th layer lThe size of the feature map is defined by D, which represents the number of feature channels or the dimension. By setting v cluster centers and clustering all feature maps at level L, each level will generate v global feature vectors, describing the residual from each feature vector to the cluster center. in It can be considered a soft assignment function, representing the weight of each feature assigned to each cluster center.

[0043]

[0044] Finally, the residuals of each layer are summed to obtain the final residual m. v If ∈V×D, then this image can be considered as a global feature vector, which describes the distance of all local features on the image to the cluster center.

[0045]

[0046] 3. Network training methods and loss function design

[0047] NetVLAD is currently the most widely used location recognition network. It achieves weakly supervised training by defining positive and negative samples for image I. K , with I K Relevant images are considered positive samples, while those that are not are considered negative samples. During training, the image that best matches the positive sample is selected for training. However, NetVLAD's positive samples are determined by comparing them with image I. K The positive and negative samples are defined by their Euclidean distance, meaning that images that are closer are considered to have more similar image features. However, when positive samples include images of different scales, the system tends to select the closer images while ignoring scale information. But in real-world navigation tasks, images of different scales are also crucial; images that are closer can sometimes cause training to fail due to different viewpoints. Therefore, unlike the NetVLAD network which only uses Euclidean distance to determine positive and negative samples, the proposed 3D-NetVLAD defines positive and negative samples in two ways:

[0048] (1) Define images whose distance from the query image is less than dmin as positive samples and those whose distance is greater than dmax as negative samples. That is, the positive samples that meet the requirements are contained in a sphere with the query image q as the center and the radius dmin.

[0049] (2) The viewpoint is limited, and positive samples are defined as the set of images with a viewpoint q less than θ of the query image. That is, the two can have a larger common viewing area, which can more accurately limit the range of positive samples with features similar to the query image.

[0050] The final set of positive sample images is the intersection of the two above.

[0051] 3D-NetVLAD constructs a triplet dataset {q,p,n} for weakly supervised training. Specifically, for a query image q, it finds a set of positive sample images {p} that satisfy the above range with respect to q, and a set of negative sample images {n} that are far away from q. The goal is to make the distance d(V(q),V(p)) between the query image features and the positive sample image features much smaller than the distance d(V(q),V(n)) between the query image features and the negative sample image features, where V() represents the global features extracted for each image in Section 2. The triplet loss function is set as follows:

[0052]

[0053] Where δ is the threshold for controlling the similarity of features and thus determining whether loss occurs. The training process of triples is as follows: Figure 3 As shown, the main goal is to reduce the distance between matching images while ensuring that the distance between mismatched images is greater than the distance between matching image pairs, thereby improving the network's recognition performance. The training data for 3D-NetVLAD uses a self-made asteroid image dataset, detailed in Section 1.

[0054] 4. Location Identification

[0055] The navigation image to be queried is input into the trained 3D-NetVLAD and the database generated by the flight path segment. 3D-NetVLAD can then output k images in the database that are similar to the image to be queried. Since the position and attitude of these images are known, the location of the image to be queried can be determined through 2D-3D image matching, thus enabling the detector to navigate autonomously.

[0056] Figure 5 These are real and simulated images of the asteroid. As can be seen from the images, compared with the real image, the texture and shadows of the simulated image are more realistic and can basically reflect the surface features of the real image.

[0057] During asteroid image generation, the altitude of the imaging orbit is controlled within a certain range. Each set of images is generated at a 5-kilometer interval, and each set contains 51 images. When generating image pairs, each image is paired with images taken from adjacent angles at other altitudes.

[0058] The contents not described in detail in this specification are common knowledge to those skilled in the art.

[0059] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for identifying asteroid landing sites based on 3D-NetVLAD, characterized in that, include: Generate a dataset containing multiple asteroid images; to generate images at different scales, multiple cameras are placed at different orbital altitudes to take pictures of the target asteroid, with the optical axis of the camera always pointing towards the center of mass of the asteroid to ensure that the asteroid is at the center of mass of the image; Based on asteroid image datasets, the 3D-NetVLAD network performs multi-scale sampling, extracts feature maps, and obtains global feature vectors. The 3D-NetVLAD network is trained with weak supervision using a triplet dataset. During training, positive and negative samples are defined using the following two conditions: (1) Define images whose distance from the query image is less than dmin as positive samples and those whose distance is greater than dmax as negative samples. Positive samples that meet the requirements are contained in a sphere with the query image q as the center and the radius dmin. (2) The limitation on the viewpoint has been added, defining a positive sample as one whose viewpoint is less than that of the query image q. A collection of images within a specified range; The final set of positive sample images is the intersection of the two conditions above; The navigation image to be queried is input into the trained 3D-NetVLAD network, along with the database generated from the asteroid's flyby segment. 3D-NetVLAD outputs k images from the database that are similar to the image to be queried, thus determining the location of the navigation image to be queried.

2. The asteroid landing site identification method according to claim 1, characterized in that, A dataset containing images of multiple asteroids was generated using Blender software.

3. The method for identifying asteroid landing sites according to claim 2, characterized in that, The process of generating a dataset containing multiple asteroid images using Blender software includes: adding an asteroid model, using the asteroid's center of mass as the origin of the world coordinate system, placing four light sources near the asteroid, and simulating camera imaging under different light intensities and directions.

4. The method for identifying asteroid landing sites according to claim 1, characterized in that, The 3D-NetVLAD network first performs multi-scale sampling on the input image through a feature pyramid, then extracts feature maps from images at different scales, and finally encodes the feature maps at different levels to form feature vectors.

5. The asteroid landing site identification method according to claim 4, characterized in that, Obtained through a multi-scale feature extraction network Hierarchical feature map ,in , and Indicates the first Layer feature map Size, Indicates the number of feature channels or dimensions; settings Each cluster center, for If all feature maps at each level are clustered, then each level will form... Each global feature vector describes the residual from the cluster center to each feature vector. ,in It is a soft assignment function, representing the weight of each feature assigned to each cluster center; Sum the residuals of each layer to obtain the final residual. The final residual is represented as a global feature vector, which describes the distance from all local features on the image to the cluster center.

6. The method for identifying asteroid landing sites according to claim 1, characterized in that, 3D-NetVLAD network for constructing triplet datasets Weakly supervised training is performed on a query image. Find the match on the dataset The set of positive sample images that meet the above range and a set of negative sample images that are far apart. The purpose is to query the distance between image features and positive sample image features. Much smaller than the distance between the query image features and the negative sample image features. ,in This represents the global features extracted from each image.

7. The method for identifying asteroid landing sites according to claim 6, characterized in that, During the weakly supervised training of the 3D-NetVLAD network, the triplet loss function is set as follows: in A threshold is used to control the similarity of features and thus determine whether a loss occurs.

8. A computer-readable storage medium having stored thereon computer program instructions, which, when loaded and run by a processor, cause the processor to perform the method as described in any one of claims 1 to 7.

Images