A High-Precision Landing and Positioning Method Based on Event Camera and Terrain Feature Matching

Abstract

This invention proposes a novel high-precision landing and positioning method based on event camera and terrain feature matching, specifically including the following steps: acquiring pulse event images of the target landing area on the surface of a celestial body using an event camera; then preprocessing the acquired pulse event images to make them more suitable for a spiking neural network; training the spiking neural network (SNN) and extracting terrain features from the input pulse event images; using the extracted terrain features for feature matching based on shadow edge contour features and crater features; finally, calculating the position and attitude of the lander to complete the positioning. This invention combines an event camera with an SNN network and applies it to the field of planetary positioning and landing. Through a trained spiking neural network, the required terrain features are extracted, and then feature matching is performed to ultimately achieve the positioning of the lander and ensure its safe landing.

Description

Technical Field

[0001] This invention relates to a high-precision landing and positioning method for the moon or other celestial bodies based on event camera and terrain feature matching, belonging to the field of visual navigation, particularly a positioning and navigation method based on event camera and spiking neural network for the landing phase of celestial bodies such as the moon. Background technology:

[0002] In recent years, some countries have invested a lot of effort in deep space exploration, among which the landing of landers on the surface of celestial bodies is particularly important, requiring high-precision positioning and terrain navigation to ensure the safety of personnel and exploration equipment.

[0003] An event camera is a neuromorphic sensor that simulates biological neurons, with a reaction time on the microsecond level. It records an asynchronous stream of brightness changes for each pixel, also known as an "event." It breaks the concept of a frame, ensuring a certain temporal resolution and dynamic range by changing the way information is acquired and processed, while also featuring low power consumption and high computational efficiency. Furthermore, event cameras can still function normally in high-dynamic and high-speed environments, offering high temporal resolution in high-dynamic conditions and the advantage of no motion blur in high-speed conditions. Compared to traditional cameras, event cameras can capture moving objects well even in darker environments, reducing environmental limitations. In addition, event cameras offer low latency and no motion blur; the time interval between adjacent events can be less than 1 millisecond, demonstrating excellent robustness to motion blur, low latency, and high dynamic range.

[0004] Traditional cameras, whether using CMOS, CCD, or RGBD sensors, all suffer from latency issues. This is because they can only acquire images at a constant frequency, limited by the frame rate parameter. The event camera used in this invention effectively solves these problems. The event camera is characterized by high speed, low data volume, low power consumption, and a large dynamic range. It can acquire the optical scattering signals of each pixel in the target landing area, recording them as a time pulse sequence and storing useful information. The event camera operates by generating an output whenever photons move; otherwise, the output is zero. The corresponding output is a time pulse sequence, recording a transmitted array containing only 0s and 1s. The recording method involves releasing one time pulse within one time step, and each pixel in the captured pulse event image operates independently and asynchronously. Event cameras are currently widely used in many fields with excellent results and significant development potential, such as ultrafast vision-driven control, electric line tracking, aircraft dynamic obstacle avoidance, low-latency high-bandwidth control, high-speed high dynamic range, and continuous-time visual inertial odometry.

[0005] SNN (Spiking Neural Network), also known as the third-generation neural network, utilizes a spiking neuron model designed based on biological neuron models, possessing powerful computational and temporal processing capabilities. The most commonly used mathematical model for the spiking neuron's point dynamics is the leakage integral-discharge model, which mimics the charging, leakage, and discharging process of biological neurons. When determining the output of a spiking neuron, only the correct neuron fires at the highest frequency, while other neurons remain silent. The neurons in a spiking neural network are task-driven and do not need to be constantly running, thus reducing energy consumption and fully utilizing spatiotemporal information to achieve an efficient spiking neural network framework. The SNN uses pulse event images composed of time-stamped pulse sequences captured by an event camera as input, feeding them into a pre-trained SNN network. The output neuronal pulses from the SNN model are then decoded. The SNN network is trained using supervised learning methods, employing a biomimetic learning algorithm based on synaptic plasticity rules. The strength of connections between neurons is adjusted according to the order of neuronal firing to complete the training. The entire SNN network model is essentially composed of a series of encoding-decoding structures, using a model that best fits the biological neuron mechanism for computational analysis. The output of the SNN network changed from the continuous output of the second-generation network to the binary output. This spike training enhanced the connection of the processed data and had more powerful computing units and computing power.

[0006] To achieve high-precision landing navigation for celestial bodies, a terrain feature database needs to be established, specifically including two databases: a crater feature database and a shadow edge contour line feature database. Based on existing data, numerous craters of varying sizes and various shadow edge contour lines exist within the region. Since the locations of craters do not change over long periods, they are ideally suited for use as navigation map beacons. Therefore, this patent further calibrates the crater and shadow edge contour line features based on pulse event images of the selected target landing area to establish the terrain feature database.

[0007] Feature matching is applied using two methods: shadow edge contour features and crater features, to calculate and locate the lander's position and attitude. Pulse event images are captured using an event camera, and an SNN network is used to extract crater features and surrounding shadow contour features. These features are then matched with a pre-established terrain feature database to achieve pose calculation and navigation.

[0008] The process of terrain feature matching and lander pose calculation involves first defining the target landing area. During the lander's descent and landing, the shadows formed by the solar altitude angle are extracted, and their contour lines are matched against shadow contour lines in the terrain feature database. Simultaneously, the morphology and distribution characteristics of extracted craters are also used for matching. This information fusion improves the positioning accuracy of the camera pose, achieving high-precision navigation. Furthermore, due to differences in altitude and viewing angle during shooting, the results for the same crater may differ at different shooting times. However, the differences between pulse event images at different times can be adjusted using affine transformations. Pulse event images of craters possess affine invariance, which is a crucial matching criterion for feature matching. Feature matching utilizes methods based on shadow contour lines and crater features. The shadow contour line matching algorithm is a fundamental feature matching method; its main idea is to extract the shadow contours from the pulse event images and match them against shadow contour lines in the terrain feature database. In addition, craters are a significant terrain feature on the lunar surface, and utilizing crater features for navigation and landing helps improve accuracy. Specifically, it employs optical and inertial navigation combined with crater detection and matching to create a feature matching method based on crater characteristics. Finally, using the matching results, the coordinate transformation matrix is ​​calculated using the least squares method. Then, by applying the known intrinsic parameter matrix from the event camera, the lander's pose is determined, and a suitable landing location is found. Summary of the Invention

[0009] This invention proposes a high-precision landing and positioning method based on event camera and terrain feature matching. First, an event camera capable of capturing images of a celestial body at high speeds without motion blur is used to obtain pulse event images containing temporal pulse information of the celestial body surface. After certain preprocessing, the pulse event images are input into pre-trained SNN networks for extracting shadow edge contour features and crater features, respectively, to extract the terrain feature information required for matching. Then, feature matching is performed between the two types of features and a terrain feature database, and the matching results are fused. Finally, the lander's pose is calculated. This invention uses two types of features: one based on the shadow edge contour features present in the pulse event images, and the other based on the morphology and distribution of detected craters. These two features are used to complete the feature matching process. Finally, the landing position and attitude of the lander are analyzed and calculated based on the matching results to locate the optimal landing site for a safe landing.

[0010] This invention primarily relies on pulse event image acquisition from an event camera and high-precision feature matching. It employs SNN networks for extracting shadow edge contour features and crater features, along with feature matching algorithms based on different features, to achieve high-precision landing and positioning, ultimately ensuring the lander safely lands at a suitable location. The specific technical solution is as follows:

[0011] The high-precision landing and positioning method based on event camera and terrain feature matching proposed in this invention has the following process: Figure 1 As shown, its features are: 1) The method of acquiring pulse event images by the event camera differs from the traditional camera's frame-based shooting method. Each photon represents one photon time, and the output is a sequence of time pulses, recorded using only simple 0 and 1 signals. The event camera has an extremely short reaction time, limited only by the photodetector, thus possessing extremely high temporal resolution and good dynamic range. It can easily capture and photograph high-speed moving targets with clear image quality. In addition, the event camera can also capture moving objects in low-light environments, demonstrating high perception capabilities. 2) The SNN network used is a novel network with a natural connection to the event camera. The output of the event camera can be input and learned by the spiking neural network. This network has strong computational power, strong temporal processing capabilities, low energy consumption, and rich network learning methods. This network is also well-suited to the data format output by the event camera. 3) The feature matching algorithm in the invention is based on shadow edge contour features and crater features, essentially a terrain navigation based on feature matching. Based on known stellar illumination models, renderings can be created in advance, and simulated renderings of the target landing area can be calculated at any time, thus providing a wealth of data.

[0012] In this invention, the various parts of the system are described as follows:

[0013] 1. The event camera described in the patent, characterized in that: the pixel detectors in the event camera operate asynchronously, and the operation mode of the event camera is as follows: Figure 3 As shown, the optical scattering signal input is accumulated. Within one accumulation cycle, if the accumulation result reaches a threshold condition, a high-level pulse is immediately output, and the accumulation result is simultaneously cleared to zero. Then, a new accumulation cycle begins. If the accumulation result does not reach the threshold condition within one accumulation cycle, a low-level pulse is continuously output. Each pixel detector of the event camera acquires the optical scattering signal of the target landing area according to the above method and forms a corresponding time pulse sequence, ultimately forming a pulse event image.

[0014] 2. The preprocessing operation described in the patent refers to preprocessing the pulse event image before it is input into the SNN network, specifically including: noise reduction processing, detail enhancement processing, geometric correction processing, filtering processing, and smoothing processing.

[0015] 3. The spiking neural network for extracting shadow edge contour features and the spiking neural network for extracting crater features described in the patent refer to a new generation of artificial neural network models inspired by biology. These models use event-driven pulse time series to transmit information and process pulse event images output by an event camera. The pre-trained feature extraction spiking neural network refers to the fact that the invention pre-created a training set and conducted multiple rounds of training on the feature extraction spiking neural network. The training employs a supervised learning method, utilizing a biomimetic learning algorithm based on synaptic plasticity rules to adjust the strength of connections between neurons according to the order of neuron activation. The trained feature extraction spiking neural network can then extract terrain features from the input pulse event images. The entire feature extraction spiking neural network model is essentially composed of a series of encoder-decoder structures, using a model that best fits the biological neuron mechanism for computational analysis.

[0016] 4. The terrain features described in the patent include two types: shadow edge contour features in the pulse event image and crater morphology and distribution features. The extracted features are then matched with the terrain feature database.

[0017] 5. The feature matching described in the patent comprises two steps: the first step is to perform feature matching based on shadow edge contour features and crater features respectively; the second step is to fuse the results of these two feature matching methods to obtain the final feature matching result. The feature matching based on shadow edge contour features refers to using terrain shadow edge contour matching to find the registration between the event camera coordinate system and the real geographic data coordinate system; the feature matching based on crater features refers to matching the detected crater features with crater feature information in the terrain feature database.

[0018] 6. The terrain feature database described in the patent includes a crater feature database and a shadow edge contour feature database: the crater feature database is a collection of data on the shape and distribution of all craters in the target landing area, used for feature matching with the extracted crater features; the shadow edge contour feature database is a collection of light and dark boundary curves generated by illumination at different times in the target landing area, used for feature matching with the extracted shadow edge contour features.

[0019] 7. The pose calculation described in the patent is based on a certain correspondence established by the feature matching results, and then the coordinate transformation matrix is ​​calculated using the least squares method. Then, based on the known event camera intrinsic parameter matrix, the pose of the lander is calculated, and the position and attitude of the lander in the target landing area are obtained.

[0020] The main features of this invention are as follows: This invention uses an event camera to acquire pulse event images, which are then preprocessed and input into a pre-trained spiking neural network for extracting shadow edge contour features and an SNN network for extracting crater features to identify craters and extract terrain features. The shadow edge contour features and crater morphology and distribution features are extracted by the two SNN networks respectively, and then matched with two terrain feature databases and the results are fused. Finally, the matching results are used to calculate the pose of the lander to achieve navigation and complete high-precision landing and positioning.

[0021] Benefits and application prospects of this invention: It can be applied to the high-precision landing and positioning of landers during celestial surface exploration. Attached Figure Description

[0022] Figure 1 Flowchart of the present invention

[0023] Figure 2 Schematic diagram of neuron model and its working principle

[0024] Figure 3 Schematic diagram of how the event camera works Detailed Implementation

[0025] like Figure 1 The image shows the entire process of a high-precision landing and positioning method based on event camera and terrain feature matching.

[0026] The embodiments of this invention employ an event camera that outputs only local pixel-level brightness change-related pulse event images. When these pixel-level brightness changes exceed a set threshold, the event camera timestamps them at microsecond resolution and outputs an asynchronous event stream. This invention utilizes an asynchronous time-based image sensor (ATIS event camera), which introduces a time-interval-based light intensity measurement circuit on top of a DVS to achieve image reconstruction. The ATIS pixel structure is divided into two parts (A and B), containing two photosensors, capable of providing event information while also providing a certain amount of grayscale information. Since the time required for the same voltage change varies under different light intensities, the magnitude of the light intensity can be inferred by establishing a mapping between light intensity and time, thereby outputting the light intensity camera at the pixel where the light intensity changes. The ATIS event camera introduces a global emission mechanism. Initially, it obtains a complete pulse event image as the background. Then, the target landing area continuously generates pulses, continuously triggering the light intensity measurement circuit to obtain the grayscale of the moving area to update the background. The ATIS event camera used in this invention has a high resolution of 304×240, a high dynamic range of 143dB, and lower power consumption, and can trigger a global HDR output once per event.

[0027] like Figure 3 The diagram shown illustrates the operation of the event camera used in this invention. When acquiring the initial pulse event image, this invention selects to use an event camera, resulting in more accurate information and better capture of high-speed moving objects.

[0028] Next, the pulse event images are preprocessed and input into SNN networks for extracting shadow edge contour features and SNN networks for extracting crater features. This is a big data algorithm with three different topologies. The first is a feedforward SNN network, where neurons are arranged hierarchically, with one or more hidden layers between the input and output layers. The second is a recursive SNN network, which has feedback loops, and the output of a neuron is a recursive function of the neuron's output at a previous time step. The last is a hybrid SNN network, which combines the first two structures.

[0029] The two pre-trained SNN networks refer to the fact that this invention pre-created training and validation sets and conducted multiple rounds of training on the two SNN networks. The training employed a supervised learning method, utilizing a biomimetic learning algorithm based on synaptic plasticity rules to adjust the strength of connections between neurons according to the order of neuron activation. After training, these two SNN networks can identify craters and extract shadow edge contour features, as well as crater features themselves. These features are then matched with a terrain feature database for lander pose calculation, ultimately achieving navigation and positioning functionality.

[0030] A typical lander's descent trajectory can be divided into five segments: the parking trajectory, the transition trajectory, the braking and deceleration phase, the approach phase, and the final descent phase. The latter three segments are collectively referred to as the powered descent phase. Visual navigation based on event cameras plays a role throughout the entire descent process. However, during the transition and braking / deceleration phases, navigation techniques using shadow edge contours and crater features as markers are highly valuable, enabling navigation accuracy to meet the requirements of high-precision landing.

[0031] The invention will be further illustrated below with examples. For instance... Figure 1 As shown, the high-precision landing and localization method based on event camera and terrain feature matching includes the following steps:

[0032] S1. Use the event camera to capture pulse event images of the lunar south pole. Obtain optical scattering information of the target landing area, and use the detectors of each pixel of the event camera to obtain time pulse sequences. The collection of time pulse sequences of all pixels constitutes the pulse event image.

[0033] S2. Preprocessing operations are performed on the pulse event images, including denoising, detail enhancement, geometric correction, filtering, and smoothing. This makes the shadow edge contours of the pulse event images in the dataset clearer, facilitating further extraction and analysis, avoiding the influence of angle or lighting shadows during shooting, eliminating irrelevant information, and restoring true and effective information.

[0034] S3. An SNN network for extracting shadow edge contour features and an SNN network for extracting crater features were constructed and trained. The two SNN networks were trained using the training set respectively, so that the two SNN networks could accurately identify craters and extract useful terrain features from the pulse event image. After specific analysis, it was found that the methods based on shadow edge contour features and crater features had the best effect. Therefore, this patent selected these two terrain features as the main terrain features.

[0035] S31. Before training the SNN network, a dataset is constructed. When constructing the dataset, the existing LRO and Chang'e camera are used to segment the captured complete images through upsampling, downsampling and random segmentation. Then, the terrain feature extraction algorithm is used to manually label the images to create a dataset. 80% of the data in the dataset is used as the training set and the remaining 20% ​​is used as the validation set.

[0036] S32. Load the training set into the SNN network for extracting shadow edge contour features and the SNN network for extracting crater features, where the parameters of the data are 92160*30720 pixels, 8 bits / pixel.

[0037] S33. Configure the basic parameters of the two SNN networks respectively. For example, by adjusting the parameter of test_model, you can choose whether the SNN network is in the testing or training function. Setting the parameter to true will enable the training function. You can also set the update interval, the interval for saving the SNN network connections, and some basic parameters of the neuron model.

[0038] S34. The accuracy of the SNN network for extracting shadow edge contour features and the SNN network for extracting crater features are verified using a validation set. Once a certain level of accuracy is achieved, the network can be put into use.

[0039] S4. The positioning and navigation work is completed through feature matching.

[0040] S41. Before performing feature matching, construct a shadow edge contour feature library and a crater feature library. The crater feature library is constructed by acquiring existing DOM images of the target landing area using an LRO camera, extracting features using the LBP feature extraction algorithm, and manually supplementing and calibrating them. First, obtain the diameter and location information of the craters, and then obtain their shape and distribution characteristics to form the crater feature library. The shadow edge contour feature library is constructed by using a lunar illumination mathematical model of the target landing area, obtaining DOM simulation images at different times through 3D rendering, and then using the Canny edge extraction algorithm to obtain the corresponding shadow edge contours, and then establishing the shadow edge contour feature library.

[0041] S42. Feature matching based on shadow edge contour features. Extract shadow edge contour features from the pulse event image, and use information from the terrain feature database to match and find the registration between the pulse event image coordinate system and the real geographic data coordinate system. This allows the attitude and position information of the lander to be determined, so as to find a safe and suitable landing point.

[0042] S43. Feature matching based on the morphology and distribution characteristics of meteorite craters. Meteorite craters are the most prominent topographic features on the surface of celestial bodies, and are easy to match and track as navigational landmarks. Over time, meteorite craters exhibit consistent geometric contours (circles or ellipses) and characteristics of light and dark areas under different lighting conditions and camera poses. This patent utilizes these unique light and dark structural features of meteorite craters to analyze their structural shapes and generate corresponding templates, achieving feature matching with meteorite crater features in the topographic feature database.

[0043] S5. Calculate the lander's position and attitude using the results of feature matching.

[0044] S51. First, convert the impulse event images output by the two SNN networks into traditional images before solving them.

[0045] S52. Substitute the successfully matched terrain features into the formula, use the least squares method to calculate the coordinate transformation matrix, and then use the known event camera intrinsic parameter matrix to solve the lander's pose, thereby obtaining the lander's position and attitude in the target landing area, achieving high-precision navigation and safe landing.

Claims

1. A high-precision landing positioning method based on event camera and terrain feature matching, characterized by: The event camera is applied to acquire optical scattering signals of the target landing area; the optical scattering signals are recorded as time pulse sequences by each pixel detector of the event camera respectively; in a detection period, the collection of the time pulse sequences acquired by all the pixels of the event camera is recorded as a pulse event image; after the pulse event image of the target landing area is preprocessed, the pulse event image is output to the pulse neural network for extracting shadow edge contour line features and the pulse neural network for extracting meteorite crater features to extract terrain features respectively; finally, the extracted shadow edge contour line features and meteorite crater features are matched with the pre-established terrain feature database respectively, and the terrain feature matching includes the following steps: first, the extracted shadow edge contour line features are matched with the shadow edge contour line feature database, and the extracted meteorite crater features are matched with the meteorite crater feature database; second, the matching results of the two kinds of features are fused to obtain the final feature matching result; and the final feature matching result is used for pose solving to obtain the position and attitude of the lander in the target landing area.

2. The method of claim 1, wherein the event camera is a dynamic vision sensor (DVS) camera. The pixel detectors in the event camera work asynchronously, accumulate the optical scattering signals input thereto, and output a high-level pulse as soon as the accumulation result reaches a threshold condition in a accumulation period, and clear the accumulation result and start a new accumulation operation; if the accumulation result does not reach the threshold condition in an accumulation period, a low-level pulse is output continuously; each pixel detector of the event camera acquires the optical scattering signals of the target landing area and forms corresponding time pulse sequences according to the above method, and finally forms a pulse event image.

3. The high-precision landing positioning method based on the event camera and terrain feature matching according to claim 1, wherein the preprocessing operation is denoising processing, detail enhancement processing, geometric correction processing, filtering processing and smoothing processing on the pulse event image.

4. The high-precision landing positioning method based on the event camera and terrain feature matching according to claim 1, wherein the pulse neural network for extracting shadow edge contour line features and the pulse neural network for extracting meteorite crater features are a new generation of artificial neural network model derived from biological inspiration, which uses event-driven pulse time sequences to transmit information and processes the pulse event image output by the event camera; and the pre-trained pulse neural network is a pulse neural network that has been trained for multiple rounds after a training set and a verification set are prepared, wherein supervised learning is used for training, and a bionic learning algorithm based on synaptic plasticity rules is used to adjust the strength of the connection between neurons according to the order of neuron firing to complete the training.

5. The high-precision landing positioning method based on the event camera and terrain feature matching according to claim 1 or 4, wherein the terrain features include shadow edge contour line features and meteorite crater shapes and distribution characteristics in the pulse event image, which are used for feature matching with the terrain feature database.

6. The high-precision landing positioning method based on event camera and terrain feature matching according to claim 1, wherein the terrain feature database comprises a crater feature database and a shadow edge contour feature database; the crater feature database is a collection of data on the shapes and distribution of all craters in the target landing area; and the shadow edge contour feature database is a collection of bright-dark boundary curves generated by light at different times in the target landing area.

7. The high-precision landing positioning method based on event camera and terrain feature matching according to claim 1, wherein the pose solution is obtained by using the least square method to calculate a coordinate transformation matrix based on the corresponding relationship established according to the feature matching result, and then using a known event camera intrinsic matrix to solve the pose of the lander and obtain the position and attitude of the lander in the target landing area.

Images