Supersonic airliner scene matching navigation method based on neural implicit surface

Abstract

The invention provides a supersonic airliner scene matching navigation method based on a neural implicit surface, belongs to the technical field of aircraft navigation and computer vision, and solves the problems that an existing scene matching-based navigation method needs to store ground image data in advance, the data storage requirement is high, feature points are easy to lose in a complex climatic environment, and the navigation efficiency is high. The method comprises the following steps: collecting aerial images near a conventional airline of a passenger plane, and preprocessing the aerial images to obtain a camera pose of each aerial image; according to the camera pose and the corresponding aerial image, the neural implicit surface of the area nearby the supersonic airliner route is trained in an off-line mode; giving an initial position by using a supersonic airliner airborne sensor, and rendering a ground image corresponding to a specified position by using the trained neural implicit surface; setting the position of the passenger plane as particles, setting the difference between the rendered image and the real-time shot image of the passenger plane as loss, and calculating the accurate position of the supersonic passenger plane on the airline through a particle swarm algorithm.

Description

Technical Field

[0001] The invention relates to a supersonic passenger aircraft scene matching navigation method based on neural implicit surface, belonging to the technical field of aircraft navigation and computer vision. Background Art

[0002] Supersonic passenger aircraft refer to civil aircraft with a cruising speed exceeding Mach 1, which is particularly suitable for long-distance transoceanic flights. In recent years, thanks to the progress of new materials and new processes, it has become possible to develop reliable, quiet and economical supersonic technology. Reliable and accurate navigation of supersonic passenger aircraft is a key technology for smooth operation. In the existing technology, the solution based on Beidou / GPS needs to rely on external facilities, is easily interfered with, and has insufficient reliability. The solution based on inertial navigation has time drift and cumulative errors and is not suitable for long-term navigation. The method based on radar / laser to obtain elevation information for matching has high energy consumption, large volume, and requires a long matching area. In comparison, the method based on visual images for scene matching has the advantages of not relying on external equipment, strong anti-interference ability; no cumulative error, high positioning accuracy; passive detection, and low energy consumption. It has unique advantages in supersonic passenger aircraft navigation.

[0003] Existing scene matching-based navigation methods require pre-storage of ground image data, which has high data storage requirements. Moreover, most of them use feature matching methods, which are prone to feature point loss in complex climate environments and insufficient matching reliability. Since 2020, implicit representation methods represented by neural radiation fields have received widespread attention due to their continuity, compactness and superior performance. They have brought new ideas to image processing, but there are still problems of large computational complexity and slow running speed. Summary of the invention

[0004] In order to solve the problems that the existing scene matching-based navigation method needs to pre-store ground image data, has high data storage requirements, is prone to feature point loss in complex climate environments, and has insufficient matching reliability, the present invention proposes a supersonic passenger aircraft scene matching navigation method based on neural implicit surfaces.

[0005] The technical solution adopted by the present invention to solve the above problems is: the present invention specifically comprises the following steps:

[0006] Step 1: Collect aerial images near the regular flight routes of passenger aircraft and pre-process the aerial images to obtain the camera pose of each aerial image;

[0007] Step 2: Offline training of the neural implicit surface of the area near the supersonic aircraft route based on the camera pose and the corresponding aerial images;

[0008] Step 3: Use the onboard sensor of the supersonic passenger aircraft to give the initial position, and use the trained neural implicit surface to render the ground image corresponding to the specified position;

[0009] Step 4: Set the position of the passenger plane as a particle, set the difference between the rendered image and the real-time image of the passenger plane as a loss, and calculate the precise position of the supersonic passenger plane on the route through the particle swarm algorithm.

[0010] Preferably, the preprocessing of the aerial image in step 1 specifically includes:

[0011] Feature point extraction, feature point matching and sparse point cloud reconstruction are performed on aerial images to obtain the camera pose of each aerial image.

[0012] Preferably, the neural implicit surface in step 2 is a variant of the neural radiation field, and the neural implicit surface offline training specifically includes:

[0013] Step 2.1: Encode the camera pose and the corresponding aerial image into the neural network as a ground scene;

[0014] Step 2.2: Perform high-frequency encoding on the encoding position x in the neural network to complete the offline training of the neural implicit surface;

[0015] The ground scene is encoded into the neural network as follows:

[0016]

[0017] In formula (1) to formula (3), σ is density, f is internal implicit feature, MLP is multi-layer perceptron, θ is G For geometric representation, θ C For color representation, x is the encoding position, c is the color, d is the direction, S is the signed distance equation and is represented by MLP, Φ S is the sigmoid equation;

[0018] The expression of high frequency coding is:

[0019] r(x)=(sin(2 0 πx),cos(2 0 πx),…,sin(2 L-1 πx),cos(2 L-1 πx)) (4);

[0020] In formula (4), L is the number of high-frequency encoding times.

[0021] Preferably, step 3 specifically includes:

[0022] Step 3.1: At any time period of the supersonic passenger aircraft's flight, use the sensors carried on the supersonic passenger aircraft to calculate the initial position of the supersonic passenger aircraft at the corresponding time period;

[0023] Step 3.2: The trained neural implicit surface of the initial position of the supersonic passenger plane is high-frequency encoded and input into the multi-layer perceptron in the neural radiation field to obtain the volume density and color corresponding to each sampling point of the supersonic passenger plane scene in three-dimensional space;

[0024] Step 3.3: Using a volume rendering algorithm, the volume density and color are rendered into the color of the corresponding pixel of the supersonic passenger aircraft image, thereby obtaining a ground image corresponding to the initial position of the supersonic passenger aircraft;

[0025] Step 3.4: Calculate the difference between the ground image and the real-time ground image using image consistency. If the difference is greater than a preset value, repeat steps 3.1 to 3.3 to recalculate the ground image until the difference is less than a preset value.

[0026] The calculation formula for the color corresponding to each sampling point in the supersonic passenger aircraft scene is:

[0027]

[0028] In formula (5), N S is the number of sampling points, δ i =t i+1 -t i is the distance between sampling points, α i is the rendering weight.

[0029] Preferably, in step 3.4, calculating the difference between the ground image and the real-time captured ground image by using image consistency specifically includes:

[0030] The weighted PSNR, LPIPS and SSIM indicators are used to calculate the difference between the ground image and the real-time ground image, where SNR is the peak signal-to-noise ratio, LPIPS is the learning-perceived image block similarity, and SSIM is the structural similarity.

[0031] Preferably, step 4 specifically includes:

[0032] Step 4.1: Sampling near the initial position of the supersonic passenger aircraft to obtain a candidate position set;

[0033] Step 4.2: Render the image pixels corresponding to each position in the candidate position set through step 2 to obtain the ground image corresponding to each sampling point in the candidate position set;

[0034] Step 4.3: Calculate the difference between the corresponding ground image and the real-time ground image through step 3;

[0035] Step 4.4: Taking the candidate positions as particles and the image difference as the objective function, the particle swarm algorithm is used to calculate the candidate positions for the next iteration;

[0036] Step 4.5: Repeat steps 4.2 to 4.4 until the candidate positions converge and the rendered ground image is exactly the same as the ground image taken in real time. The particle position obtained last time is output as the precise position of the supersonic passenger plane on the route.

[0037] The beneficial effects of the present invention are:

[0038] (1) The present invention provides a supersonic passenger aircraft scene matching navigation method based on neural implicit surface, which does not rely on ground scene feature points, can achieve image registration under complex meteorological conditions or when the ground scene changes, and has better reliability;

[0039] (2) The present invention does not rely on a ground scene library, but uses a neural radiation field to perform implicit scene representation, and stores ground information in a neural network, thereby reducing data storage requirements;

[0040] (3) The present invention proposes a rendering image and captured image matching method based on a particle swarm algorithm, which can achieve fast and accurate image matching and improve navigation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a supersonic passenger aircraft scene matching navigation method based on neural implicit surface provided by the present invention;

[0042] Figure 2 A first ground scene map used by the present invention;

[0043] Figure 3 A second ground scene map used by the present invention;

[0044] Figure 4 A diagram of the neural implicit surface field structure used by the present invention;

[0045] Figure 5 A schematic diagram of the pose estimation result of the first ground scene image used by the present invention;

[0046] Figure 6 A schematic diagram of the pose estimation result of the second ground scene image used by the present invention. DETAILED DESCRIPTION

[0047] Combination Figure 1 To illustrate this embodiment, Figure 1 As shown, the steps of a supersonic passenger aircraft scene matching navigation method based on neural implicit surface described in this embodiment include:

[0048] S1: path scene collection;

[0049] S2: neural surface scene image fitting;

[0050] S201: Extract feature points, match feature points, and reconstruct sparse point clouds from the collected aerial images near the regular routes of passenger aircraft to obtain the camera pose of each image. Use the ground aerial images and camera pose as input to train a neural surface field offline to store ground scene information. The neural implicit surface is a variant of the neural radiant field, which encodes a scene into the neural network through the following formula:

[0051]

[0052] In formula (1) to formula (3), σ is density, f is internal implicit feature, MLP is multi-layer perceptron, θ is G For geometric representation, θ C For color representation, x is the encoding position, c is the color, d is the direction, S is the signed distance equation and is represented by MLP, Φ S is the sigmoid equation;

[0053] S202: In order to better fit the high-frequency information in the image, it is necessary to use formula (4) to perform high-frequency encoding on the position x:

[0054] r(x)=(sin(2 0 πx),cos(2 0 πx),…,sin(2 L-1 πx),cos(2 L-1 πx)) (4);

[0055] In formula (4), L is the number of high-frequency encoding times. In the method of this embodiment, 8 encodings are performed.

[0056] S3: given initial position;

[0057] At a certain moment during the flight of the passenger plane, the initial position of the passenger plane is obtained by using other sensors on board, such as satellite, inertial navigation, and radio navigation.

[0058] S4: Rendering image;

[0059] The initial position of the passenger plane is encoded with high frequency and input into the multi-layer perceptron in the neural radiation field, and the volume density and color corresponding to each point in the three-dimensional space are obtained. Then, the volume density and color are rendered into the color of each pixel of the image using the volume rendering algorithm, so that the ground image corresponding to the initial position is obtained. Given the camera ray r(t) = o + td, where o is the center of the camera, the color can be obtained by formula (5):

[0060]

[0061] In formula (5), T(t) is the cumulative density, t n is the near boundary of the scene, t f is the far boundary of the scene, r(t)=o+td is the given camera ray, o is the camera center, and for the convenience of calculation, the neural implicit surface approximates the integral as a linear combination of samples:

[0062]

[0063] In formula (6), N S is the number of sampling points, δ i =t i+1 -t i is the distance between sampling points, α i is the rendering weight.

[0064] S5: Subtract the rendered image from the captured image;

[0065] Traditional scene matching algorithms need to use algorithms such as LBP (local binary pattern), HOG (histogram of oriented gradients), SIFT (scale-invariant feature transform) to extract feature points of captured images and stored images for mirror matching. However, ground features are difficult to extract under complex meteorological conditions, which can easily lead to matching failure. In this step, the method of this embodiment uses weighted PSNR, LPIPS and SSIM indicators to calculate the difference between the ground image and the real-time captured ground image, where SNR is the peak signal-to-noise ratio, LPIPS is the learning perception of image block similarity, and SSIM is the structural similarity, which has higher reliability. Among them, PSNR represents the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy; LPIPS judges the difference between two images through a neural network trained with a large-scale data set; SSIM measures the similarity of two images from three aspects: brightness, contrast, and structure.

[0066] S6: Optimize position;

[0067] After the initial position of the aircraft is given and the difference between the rendered image and the photographed image is used as the loss, the precise position of the aircraft is calculated using an iterative method. First, a set of candidate positions is obtained by sampling near the initial position of the aircraft; secondly, the image pixels corresponding to each position are rendered through S4. In order to reduce the amount of calculation and increase the operation speed, the resolution of the rendered image can be appropriately reduced; then the difference between the rendered image and the photographed image is calculated according to S5; finally, the candidate position is used as a particle and the image difference is used as the objective function to obtain the candidate position of the next iteration using the particle swarm algorithm. After repeatedly executing S4 and S5, the candidate position will converge and make the photographed image approximate to the rendered image. At this time, the particle position is the precise position of the supersonic aircraft on the route.

[0068] S7: Output position.

[0069] Example

[0070] Combination Figure 2-6 This embodiment is described. The steps of a supersonic passenger aircraft scene matching navigation method based on neural implicit surface described in this embodiment include:

[0071] Step 1: After obtaining the scene below the aircraft path, this embodiment uses two scene aerial image data sets, scene A is as follows: Figure 2 As shown, scene B is Figure 3 As shown, a neural implicit surface field is trained to express it implicitly, such as Figure 4 First, feature extraction, feature matching and sparse point cloud reconstruction are performed on the aerial photography data set to obtain the camera pose. This can usually be done using the 3D reconstruction software COLMAP. The camera pose of scene A used in this embodiment is as follows: Figure 5 As shown, the camera pose of scene B is Figure 6 Then construct Figure 4 The neural implicit surface field model shown in the figure includes a geometric expression neural network MLP θG and color expression neural network MLP θC , both use fully connected layers to store the 3D geometry and color information of the ground scene. Figure 4 Here x represents the 3D position of the sampling point, f represents the value of the signed distance equation, d represents the direction of the camera at the sampling point, n represents the normal direction, and R, G, and B represent red, green, and blue, respectively. The data is then implicitly expressed using a neural implicit surface field, including spatial 3D point sampling, high-frequency encoding of sampling point coordinates, sampling point density color mapping, volume rendering, color difference calculation, and loss back propagation.

[0072] Step 2, based on the trained neural implicit surface, given the initial rough three-dimensional coordinates of a navigation position, the volume rendering method is used to give the rendered image corresponding to the above position. Specifically, it includes using sine / cosine functions to perform high-frequency encoding on the input coordinates, inputting the high-frequency signal into the geometric expression neural network to obtain the volume density of the spatial point, and inputting the high-frequency signal into the color expression neural network to obtain the color of the spatial point. Three-dimensional point sampling is performed on the line formed by the image pixel and the camera center, and the color corresponding to the pixel is obtained by using the volume rendering method based on the density and color of the sampling point. The image can be formed by obtaining the colors corresponding to all pixels.

[0073] Step 3, based on the rendered image of the given rough position and the real image of the aircraft taken at the navigation position, the particle swarm algorithm is used to search for the precise position of the aircraft. First, candidate points are randomly set near the given rough position to initialize the particle swarm algorithm. Then, the image corresponding to each candidate point position is rendered using the method proposed in step 2, and then the difference between the rendered image and the actual image is calculated by weighted PSNR, LPIPS and SSIM indicators, and used as the loss of iterative optimization of the particle swarm algorithm. Finally, the particle swarm algorithm is iterated multiple times to reduce the difference between the rendered image and the actual image. At this time, the position of the particles is the estimated position of the aircraft.

[0074] In this embodiment, 5 target navigation points are set for each scene, and the obtained position estimation results are shown in Table 1 and Table 2.

[0075] Table 1

[0076] point 1 2 3 4 5 True Value [-1.00,0.57,0.00] [1.00,0.57,0.00] [-1.00,-0.57,0.00] [1.00,-0.57,0.00] [-1.00,-0.06,0.00] Valuation [-1.03,0.58,+0.01] [0.88,0.74,-0.00] [-0.88,-0.73,0.01] [1.01,-0.41,0.01] [-0.99,-0.24,0.01]

[0077] Table 2

[0078] point 1 2 3 4 5 True Value [-1.00,0.57,0.00] [1.00,0.57,0.00] [-1.00,-0.57,0.00] [1.00,-0.57,0.00] [-1.18,-0.06,0.00] Valuation [-1.08,0.39,0.00] [0.89,0.75,0.02] [-0.95,-0.63,-0.02] [1.04,-0.50,0.02] [-0.97,-0.25,-0.03]

[0079] From the results in Table 1 and Table 2, it can be seen that the present embodiment can accurately estimate the position of a supersonic passenger aircraft using ground images. At the same time, the method of the present embodiment has the advantages of strong robustness to ground features and small storage space requirements.

[0080] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A supersonic passenger aircraft scene matching navigation method based on neural implicit surface, characterized in that: The steps of the supersonic passenger aircraft scene matching navigation method based on neural implicit surface include: Step 1: Collect aerial images near the regular flight routes of passenger aircraft and pre-process the aerial images to obtain the camera pose of each aerial image; Step 2: Offline training of the neural implicit surface of the area near the supersonic aircraft route based on the camera pose and the corresponding aerial images; Step 3: Use the onboard sensor of the supersonic passenger aircraft to give the initial position, and use the trained neural implicit surface to render the ground image corresponding to the specified position; Step 4: Set the position of the passenger plane as particles, set the difference between the rendered image and the real-time image of the passenger plane as loss, and calculate the precise position of the supersonic passenger plane on the route through the particle swarm algorithm.

2. The method for supersonic passenger aircraft scene matching navigation based on neural implicit surface according to claim 1, characterized in that: The preprocessing of the aerial image in step 1 specifically includes: Feature point extraction, feature point matching and sparse point cloud reconstruction are performed on aerial images to obtain the camera pose of each aerial image.

3. The method for supersonic passenger aircraft scene matching navigation based on neural implicit surface according to claim 1, characterized in that: The neural implicit surface in step 2 is a variant of the neural radiation field. The offline training of the neural implicit surface specifically includes: Step 2.1: Encode the camera pose and the corresponding aerial image into the neural network as a ground scene; Step 2.2: Perform high-frequency encoding on the encoding position x in the neural network to complete the offline training of the neural implicit surface; The ground scene is encoded into the neural network as follows: In formula (1) to formula (3), σ is density, f is internal implicit feature, MLP is multi-layer perceptron, θ is G For geometric representation, θ C For color representation, x is the encoding position, c is the color, d is the direction, S is the signed distance equation and is represented by MLP, Φ S is the sigmoid equation; The expression of high frequency coding is: r(x)=(sin(2 0 πx),cos(2 0 πx),…,sin(2 L-1 πx),cos(2 L-1 πx)) (4); In formula (4), L is the number of high-frequency encoding times.

4. The method for supersonic passenger aircraft scene matching navigation based on neural implicit surface according to claim 1, characterized in that: Step 3 specifically includes: Step 3.1: At any time period of the supersonic passenger aircraft's flight, use the sensors carried on the supersonic passenger aircraft to calculate the initial position of the supersonic passenger aircraft at the corresponding time period; Step 3.2: The trained neural implicit surface of the initial position of the supersonic passenger plane is high-frequency encoded and input into the multi-layer perceptron in the neural radiation field to obtain the volume density and color corresponding to each sampling point of the supersonic passenger plane scene in three-dimensional space; Step 3.3: Using a volume rendering algorithm, the volume density and color are rendered into the color of the corresponding pixel of the supersonic passenger aircraft image, thereby obtaining a ground image corresponding to the initial position of the supersonic passenger aircraft; Step 3.4: Calculate the difference between the ground image and the real-time ground image using image consistency. If the difference is greater than a preset value, repeat steps 3.1 to 3.3 to recalculate the ground image until the difference is less than a preset value. The calculation formula for the color corresponding to each sampling point in the supersonic passenger aircraft scene is: In formula (5), N S is the number of sampling points, δ i =t i+1 -t i is the distance between sampling points, α i is the rendering weight.

5. The method for supersonic passenger aircraft scene matching navigation based on neural implicit surface according to claim 4, characterized in that: In step 3.4, the difference between the ground image and the real-time ground image is calculated by using image consistency, which specifically includes: The weighted PSNR, LPIPS and SSIM indicators are used to calculate the difference between the ground image and the real-time ground image, where SNR is the peak signal-to-noise ratio, LPIPS is the learning-perceived image block similarity, and SSIM is the structural similarity.

6. The method for supersonic passenger aircraft scene matching navigation based on neural implicit surface according to claim 1, characterized in that: Step 4 specifically includes: Step 4.1: Sampling near the initial position of the supersonic passenger aircraft to obtain a candidate position set; Step 4.2: Render the image pixels corresponding to each position in the candidate position set through step 2 to obtain the ground image corresponding to each sampling point in the candidate position set; Step 4.3: Calculate the difference between the corresponding ground image and the real-time ground image through step 3; Step 4.4: Taking the candidate positions as particles and the image difference as the objective function, the particle swarm algorithm is used to calculate the candidate positions for the next iteration; Step 4.5: Repeat steps 4.2 to 4.4 until the candidate positions converge and the rendered ground image is exactly the same as the ground image taken in real time. The particle position obtained last time is output as the precise position of the supersonic passenger plane on the route.

Images