A vector-based aircraft scene matching navigation method and system

Abstract

The application discloses a kind of vector-based aircraft scene matching navigation method and system, including before flight, training semantic segmentation neural network, obtain the vector data of target area, network model and vector data are loaded to aircraft;After aircraft enters scene matching area, vector data is projected, and form multiple view reference map;Real-time graph is carried out semantic segmentation, and the edge of each type of ground object is extracted;Each reference map is matched with real-time graph;Real-time graph is carried out Euclidean distance transformation;According to the distance between edge iterative calculation camera pose parameter is matched with edge, and then realize the positioning of aircraft.The application uses vector data to make reference map, with the advantages of small data quantity, can be lossless transformation;Using the height and category information of ground object can improve matching precision;Matching algorithm has good robustness to the increase, decrease and misplacement of edge, and can adapt to the matching between multi-view heterogeneous images.

Description

Technical Field

[0001] This invention belongs to the field of autonomous navigation technology for aircraft, specifically relating to a vector-based aircraft scene matching navigation method and system. Background Technology

[0002] In recent years, unmanned aerial vehicles (UAVs) have played a significant role in many fields such as military and policing, and their navigation and positioning technology in complex environments has received considerable attention. Because satellite navigation can be rendered ineffective by enemy electromagnetic interference during missions, and the errors in inertial navigation systems accumulate over time, scene matching techniques are commonly used to correct inertial navigation system errors and achieve precise autonomous navigation for the aircraft.

[0003] Traditional scene matching technology typically involves storing raster image data of the target area on the aircraft as a reference map beforehand. When flying to the scene matching area, the camera captures images of the ground as a real-time image. The two images are then registered to determine the aircraft's position. Current technology has the following problems: 1) The storage space required for raster data as a reference map is too large, while the aircraft's storage space is limited. A single reference map can only represent a single viewpoint or orthographic projection. If the aircraft travels from other views due to obstacle avoidance or other requirements, it will lead to cross-view matching. In this case, the real-time image will undergo complex changes due to factors such as changes in ground feature occlusion and projection distortion, resulting in matching failure. Increasing the reference map's viewpoint and range will exponentially increase the data volume and mapping difficulty, also limiting the aircraft's flight range and hindering flexible trajectory planning; 2) The reference map only contains the horizontal coordinates and grayscale values ​​of the ground scene. 1) The matching algorithm lacks information on the type and elevation of ground features, resulting in limited information dimensions; 2) The matching model used is too simplistic, such as using homography to describe the relationship between different images. However, homography requires that the targets be on the same plane, while various ground features often have different elevations, leading to significant errors in the method. This is especially true for low-flying aircraft like cruise missiles, where the elevation variations of ground features cannot be ignored; 3) Traditional matching algorithms treat all matching points as equivalent, without considering the impact of ground feature categories on the matching. For example, buildings and roads are relatively stable, while the location and grayscale characteristics of vegetation and water bodies are highly susceptible to change.

[0004] Considering these issues, using vector data to create benchmark maps has become a new approach because vector data has a small data volume, can be losslessly transformed between different viewpoints, and can record elevation and feature category information. In recent years, the development of semantic segmentation technology based on deep learning has also made it possible to quickly and accurately extract feature edges from real-time images. However, in the process of matching feature edges, it is difficult to construct explicit correspondences between points as in traditional raster images. Therefore, how to perform matching using heterogeneous image edges that are added, removed, or misaligned is a major challenge. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a vector-based aircraft scene matching navigation method, comprising the following steps:

[0006] Step 1: Train the semantic segmentation neural network, obtain vector data of the target region, and load the trained semantic segmentation neural network and vector data onto the aircraft;

[0007] Step 2: After the aircraft enters the scene matching area, multiple predicted camera positions are selected at intervals within the estimated scene matching area. The vector data pre-loaded on the aircraft is projected onto the predicted real-time map view to form a reference map with multiple views.

[0008] Step 3: Use the trained semantic segmentation neural network to segment the real-time image captured by the aircraft and extract the edges of various ground features;

[0009] Step 4: Perform a coarse match between the baseline image of each viewpoint and the real-time image from which the ground feature edges were extracted in Step 3. Calculate the baseline image with the highest matching degree according to the edge category and distance to obtain the approximate position of the aircraft at the moment the real-time image was captured.

[0010] Step 4.1: Set the sampling line spacing and tolerance distance;

[0011] Step 4.2: Generate arrays of horizontal and vertical edge positions for the real-time image and each reference image at the sampling line location;

[0012] Step 4.3: Match the real-time image with each baseline image and calculate the matching score for each baseline image;

[0013] Step 4.4: Sort the scores of each benchmark image and obtain the (X) corresponding to the benchmark image with the highest score. S Y S This serves as the approximate location of the aircraft captured in real-time images.

[0014] Step 5: Perform Euclidean distance transformation on the real-time images captured by the aircraft;

[0015] Step 6: Based on the distance between the edge of the reference image projected onto the real-time image view and the edge of the real-time image after Euclidean distance transformation, the camera pose parameters are iteratively calculated according to the least squares method to perform fine registration of the edge, thereby achieving the positioning of the aircraft.

[0016] Step 6.1: Select ground control points in the vector data and project them onto the real-time map view to obtain the projected points;

[0017] Step 6.2: Optimize the distance from the projection point to the edge of the same category in the real-time image using the least squares method, and iteratively calculate the camera pose parameters.

[0018] Furthermore, in step 1, a suitable semantic segmentation neural network model and dataset are selected based on the type of aircraft and mission requirements, and the semantic segmentation neural network model is trained using the dataset. Imagery of the target area is acquired using remote sensing satellites or reconnaissance drones, and then digital photogrammetry, manual mapping, and supervised classification mapping techniques are used to create vector data containing horizontal coordinate information, elevation information, and ground feature category information, assigning category numbers to each type of ground feature. The trained semantic segmentation neural network model and vector data are then loaded onto the aircraft.

[0019] Furthermore, in step 2, after the aircraft enters the scene matching zone, within the horizontal coordinate range of the matching zone predicted by the inertial navigation system, starting from one corner of the predicted matching zone, an X-axis position is selected every ss in the X direction. s Value, a Y value is selected every ss in the Y direction. s Values, forming multiple pairs (X) s Y s The value is used as the predicted camera position, where ss is the set spacing value, and X is the distance between the points. s Y s The x and y coordinates of the predicted camera center in the geographic coordinate system.

[0020] The vector data pre-loaded on the aircraft is projected onto the predicted real-time map view to form a reference map. The specific process is as follows:

[0021] ① For each pair (X) S Y S Based on the camera attitude given by the inertial navigation system and the aircraft altitude given by the altimeter, ground feature vector data within a certain range is extracted.

[0022] The method for calculating the cropping range is as follows: Substitute the image plane coordinates (x, y) of the points on the real-time image border captured by the aircraft into formulas (1)-(2), calculate the coordinates (X, Y) of the corresponding ground points in the geographic coordinate system, and obtain the coordinates of the cropping range border. The formula is:

[0023]

[0024]

[0025] In the formula, x0 and y0 are the image plane coordinates of the principal point, f is the focal length of the camera, and X... S Y S Z S Z represents the coordinates of the camera center in the geographic coordinate system, and Z is the ground elevation at (X, Y) estimated from elevation data. S For the aircraft altimeter (X) S Y SThe corresponding heights, a1, a2, a3, b1, b2, b3, c1, c2, and c3, are determined by the camera attitude angles. The formula for calculating the nine direction cosines composed of ω and κ is as follows:

[0026]

[0027] Since the value of (X, Y) is uncertain, an approximate elevation value needs to be assumed first. The ground coordinates (X, Y) are then calculated by substituting them into the collinearity equation formulas (1)-(2). The elevation at (X, Y) is then interpolated from the elevations of nearby points. This process is repeated multiple times until the error between two consecutive (X, Y) values ​​is less than the set limit. The boundary of the obtained ground point range is then extended outward by a certain distance before vector data is extracted.

[0028] ② Project the captured vector data onto the predicted real-time viewpoint to form a reference image B0. The predicted real-time image is the viewpoint of the aircraft's downward-looking camera at position (X). S Y S The real-time image taken at point X follows these rules during projection: the projection order is from far to near, meaning the elements are projected from the camera center (X). S Y S Z S The farther away the feature, the earlier it appears in the projection order; the edges of features are projected first, then their internal pixels are filled; features projected later will cover features projected earlier; the pixel value of a pixel on the reference map is set to the category number of the feature it is covered by. The following collinearity equation is used for projection:

[0029]

[0030]

[0031] In the formula, x and y are the image plane coordinates of the projected vector feature, and X... A Y A Z A These are the coordinates of vector features in the geographic coordinate system.

[0032] ③ Extract the edges of various land features in the projected reference map B0 to obtain reference map B. The specific method is as follows:

[0033] First, generate an edge-modified reference map B with the same size as the projected reference map B0, and initialize all pixel values ​​of B to 0. Then, traverse the pixels of reference maps B and B0 in left-to-right and top-to-bottom order, and perform the following processing: if B(i,j)≠0, skip the position and do not process it; if B0(i,j)≠B0(i+1,j), let B(i,j)=B0(i,j) and B(i+1,j)=B0(i+1,j); if B0(i,j)≠B0(i,j+1), let B(i,j)=B0(i,j) and B(i,j+1)=B0(i,j+1); otherwise, skip the position and do not process it.

[0034] Thus, for each pair (X) S Y S A baseline map B was established for each of them, resulting in baseline maps from multiple perspectives.

[0035] Furthermore, in step 3, the real-time image captured by the aircraft's downward-looking camera is input into the trained semantic segmentation neural network model for segmentation processing to obtain a real-time image T0. The edges of various ground features in the real-time image T0 are extracted to obtain a real-time image T. The edge extraction method is the same as the method for extracting ground feature edges from the reference image B0 in step 2.

[0036] Furthermore, the method for generating the horizontal and vertical edge position arrays of the reference image in step 4.2 is as follows: ① Generate the horizontal edge position array of the real-time image T by taking one row from top to bottom every sl rows as a horizontal sampling line in the real-time image T, traversing the pixels of that row from left to right. If the pixel value at a certain position is not 0 and the pixel value to its right is 0, then add the column number of that position to the edge position array of that row. If several consecutive positions have pixel values ​​that are not 0, then take the median of their column numbers and add it to the edge position array of that row; ② Generate the vertical edge position array of the real-time image T by taking one column from left to right every sl columns as a vertical sampling line in the real-time image T, traversing the pixels of that column from top to bottom. If the pixel value at a certain position is not 0 and the pixel value below it is 0, then add the row number of that position to the edge position array of that column. If several consecutive positions have pixel values ​​that are not 0, then take the median of their row numbers and add it to the edge position array of that column; ③ Generate each reference image B i The horizontal edge position array is generated using the same method as generating the horizontal edge position array for the real-time image T; ④ Generate each reference image B i The method for generating the vertical edge position array is the same as that for generating the vertical edge position array of the real-time graph T.

[0037] Furthermore, in step 4.3, for each sampling line l, if its corresponding real-time image edge position array t l If it is not empty, then it is considered valid, and denoted as t. lLet m be the number of elements, and let b be the array representing the corresponding edge positions of the reference map. l The number of elements is n, and the traversal of t is... l For each element k in b, l Find a k′ among the elements that satisfies the following condition:

[0038] B i (l, k′) = T(l, k) (6)

[0039] In the formula, B i (l, k′) refers to the reference map B corresponding to sampling line l and digital k′. i The pixel value at the location, T(l,k), refers to the pixel value at the location T in the real-time image corresponding to the sampling line l and the number k. When the sampling line is horizontal, k corresponds to the column number; when the sampling line is vertical, k corresponds to the row number.

[0040] Among all k′ that satisfy the condition, select the k′ with the smallest |k′-k|. If |k′-k|≤w, assign a score of w-|k′-k| to k, considering element k a successful match; otherwise, assign a score of 0 to k, considering element k a failed match. After traversing t... l After determining the elements, calculate the score for that sampling line. l The calculation formula is:

[0041]

[0042] In the formula, s k For each k, s represents the score; s is the number of k elements that are successfully matched on the sampling line l; w is the tolerance distance; and m is the real-time image edge position array t. l The number of elements, where n is the array of edge positions of the baseline image b. l The number of elements.

[0043] The score of all valid sampling lines l The mean value is the score of the baseline chart, calculated using the following formula:

[0044]

[0045] In the formula, n_l is the number of valid sampling lines.

[0046] Furthermore, in step 5, the distance transformation range w_d is set, and a square distance transformation template m with a side length of (2w_d+1) is generated. d , where m d The value at each position is its Euclidean distance to the center of the template. The formula for calculating the Euclidean distance d from point (x1, y1) to point (x2, y2) is:

[0047]

[0048] For each land cover category c present in the real-time map T, generate a distance transformation map T with w_d pixels of boundary padding applied to T. Dc and T Dc All pixel values ​​are set to T Dc Diagonal length M. Real-time graph T, distance transformation graph T Dc Distance transformation template m d Stack T and T Dc Align the pixel with center, then iterate through the pixels in T, and denote each pixel with a value of c as T. ci For each T ci Perform the following transformation: Transform template m d The center and T ci Alignment, comparison m d Each value and the corresponding position of T Dc The magnitude of the value of m d If the value is small, then put T Dc Replace the value here with m d The value of .

[0049] Furthermore, in step 6.1, the camera pose vector is set. The initial value, where (X) S Y S Let Z be the coarse matching result from step 4. S For the aircraft altimeter (X) S Y S The corresponding height, ω, κ are the inertial navigation system (X S Y S The corresponding value; in step 2, for (X) S Y S In the extracted ground feature vector data, sampling lines with equal intervals in the X and Y directions are set, and the set of intersection points between these lines and the vector data contour lines is taken as the ground control point set x. b Based on the value of p, the vector data is projected onto the real-time view using the projection method in step 2. Here, the real-time view is the position (X) of the aircraft's downward-looking camera obtained through coarse matching in step 4. S Y S A real-time image taken at location ) and x b The projected set of points is denoted as the projected set of ground control points x′. b If a ground control point is covered by other elements after projection, then that point is not counted.

[0050] Furthermore, in step 6.2, if the number of projection points of the ground control points is denoted as n_b, then the nearest edge distance vector d(p) is a vector with n_b rows, and each element of d(p) represents a projection point. The Euclidean distance to the nearest edge of the same type in the real-time graph is calculated using the following formula:

[0051] d i =T Dc (x′ i y′ i (10)

[0052] In the formula, (x′ i y′ i ) represents the i-th ground control point The coordinates T projected onto the real-time graph Dc Is with Distance transformation graph corresponding to category number, T Dc (x′ i y′ i ) represents T Dc (x′) i y′ i The value at position ).

[0053] d i Let T be the value of the i-th element in d(p), if T Dc (x′ i y′ i ) is M or T Dc In (x′) i y′ i If a ground control point has a value of M in its 4-neighborhood, then that ground control point is discarded and will not be included in subsequent calculations.

[0054] The Jacobian matrix J of d(p) is calculated using the following formula:

[0055]

[0056] In the formula, J i Let J be the value of the i-th row of matrix J corresponding to the i-th ground control point. and It is x′ i y′ i Substituting the derivatives of x and y with respect to each element in p into the collinearity equation, T Dc This is the distance transformation graph of the category corresponding to this point, T. Dc In (x′) i y′ i The partial derivative at () can be approximated by the following formula:

[0057]

[0058]

[0059] Based on the reliability and stability of various land features, weights are assigned to the points corresponding to each type of land feature, forming a diagonal matrix P. The correction value Δp of parameter p is calculated using the least squares method, with the following formula:

[0060] Δp=-(J T PJ) -1 J T Pd(p) (14)

[0061] The value of parameter p is adjusted to p′, and the formula is:

[0062] p′=p+Δp (15)

[0063] Then, determine whether the correction values ​​of each parameter in Δp exceed the set limit. If they exceed the limit, return to step 6.1, use p′ as the new initial value of p, repeat the subsequent process, and increment the iteration count by 1. If they do not exceed the limit or the iteration count reaches the set upper limit, output p′ to obtain the camera pose at the moment of real-time image capture.

[0064] By adding the relative pose parameters of the aircraft and the camera to p′, the position and attitude of the aircraft at the moment of real-time image capture can be obtained, thus realizing the navigation function.

[0065] The present invention also provides a vector-based aircraft scene matching navigation system for implementing the vector-based aircraft scene matching navigation method described above.

[0066] Furthermore, it includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a vector-based aircraft scene matching navigation method as described above.

[0067] Compared with the prior art, the present invention has the following advantages:

[0068] 1) This invention uses vector data instead of traditional raster images to create a reference map, which reduces the amount of data by a lot, allows for lossless transformation between different viewpoints, enables multi-view and multi-dimensional matching, and increases the success rate of matching in complex combat environments.

[0069] 2) This invention considers the elevation and land feature category information in vector data, which improves the shortcomings of traditional methods that treat the ground as a plane and ignore the undulation and mutual occlusion of land features. It prevents mismatch between the edges of different categories and weights different categories according to the stability of land features, making the results more accurate and reliable.

[0070] 3) This invention has excellent robustness to the addition, subtraction and misalignment of edges between heterogeneous images, and can also match correctly under abnormal conditions with large edge differences;

[0071] 4) This invention does not require establishing a clear correspondence between points, which solves the problem of finding corresponding point pairs between edges of heterogeneous images and can well adapt to the registration of edges of heterogeneous images.

[0072] In summary, this invention overcomes the drawbacks of using raster data as a reference map, and utilizes the characteristics of vector data (small data volume, lossless transformation) and elevation and ground feature category information. It has good robustness to the addition, subtraction, and misalignment of edges between heterogeneous images, and can achieve accurate registration and positioning of aircraft from multiple perspectives, which has significant value in military, police and other fields. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0074] Figure 1 This is a flowchart of a vector-based aircraft scene matching navigation method according to an embodiment of the present invention.

[0075] Figure 2 This is a projection diagram of an embodiment of the present invention.

[0076] Figure 3 This is an example diagram of a distance transformation template in an embodiment of the present invention.

[0077] Figure 4 This is a flowchart illustrating the fine registration process in an embodiment of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0079] Example 1

[0080] This embodiment mainly involves three coordinate systems: vector data and the final calculated positions of the aircraft and camera are represented using a three-dimensional geographic coordinate system, with the unit being meters; when points on the image need to be projected onto the ground data, an image plane coordinate system is used, with the unit being meters; and when performing image processing operations such as traversing and assigning values ​​to the image by pixels, an image pixel coordinate system is used, with the unit being pixels.

[0081] In this embodiment, the target area refers to all areas that the aircraft may pass through, that is, the wide area from the aircraft's takeoff point to its destination; the scene matching area is an area in the target area that is rich in information, has obvious features, and can be matched; the matching area predicted by the inertial navigation system refers to a region that is useful for the current scene matching, which is selected from the scene matching area based on the position predicted by the inertial navigation system during the flight; that is, the target area includes the scene matching area, and the scene matching area includes the matching area predicted by the inertial navigation system.

[0082] like Figure 1 As shown, this embodiment of the invention provides a vector-based aircraft scene matching navigation method, which includes the following steps:

[0083] Step 1: Train the semantic segmentation neural network, obtain vector data of the target region, and load the trained semantic segmentation neural network and vector data onto the aircraft.

[0084] Based on the type of aircraft and mission requirements, a suitable semantic segmentation neural network model and dataset are selected, and the model is trained using the dataset. For example, for extremely fast aircraft like cruise missiles, a real-time semantic segmentation neural network model with fast inference speed, such as BiseNet, is chosen; for slower-moving UAVs requiring precise positioning, a semantic segmentation neural network model with slower inference speed but more accurate results, such as U-Net, is used. This embodiment simulates scene matching for cruise missiles, using the BiseNet network model. Since cruise missiles often fly at low altitudes to evade enemy radar, a semantic segmentation dataset of aerial images with a spatial resolution comparable to their camera is selected for training.

[0085] Before executing the mission, images of the target area are acquired using remote sensing satellites or reconnaissance drones. These images are then processed using digital photogrammetry, manual mapping, and supervised classification methods to create vector data containing horizontal coordinates, elevation information, and feature category information. If existing data is sufficiently reliable, previously generated data can be used directly. Category information is essential in the vector data; each type of feature is assigned a category number, for example, buildings are categorized as 1, roads as 2, water bodies as 3, vegetation as 4, and others as 5. Buildings also require elevation information. For buildings whose elevations cannot be determined, their height can be estimated based on the number of floors. Other features close to the ground can be estimated based on the local average elevation.

[0086] The trained semantic segmentation neural network model and vector data are loaded onto the aircraft.

[0087] Step 2: After the aircraft enters the scene matching zone, multiple predicted camera positions are selected at intervals within the estimated scene matching zone. The vector data pre-loaded on the aircraft is projected onto the predicted real-time map view to form a reference map with multiple views.

[0088] After entering the scene matching zone, within the horizontal coordinate range of the matching zone predicted by the inertial navigation system, starting from one corner of the predicted matching zone, an X-axis position is selected every ss in the X direction. s Value, a Y value is selected every ss in the Y direction. s Values, forming multiple pairs (X) s Y s The value is used as the predicted camera position, where ss is the set spacing value, and X is the distance between the points. s Y s The x and y coordinates of the predicted camera center in the geographic coordinate system.

[0089] The vector data pre-loaded on the aircraft is projected onto the predicted real-time map view to form a reference map. The specific process is as follows:

[0090] ① For each pair (X) S Y S Based on the camera attitude given by the inertial navigation system and the aircraft altitude given by the altimeter, ground feature vector data within a certain range is extracted;

[0091] The method for calculating the cropping range is as follows: Substitute the image plane coordinates (x, y) of the points on the real-time image border captured by the aircraft into formulas (1)-(2), calculate the coordinates (X, Y) of the corresponding ground points in the geographic coordinate system, and obtain the coordinates of the cropping range border. The formula is:

[0092]

[0093]

[0094] In the formula, x0 and y0 are the image plane coordinates of the principal point, f is the focal length of the camera, and X... S Y S Z S Z represents the coordinates of the camera center in the geographic coordinate system, and Z is the ground elevation at (X, Y) estimated from elevation data. S Given from the altimeter on the aircraft, a1, a2, a3, b1, b2, b3, c1, c2, and c3 are determined by the camera attitude angles. The formula for calculating the nine direction cosines composed of ω and κ is as follows:

[0095]

[0096] Since the value of (X, Y) is uncertain, an approximate elevation value needs to be assumed first. The ground coordinates (X, Y) are then calculated by substituting them into the collinearity equation formulas (1)-(2). The elevation at (X, Y) is then interpolated from the elevations of nearby points. This process is repeated multiple times until the error between two consecutive (X, Y) values ​​is less than the set limit. Because the intercepted range calculated in this way has an error compared to the actual range that can be captured in the real-time image, the boundary of the obtained ground point range is extended outward by a certain distance before vector data interception is performed.

[0097] ② Project the captured vector data onto the predicted real-time image view (the aircraft's downward-looking camera at position (X)). S Y S The real-time image taken at point X forms the reference image B0. The projection process follows these rules: the projection order is from far to near, that is, the elements are projected from the center of the camera (X). S Y S Z S The farther away the feature, the earlier it appears in the projection order; the edges of features are projected first, then their internal pixels are filled; features projected later will cover features projected earlier; the pixel value of a pixel on the reference map is set to the category number of the feature it is covered by; the following collinearity equation is used for projection:

[0098]

[0099]

[0100] In the formula, x and y are the image plane coordinates of the projected vector feature, and X... A Y A Z A These are the coordinates of vector features in the geographic coordinate system.

[0101] Projection methods for vector data, such as Figure 2 As shown, the size of the real-time graph has been exaggerated for visual clarity.

[0102] ③ Extract the edges of various land features in the projected reference map B0 to obtain reference map B. The specific method is as follows:

[0103] First, generate an edge-modified reference map B with the same size as the projected reference map B0, and initialize all pixel values ​​of B to 0. Then, traverse the pixels of reference maps B and B0 in left-to-right and top-to-bottom order, and perform the following processing: if B(i,j)≠0, skip the position and do not process it; if B0(i,j)≠B0(i+1,j), let B(i,j)=B0(i,j) and B(i+1,j)=B0(i+1,j); if B0(i,j)≠B0(i,j+1), let B(i,j)=B0(i,j) and B(i,j+1)=B0(i,j+1); otherwise, skip the position and do not process it.

[0104] Thus, for each pair (X) S Y S A baseline map B was established for each of them, resulting in baseline maps from multiple perspectives.

[0105] Step 3: Use the trained semantic segmentation neural network to segment the real-time image captured by the aircraft and extract the edges of various ground features.

[0106] The real-time image captured by the aircraft's downward-looking camera is input into the trained semantic segmentation neural network model for segmentation processing to obtain real-time image T0. The edges of various ground features in real-time image T0 are extracted to obtain real-time image T. The edge extraction method is the same as the method for extracting ground feature edges from the reference image B0 in step 2.

[0107] Step 4: Perform a coarse match between the baseline image of each viewpoint and the real-time image from which the ground feature edges were extracted in Step 3. Calculate the baseline image with the highest matching degree according to the edge category and distance to obtain the approximate position of the aircraft at the moment the real-time image was captured.

[0108] Step 4.1: Set the sampling line spacing sl and tolerance distance w.

[0109] A larger sampling line spacing (sl) results in faster matching, but an excessively large sl can lead to reduced accuracy. A larger tolerance distance (w) increases the tolerance for edge misalignment in coarse matching, but increasing w slows down the matching speed and reduces accuracy. Therefore, these two parameters need to be set according to computer performance and task requirements. In this example, experiments showed that setting sl to 1 / 20 of the real-time image side length and w to 1 / 40 of the real-time image side length yielded better overall results.

[0110] Step 4.2: Generate a real-time image T and each reference image B at the sampling line location. i The array of horizontal and vertical edge positions.

[0111] The specific methods are as follows: ① Generate the horizontal edge position array of the real-time image T. The method is as follows: Take one row from each of the sl rows from top to bottom in the real-time image T as a horizontal sampling line. Traverse the pixels of the row from left to right. If the pixel value at a certain position is not 0 and the pixel value to its right is 0, then add the column number of that position to the edge position array of that row. If several consecutive positions have pixel values ​​that are not 0, then take the median of their column numbers and add it to the edge position array of that row. ② Generate the vertical edge position array of the real-time image T. The method is as follows: Take one column from each of the sl columns from left to right in the real-time image T as a vertical sampling line. Traverse the pixels of the column from top to bottom. If the pixel value at a certain position is not 0 and the pixel value below it is 0, then add the row number of that position to the edge position array of that column. If several consecutive positions have pixel values ​​that are not 0, then take the median of their row numbers and add it to the edge position array of that column. ③ Generate each reference image B. i The horizontal edge position array is generated using the same method as generating the horizontal edge position array for the real-time image T; ④ Generate each reference image B i The method for generating the vertical edge position array is the same as that for generating the vertical edge position array of the real-time graph T.

[0112] Step 4.3, compare the real-time graph T with each baseline graph B. i All are matched, and a matching score is calculated for each baseline image.

[0113] For each sampling line l, if its corresponding real-time image edge position array t l If it is not empty, then it is considered valid, and denoted as t. l Let m be the number of elements, and let b be the array representing the corresponding edge positions of the reference map. l The number of elements is n, and the traversal of t is... l For each element k in b, l Find a k′ among the elements that satisfies the following condition:

[0114] B i (l, k′) = T(l, k) (6)

[0115] In the formula, B i (l, k′) refers to the reference map B corresponding to sampling line l and digital k′. i The pixel value at the location, T(l,k), refers to the pixel value at the location T in the real-time image corresponding to the sampling line l and the number k. When the sampling line is horizontal, k corresponds to the column number; when the sampling line is vertical, k corresponds to the row number.

[0116] Among all k′ that satisfy the condition, select the k′ with the smallest |k′-k|. If |k′-k|≤w, assign a score of w-|k′-k| to k, considering element k a successful match; otherwise, assign a score of 0 to k, considering element k a failed match. After traversing t... lAfter determining the elements, calculate the score for that sampling line. l The calculation formula is:

[0117]

[0118] In the formula, s k For each k, s represents the score; s is the number of k elements that are successfully matched on the sampling line l; w is the tolerance distance; and m is the real-time image edge position array t. l The number of elements, where n is the array of edge positions of the baseline image b. l The number of elements.

[0119] Equation (5) can be used to ignore unstable edge pixels classified as vegetation, because they may differ greatly between heterogeneous images, and coarse matching can be achieved by relying on other edges alone.

[0120] The score of all valid sampling lines l The mean value is the score of the baseline chart, calculated using the following formula:

[0121]

[0122] In the formula, n_l is the number of valid sampling lines.

[0123] Step 4.4, for each reference map B i Sort the scores and take the benchmark B with the highest score. i Corresponding (X) S Y S (This is used as a real-time image to capture the approximate location of the aircraft at the moment of the instant.)

[0124] Step 5: Perform Euclidean distance transformation on the real-time image captured by the aircraft.

[0125] Set the distance transformation range w_d, and generate a square distance transformation template m with a side length of (2w_d+1). d , where m d The value at each position is its Euclidean distance to the template. The formula for calculating the Euclidean distance d from point (x1, y1) to point (x2, y2) is:

[0126]

[0127] Figure 3 For w d Distance transformation template m when = 5 d For each land cover category c present in the real-time map T, generate a distance-transformed map T with w_d pixels of boundary padding applied to T. Dc and T DcAll pixel values ​​are set to the maximum value M. In this embodiment, M is set to the diagonal length of the real-time image. The real-time image T and the distance transformation image T are... Dc Distance transformation template m d Stack T and T Dc Align the pixel with the center of T, then iterate through the pixels in T, and denot the pixel with value c as T. ci For each T ci Perform the following transformation: Transform template m d The center and T ci Alignment, comparison m d Each value and the corresponding position of T Dc The magnitude of the value of m d If the value is small, then put T Dc Replace the value here with m d The value of .

[0128] Step 6: Based on the distance between the edge of the reference image projected onto the real-time image view and the edge of the real-time image after Euclidean distance transformation, the camera pose parameters are iteratively calculated according to the least squares method to perform fine registration of the edge, thereby achieving the positioning of the aircraft.

[0129] Step 6.1, Set the camera pose vector The initial value of .

[0130] (X) S Y S Let Z be the coarse matching result from step 4. S For the aircraft altimeter (X) S Y S The corresponding height, ω, κ are the inertial navigation system (X S Y S The corresponding value.

[0131] Step 6.2: Select ground control points in the vector data and project them onto the real-time map viewpoint to obtain the projected points.

[0132] In step 2, for (X) S Y S In the extracted ground feature vector data, sampling lines with equal intervals in the X and Y directions are set, and the set of intersection points between these lines and the vector data contour lines is taken as the ground control point set x. b Based on the value of p, the vector data is projected onto the real-time view using the projection method in step 2. Here, the real-time view is the position (X) of the aircraft's downward-looking camera obtained through coarse matching in step 4. S Y S A real-time image taken at location ) and x b The projected set of points is denoted as the projected set of ground control points x′. bIf a ground control point is covered by other elements after projection, then that point is not counted.

[0133] Step 6.3: Optimize the distance from the projection point to the edge of the same category in the real-time image using the least squares method, and iteratively calculate the camera pose parameters.

[0134] Based on the least squares principle, optimize x′ b The distance to the edge of the same category in the real-time image is used to iteratively calculate the camera pose parameter p. The specific steps are as follows:

[0135] Let n_b be the number of projected points of the ground control points. Then the nearest edge distance vector d(p) is a vector with n_b rows, where each element of d(p) represents a projected point. The Euclidean distance to the nearest edge of the same type in the real-time graph is calculated using the following formula:

[0136] d i =T Dc (x′ i y′ i (10)

[0137] In the formula, (x′ i y′ i ) represents the i-th ground control point The coordinates T projected onto the real-time graph Dc Is with Distance transformation graph corresponding to category number, T Dc (x′ i y′ i ) represents T Dc (x′) i y′ i The value at position ).

[0138] d i Let T be the value of the i-th element in d(p), if T Dc (x′ i y′ i ) is M or T Dc In (x′) i y′ i If a ground control point has a value of M in its 4-neighborhood, then that ground control point is discarded and will not be included in subsequent calculations.

[0139] The Jacobian matrix J of d(p) is calculated using the following formula:

[0140]

[0141] In the formula, J i Let J be the value of the i-th row of matrix J corresponding to the i-th ground control point. and It is x′i y′ i Substituting the derivatives of x and y with respect to each element in p into the collinearity equation, T Dc This is the distance transformation graph of the category corresponding to this point, T. Dc In (x′) i y′ i The partial derivative at () can be approximated by the following formula:

[0142]

[0143]

[0144] Based on the reliability and stability of various land features, weights are assigned to the points corresponding to each type of land feature, forming a diagonal matrix P. Taking buildings, roads, water bodies, and vegetation as examples, the weights of these four types of land features should decrease in that order, because the edges of buildings and roads are relatively stable, while the edges of water bodies and vegetation are easily changed.

[0145] The correction value Δp for the computational parameter p is calculated using the least squares method, and the formula is as follows:

[0146] Δp=-(J T PJ) -1 J T Pd(p) (14)

[0147] The value of parameter p is adjusted to p′, and the formula is:

[0148] p′=p+Δp (15)

[0149] Determine whether the correction values ​​of each parameter in Δp exceed the limit. If they do, return to step 6.1, use p′ as the new initial value of p, repeat the subsequent process, and increment the iteration count by 1. If they do not exceed the limit or the iteration count reaches the set upper limit, output p′ to obtain the camera pose at the moment of real-time image capture.

[0150] By adding the relative pose parameters of the aircraft and the camera to p′, the position and attitude of the aircraft at the moment of real-time image capture can be obtained, thus realizing the navigation function.

[0151] Example 2

[0152] Based on the same inventive concept, the present invention also provides a vector-based aircraft scene matching navigation system, including a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a vector-based aircraft scene matching navigation method as described above.

[0153] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0154] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A vector-based aircraft scene matching navigation method, characterized in that, Includes the following steps: Step 1: Train the semantic segmentation neural network, obtain vector data of the target region, and load the trained semantic segmentation neural network and vector data onto the aircraft; Step 2: After the aircraft enters the scene matching area, multiple predicted camera positions are selected at intervals within the estimated scene matching area. The vector data pre-loaded on the aircraft is projected onto the predicted real-time map view to form a reference map with multiple views. Step 3: Use the trained semantic segmentation neural network to segment the real-time image captured by the aircraft and extract the edges of various ground features; Step 4: Perform a coarse match between the baseline image of each viewpoint and the real-time image from which the ground feature edges were extracted in Step 3. Calculate the baseline image with the highest matching degree according to the edge category and distance to obtain the approximate position of the aircraft at the moment the real-time image was captured. Step 4.1: Set the sampling line spacing and tolerance distance; Step 4.2: Generate arrays of horizontal and vertical edge positions for the real-time image and each reference image at the sampling line location; The specific methods are as follows: ① Generate the horizontal edge position array of the real-time image. The method is as follows: Take one row from top to bottom in the real-time image as a horizontal sampling line. Traverse the pixels of the row from left to right. If the pixel value of a certain position is not 0 and the pixel value to its right is 0, then add the column number of the position to the edge position array of the row. If several consecutive positions have pixel values ​​that are not 0, then take the median of their column numbers and add it to the edge position array of the row. ② Generate the vertical edge position array of the real-time image. The method is as follows: Take one column from left to right in the real-time image as a vertical sampling line. Traverse the pixels of the column from top to bottom. If the pixel value of a certain position is not 0 and the pixel value below it is 0, then add the row number of the position to the edge position array of the column. If several consecutive pixel values ​​are not 0, the median of their row numbers is added to the edge position array of that column; ③ Generate the horizontal edge position array for each reference image, in the same way as generating the horizontal edge position array for the real-time image; ④ Generate the vertical edge position array for each reference image, in the same way as generating the vertical edge position array for the real-time image. Step 4.3: Match the real-time image with each baseline image and calculate the matching score for each baseline image; Step 4.4: Sort the scores of each baseline image and obtain the baseline image with the highest score (as the approximate location of the aircraft at the moment the real-time image was captured). Step 5: Perform Euclidean distance transformation on the real-time images captured by the aircraft; Step 6: Based on the distance between the edge of the reference image projected onto the real-time image view and the edge of the real-time image after Euclidean distance transformation, the camera pose parameters are iteratively calculated according to the least squares method to perform fine registration of the edge, thereby achieving the positioning of the aircraft.

2. The vector-based aircraft scene matching navigation method as described in claim 1, characterized in that: In step 1, based on the type of aircraft and mission requirements, a suitable semantic segmentation neural network model and dataset are selected, and the semantic segmentation neural network model is trained using the dataset. Images of the target area are acquired using remote sensing satellites or reconnaissance drones, and then digital photogrammetry, manual mapping, and supervised classification mapping techniques are used to create vector data containing horizontal coordinate information, elevation information, and ground feature category information, and category numbers are assigned to each type of ground feature. The trained semantic segmentation neural network model and vector data are then loaded onto the aircraft.

3. The vector-based aircraft scene matching navigation method as described in claim 1, characterized in that: In step 2, after the aircraft enters the scene matching zone, within the horizontal coordinate range of the matching zone predicted by the inertial navigation system, starting from one corner of the predicted matching zone, direction every Choose one value, direction every Choose one Values, forming multiple pairs The value is used as the predicted camera position, where For the set spacing value, The x and y coordinates of the predicted camera center in the geographic coordinate system.

4. The vector-based aircraft scene matching navigation method as described in claim 3, characterized in that: In step 2, the vector data pre-loaded on the aircraft is projected onto the predicted real-time map view to form a reference map. The specific process is as follows: ①For each pair Based on the camera attitude given by the inertial navigation system and the aircraft altitude given by the altimeter, ground feature vector data within a certain range is extracted; The method for calculating the cropping range is as follows: The image plane coordinates of points on the border of the real-time image captured by the aircraft are... Substitute into formulas (1)-(2) to calculate the coordinates of the corresponding ground points in the geographic coordinate system. The coordinates of the cutoff area's border are obtained using the following formula: （1） （2） In the formula, 、 These are the image plane coordinates of the principal point. It's the camera's focal length. 、 、 These are the coordinates of the camera center in the geographic coordinate system. Z It is estimated from elevation data. Ground elevation at the location, For the aircraft altimeter ( The corresponding height, 、 、 、 、 、 、 、 、 It is determined by the camera attitude angle The formula for calculating the nine direction cosines is as follows: （3） because The value is uncertain, so it is necessary to first assume an approximate elevation value and substitute it into the collinearity equation formulas (1)-(2) to calculate the ground coordinates. And interpolated from the elevation of nearby points. The elevation at that location was determined through multiple iterations, up to two consecutive iterations. The error is kept below the set limit until the result is obtained; the boundary of the obtained ground point range is extended outward by a certain distance before vector data is extracted. ② Project the captured vector data onto the predicted real-time graph view to form a baseline graph. The predicted real-time image is the position of the aircraft's downward-looking camera. The real-time image captured at the location is projected according to the following rules: the projection order is from far to near, that is, the elements are projected from the center of the camera. The farther away, the earlier the projection order; the edges of features are projected first, then their internal pixels are filled; features projected later will cover features projected earlier; the pixel value of a pixel on the reference map is set to the category number of the feature it is covered by; the following collinearity equation is used for projection: （4） （5） In the formula, 、 These are the image plane coordinates of the projected vector features. 、 、 These are the coordinates of the vector feature in the geographic coordinate system; ③ Extract the reference map after projection The edges of various land features in the middle are used to obtain the baseline map. The specific method is as follows: First, generate a reference image after projection. Edge-based baseline maps of the same size and will All pixel values ​​are initialized to 0, and then the reference image is traversed in order from left to right and from top to bottom. and The pixels are processed as follows: If If the position is not specified, skip that position and do not process it; if... Then let ， ;like Then let ， In other cases, this position is also skipped and not processed. Thus, for each pair Baseline maps have been established. This yields baseline images from multiple perspectives.

5. The vector-based aircraft scene matching navigation method as described in claim 4, characterized in that: In step 3, the real-time image captured by the aircraft's downward-looking camera is input into the trained semantic segmentation neural network model for segmentation processing to obtain the real-time image. Extract real-time image Real-time map obtained by analyzing the edges of various land features. The edge extraction method is the same as the baseline image in step 2. Methods for extracting the edges of ground features.

6. The vector-based aircraft scene matching navigation method as described in claim 1, characterized in that: In step 4.3, for each sampling line If its corresponding real-time image edge position array If it is not empty, then consider it valid and record it. The number of elements is Record the array of corresponding baseline image edge positions. The number of elements is traversal elements in For each ,exist Search among the elements that satisfy the following conditions : （6） In the formula, Sampling line and numbers Corresponding benchmark diagram The pixel value of the location, Sampling line and numbers Corresponding real-time graph The pixel value at the location, when the sampling line is horizontal. k Corresponding column number; when the sampling line is vertical, k Corresponding line number; All conditions are met Take the order smallest ,like , then Assigning scores View element Match successful; otherwise, give Assigning scores View element Match failed; iteration complete After determining the elements, calculate the score for that sampling line. The calculation formula is: （7） In the formula, For each The corresponding score, For sampling lines The elements that match successfully Number, Tolerance distance, Array of edge positions in real-time graph The number of elements, Array of baseline map edge positions The number of elements; All valid sampling lines The mean is the value of the baseline chart. The calculation formula is: （8） In the formula, This represents the number of valid sampling lines.

7. The vector-based aircraft scene matching navigation method as described in claim 1, characterized in that: Step 5 sets the distance transformation range. Generate a side length of Square distance transformation template ,in, The value at each position is its Euclidean distance from the center of the template. Time European distance The calculation formula is: （9） For real-time graphs Each land cover category existing in Generate a pair conduct Distance transformation map of boundary padding for pixels and will All pixel values ​​are set to diagonal length ; Real-time graph Distance Transformation Diagram Distance Transformation Template Stacking, and Center-align, then iterate through Medium pixels, each value The pixels are denoted as For each Perform the following transformations: [Transform the template] The center and Alignment, Comparison Each value and its corresponding position The magnitude of the value, if If the value is small, then... Replace the value here with The value of .

8. The vector-based aircraft scene matching navigation method as described in claim 1, characterized in that: Step 6 includes the following steps: Step 6.1: Select ground control points in the vector data and project them onto the real-time map view to obtain the projected points; Set camera pose vector The initial value, where ( Let's take the coarse matching result from step 4 as an example. For the aircraft altimeter ( The corresponding height, It is an inertial navigation system ( The corresponding value; in step 2 for Set in the extracted ground feature vector data , The ground control point set is the set of points where equally spaced sampling lines intersect with the vector data contour lines. ;according to The values ​​are projected onto the real-time image view using the projection method in step 2. Here, the real-time image is the position obtained by the aircraft's downward-looking camera in step 4 through coarse matching. Real-time images taken at the location, and... The projected set of points is denoted as the projected set of ground control points. If a ground control point is covered by other features after projection, then that point is not counted. Step 6.2: Optimize the distance from the projection point to the edge of the same category in the real-time image using the least squares method, and iteratively calculate the camera pose parameters; Let the number of projected points of the ground control points be . Then the nearest edge distance vector for The vector of rows, Each element represents a projection point. The Euclidean distance to the nearest edge of the same type in the real-time graph is calculated using the following formula: （10） In the formula, Representing the ground control points Coordinates projected onto the real-time graph. Is with Distance transformation graph corresponding to category number express middle The value at the position; yes The Middle The value of each element, if for or exist There is a value in the 4-neighborhood. If the pixel is not specified, the ground control point is discarded and will not be included in subsequent calculations. calculate Jacobi matrix The formula is: （11） In the formula, For the first Matrix corresponding to each ground control point No. The value of the row, and It is 、 After substituting into the collinear equation 、 right The derivatives of each element in the equation. This is the distance transformation diagram of the category corresponding to this point. exist The partial derivative at point can be approximated by the following formula: （12） （13） Based on the reliability and stability of various land features, weights are assigned to the points corresponding to each type of land feature, forming a diagonal matrix. Parameters are calculated using the least squares method. correction value The formula is: （14） Correction parameters The value is The formula is: （15） Then determine Check whether the correction values ​​of each parameter exceed the set limits. If they do, return to step 6.1 and adjust accordingly. As a new Initialize the value, repeat the subsequent process, and increment the iteration count by 1; if the limit is not exceeded or the iteration count reaches the set upper limit, output the result. This allows us to obtain the camera pose at the moment the real-time image was captured; and to give... By adding the relative pose parameters between the aircraft and the camera, the position and attitude of the aircraft at the moment of real-time image capture can be obtained, thus enabling navigation.

9. A vector-based aircraft scene matching navigation system, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions in the memory to execute a vector-based aircraft scene matching navigation method as described in any one of claims 1-8.

Images