A large field angle difference image matching positioning method for visual navigation

Abstract

The present application belongs to the field of airborne visual aided navigation, and particularly relates to a large field of view angle difference image retrieval matching positioning method for visual navigation, aiming at the challenges of limited visual angle, large visual line change, diverse landform types and the like of low-altitude aircraft, through a new view angle generation mechanism, a new synthetic image with high pitch angle is generated according to the feature information of the front view image, and the problem that the large cross-view image feature difference under large view angle difference leads to no correspondence is solved; in addition, the method fully utilizes the advantages of Vision Transformer in global information modeling and explicit position information coding, and through the method of non-uniform clipping, the calculation cost can be reduced without almost performance decline, and the matching accuracy and robustness of cross field of view angle images are improved. Thus, more effective technical support is provided for airborne real-time visual navigation, integrated navigation, target positioning and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of airborne visual aided navigation, and particularly relates to a large field angle difference image matching positioning method for visual navigation. BACKGROUND

[0002] Low-altitude aircraft such as unmanned aerial vehicles have shown unique and important roles in current main application fields. How to achieve low-cost, high-precision and robust positioning of low-altitude aircraft in a strong external interference environment is the key to improving the effectiveness of low-altitude aircraft. The visual navigation method mainly uses an image matching algorithm to quickly match visible light images / SAR images with a map reference library. According to the position information provided in the map reference library, autonomous precise positioning of the aircraft can be achieved. Compared with traditional inertial navigation autonomous positioning, the visual navigation error does not accumulate with the increase of vision and distance, and has the advantages of low cost, low power consumption, high flexibility, etc. At present, it has become one of the popular methods of autonomous navigation and integrated navigation.

[0003] Traditional visual navigation mainly uses airborne downward-looking images with a similar view angle to satellite maps. A downward-looking camera needs to be separately installed under the aircraft pod or fuselage, which causes problems such as increased cost, redundant equipment integration, and even cannot meet the requirement of installing a downward-looking camera for some aircraft models or tasks. Considering that the forward-looking camera in the optoelectronic pod is one of the essential sensors on board, directly matching the forward-looking image with the map library can not only meet the demand of visual navigation, but also save space and resources, which is an extremely valuable matching scheme in visual navigation. However, compared with the downward-looking camera, the pitch angle of the airborne forward-looking image is smaller, and there is a great difference in the contour features and scene distribution between the target photographed by the airborne forward-looking image and the satellite overhead image. Therefore, the traditional feature matching method is no longer applicable. These problems greatly restrict the practical application of visual navigation. Therefore, it is of great practical significance and engineering value to develop cross-matching positioning technology under large angle difference. SUMMARY

[0004] The purpose of the application is to address the challenges of limited view angle, large view change, and diverse terrain types of low-altitude aircraft, based on the latest progress in deep learning models, a large field angle difference image retrieval matching positioning method for visual navigation is proposed, which realizes aircraft target positioning based on image retrieval. A new view generation mechanism is designed to generate synthetic images with high pitch angles based on the feature information of forward-looking images, meeting the requirements of large angle matching algorithms. A high-precision cross-matching algorithm is proposed to improve the matching accuracy and robustness of images with different field angles of the same target.

[0005] The technical scheme of the application: according to the first aspect of the application, a large field angle difference image matching positioning method for navigation is proposed, which comprises:

[0006] Step 1: Acquire real-time images from the forward-looking binocular camera of the drone;

[0007] Step 2: Construct a 4-dimensional pose vector using a spherical coordinate system, and use the 4-dimensional pose vector to represent the image;

[0008] Step 3: Construct a rendering dataset for fine-tuning the diffusion model;

[0009] Step 4: Construct a stable diffusion model f using the rendering dataset described in Step 3, and generate a new perspective image of the real-time image from Step 1 using the diffusion model f;

[0010] Step 5: Train the cross-view matching model using the satellite reference image library corresponding to the real-time image in Step 1 and the new view image generated in Step 4.

[0011] Step 6: Input the new perspective image of the real-time image generated in Step 4 from Step 1 into the cross-view matching model trained in Step 5, output the corresponding reference image, and use the reference image to perform satellite reference image library image retrieval to obtain the target image positioning result.

[0012] In one possible embodiment, step 2 specifically includes the following process: when creating the image pair dataset, firstly, all content within the image is normalized to a unit cube, and then the camera viewpoints are sampled uniformly; during training, the two camera viewpoints are set as follows: and Express their relative transformations as Since the axes in the camera coordinate system always point to the center of the coordinate system, the extrinsic parameter matrix can be uniquely determined by the position of the camera in the spherical coordinate system; considering the yaw angle... Discontinuities in 2π space, using Come to Encoded representation; therefore, the input image is accompanied by Both are input into the model.

[0013] In one possible embodiment, in step 3, the Objaverse dataset is used to randomly sample N viewpoints for each object in the dataset to obtain N sets of camera extrinsic parameters; during training, two different viewpoints are randomly sampled from the N viewpoints for each object to form an image pair; this constitutes a fine-tuning dataset for fine-tuning the subsequent diffusion model.

[0014] Preferably, N is 12.

[0015] In one possible embodiment, in step 4, the diffusion model structure includes an encoder ε, a denoising U-Net network U, and a decoder D. Since 3D reconstruction of a monocular 2D image requires both low-level perceptual information (depth, shadows, texture, etc.) and high-level understanding information (category, function, structure, etc.), a dual-branch hybrid mechanism is designed. One branch inputs the CLIP embedding of the image, which is connected to the camera extrinsic parameters (R,T) to form a CLIP embedding representation containing angular information, denoted as c(x,R,T).

[0016] One branch of the denoising U-Net structure U takes the CLIP embedding of the input image as its input and connects it to the camera extrinsic parameters (R,T) to form a CLIP embedding representation containing angular information, denoted as c(x,R,T). The other branch outputs the semantically information-rich result from the U-Net, enabling the model to preserve scene details in the synthesized new perspective image. Overall, the training process of the diffusion model can be represented as solving the following objective function:

[0017]

[0018] After the network model is trained, the inference model takes a Gaussian noise image with c(x,R,T) as input and generates a new perspective image of the required viewpoint.

[0019] In one possible embodiment, step 5 specifically includes the following steps:

[0020] 5.1 Given the new perspective image {I} generated in step 4 s} and satellite reference image library {I a Each new perspective image and its corresponding aerial image are considered a positive pair, while all other image pairs are considered negative pairs.

[0021] 5.2 Train two independent Transformer encoders, with the new perspective image generated in step 4 and the satellite reference image library as inputs, respectively. First, the input image is transformed into multiple tokens through the Patchembedding module, which serve as the input to the Transformer encoder. The input image is divided into N1 and N2 P*P patches, which are flattened and then fed into the linear projection layer to generate N1 and N2 tokens.

[0022] 5.3 A learnable class token is added to each of the N1 and N2 image tokens to integrate the classification information from each layer; the class token of the last layer is input into a multi-head attention module to generate the final classification vector. This output vector is used as the embedding feature, and a soft-boundary triplet loss function is used for training.

[0023]

[0024] 5.4 The above loss function is optimized by using an adaptive sharpness-aware minimization technique to minimize the adaptive sharpness of the neural network loss function distribution, making the loss function converge more smoothly and achieving better generalization ability.

[0025] In one possible embodiment, in step 5, a non-uniform focus pruning strategy is employed to significantly save computational resources. The specific steps are as follows:

[0026] 6.1 Attention feature map generated using the last Transformer encoder in the satellite benchmark image branch This represents the contribution of each token to the final output;

[0027] 6.2 will Transform to the original image size; important areas in the image, such as target buildings and streets, receive higher attention scores, while other occluded scenes receive lower scores; determine the proportion β of patches to retain after cropping; keep the patch size unchanged and increase the image resolution. This allows for a γ-fold increase in the number of patches, enabling the magnification of details.

[0028] 6.3 Adjust the feature map size based on γ, binarize the feature map based on the scaling factor β, retain the reference map positions corresponding to a binary image of 1, and crop the reference map positions corresponding to a binary image of 0; the cropped reference map is represented as I. RC It is divided into γβN patches;

[0029] 6.4 The cropped reference image is fed into the Transformer encoder, sharing weights with the reference image encoder from step 5.

[0030] In one possible embodiment, in step 5, the cross-view matching model adopts the ViT model.

[0031] The beneficial technical effects of the present invention are as follows: The present invention solves the problem of image matching and positioning under conditions of large field of view difference. Based on the accurate matching of the generated new perspective image and satellite image, the cross-matching of airborne forward view image and satellite image and the positioning function of the aircraft are realized.

[0032] In addition, this method optimizes the resource consumption of cross-matching based on the ViT model by removing regions without key information during the matching process, thereby reducing computation and memory consumption without significantly reducing algorithm performance. Attached Figure Description

[0033] Figure 1 This is a flowchart of a preferred embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of a preferred embodiment of the present invention using a spherical coordinate system to represent the camera position and relative transformation;

[0035] Figure 3 This is a schematic diagram of the specific structure of the dual-branch hybrid mechanism of a preferred embodiment of the present invention;

[0036] Figure 4 This is an overall flowchart of a preferred embodiment of the present invention for generating images from new perspectives using a diffusion model;

[0037] Figure 5 This is a flowchart of the cross-view matching method based on the ViT model according to a preferred embodiment of the present invention.

[0038] Figure 6 This is a schematic diagram of the cutting steps in a preferred embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The features of various aspects of the embodiments of the present invention will now be described in detail. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can also be practiced without these specific details. The following description of the embodiments is merely intended to provide a better understanding of the invention by illustrating examples. The invention is not limited to any specific setups and methods provided below, but covers all improvements, substitutions, etc., to product structures and methods without departing from the spirit of the invention. In the various drawings and the following description, well-known structures and techniques are not shown to avoid unnecessarily obscuring the invention.

[0041] It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other, and the various embodiments can be referenced and cited in each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] This application provides a flowchart of a cross-view image matching and localization method that combines a diffusion model with ViT. Figure 1 A flowchart illustrating a cross-view image matching and localization method combining a diffusion model with ViT according to an embodiment of this application is shown. See also... Figure 1 The methods include:

[0043] Step 1: Acquire real-time images from the forward-looking binocular camera on the drone.

[0044] The drone cruised within a designated area along a planned path at an altitude of 2000m and a pitch angle of 10°, capturing visible light images. (Image x∈R) H×W×3 H and W represent the height and width of the image, respectively. The image corresponds to a certain angle of the target scene, and the content includes houses, roads, plants, etc.

[0045] Step 2, construct a 4-dimensional pose vector

[0046] Use spherical coordinates to represent camera position and relative transformations, such as Figure 2 As shown. Assume the object's center is at the origin, and θ and These represent the polar angle and the azimuth angle, respectively. r is the radius, i.e., the distance from the origin. When creating the image pair dataset, the input is an image x. First, the pixel values ​​within the image are normalized. Then, the spatial coordinates are mapped to the unit plane. Finally, all content of image x is normalized to a size of [-0.5, 0.5]. 3 In the unit cube, it is represented as Then θ∈[0,π], The camera viewpoint is sampled uniformly within the range of r∈[1.5, 2.2].

[0047] Let the two camera viewpoints be respectively and Express their relative transformations as Since the axes in the camera coordinate system always point to the center of the coordinate system, the extrinsic parameter matrix can be uniquely determined by the position of the camera in the spherical coordinate system.

[0048] Considering the deflection angle Discontinuities in 2π space, using Come to Encoding representation. The final image is obtained. Accompanied by 4-dimensional attitude vector The common input is fed into the model. This representation is used during both training and testing.

[0049] Step 3: Construct the fine-tuning dataset

[0050] Since the original diffusion network cannot take multimodal text embeddings as input information, the original stable diffusion model needs to be fine-tuned to incorporate the relative pose transformation information of the images, so that it can learn the relative transformation relationship between images.

[0051] This step involves constructing a rendering dataset for fine-tuning the diffusion model. Specifically, the Objaverse dataset is used, a large-scale open-source dataset containing over 80,000 3D object models. This dataset can represent these 3D models as geometric structures with fine-grained details and material properties. For each object in the dataset, 12 viewpoints are randomly sampled with the object's center as the coordinate center. The axis of each viewpoint intersects the object's center, resulting in 12 sets of camera extrinsic parameters, forming the extrinsic parameter matrix M. ⊥ During training, for each object, two different viewpoints are randomly sampled from 12 viewpoints to form image pairs. R and T represent the relative transformation relationship between two images, which can be determined based on the extrinsic parameter matrix M. ⊥ Calculate and construct the fine-tuning dataset.

[0052] Step 4: Fine-tune the diffusion model

[0053] Using the dataset constructed in step 3, input an image pair and their relative camera extrinsic parameters {(x, x...} (R,T) Given the image (R, T), construct a stable diffusion model f and fine-tune f to enable the model to learn camera parameters without destroying the original image feature representation. The specific steps are as follows:

[0054] 4.1 A dual-branch hybrid mechanism is designed using an encoder ε, a denoising U-Net network U, and a decoder D to simultaneously acquire low-level perceptual information and high-level semantic information. One branch takes as input the Contrastive Language-Image Pre-training (CLIP) embedding of the image x, which, together with the camera extrinsic parameters (R,T), is mapped into the embedding space to form a CLIP embedding representation containing pose information, denoted as c(x,R,T). The output of this branch is the high-level semantic feature map of the image.

[0055] 4.2 On the other branch, a skip connection operation is used to directly concatenate the input image x with the feature map output from the previous branch, ensuring that the newly synthesized image output by the model retains the original object's category features and detailed information. The specific structure of the dual-branch mixing mechanism is as follows: Figure 3 As shown.

[0056] 4.3 Solve the following objective function within the diffusion time step t ~ [1, 1000] to fine-tune the model:

[0057]

[0058] Where z follows the feature distribution of the encoded image x, U(z) t ,t,c(x,R,T) represents the cascaded output of the two-branch mixing mechanism at time t; after parameter backpropagation, the inference model f can iteratively generate a new perspective image from a Gaussian noise map ∈ conditioned on c(x,R,T). The overall flowchart for generating new perspective images using a diffusion model is as follows: Figure 4 As shown.

[0059] Step 5: Training the cross-view matching model

[0060] The overall process of the cross-view matching method based on the ViT model is as follows: Figure 5 As shown.

[0061] 5.1 Using publicly available datasets from the internet, add the new high-tilt-angle images obtained in step 4 to obtain a new augmented dataset. Define a set of street view query images {I}. s} and aerial reference images {I a Each street view image and its corresponding aerial image are considered a positive pair, while all other aerial images are considered negative pairs. If multiple aerial images cover the same street view image, the nearest aerial image is used as the positive pair to avoid resampling other nearby aerial images.

[0062] 5.2 Train two independent Transformer encoders, with the input query image and reference image represented as follows: First, the input image is transformed into multiple tokens using the Patch Embedding module, which serve as the input to the Transformer encoder. The input image is divided into N1 and N2 P*P patches, which are then flattened and fed into a linear projection layer to generate N1 and N2 tokens.

[0063] 5.3 A learnable class token is added to each of the N1 and N2 image tokens to integrate the classification information from each layer. The class token of the last layer is input into a multi-head attention module to generate the final classification vector. This output vector is used as the embedding feature, and the soft-boundary triplet loss function is used for training.

[0064]

[0065] 5.4 The above loss function is optimized by employing Adaptive Sharpness-Aware Minimization (ASAM) to minimize the adaptive sharpness of the neural network loss function distribution (loss landscape), making the loss function converge more smoothly and achieving better generalization ability; for a given loss function L and parameter weights w∈R k The loss of sharpness is expressed as:

[0066]

[0067] Where ∈ is the perturbation factor on the weight w, and |·|² represents the l2 norm. The above equation actually represents the maximum value of the l2 norm of ∈ in a spherical domain with radius ρ. The sharpness of the loss depends on the magnitude of the weights; that is, any scaling factor a with respect to the weight w can change the sharpness of the loss. Therefore, an invertible linear operator can be used. As a regularization operation, it can offset the influence of the weighting coefficient. Adaptive sharpness is defined as:

[0068]

[0069] Due to the strong self-attention characteristic associated with softmax, the weight scale of the Transformer encoder varies greatly. Therefore, this adaptive sharpness, which is independent of the weight scale, is very beneficial for Transformer training. By ensuring the weights simultaneously satisfy adaptive sharpness during the training of the triplet loss function, overfitting can be overcome without using any data augmentation.

[0070] Step 6: Match image non-uniform focus cropping

[0071] Having a large number of tokens incurs high computational costs. Therefore, to save computation, it is necessary to consider reducing the number of tokens. Since there are only a few shared regions between the two viewpoint images, a large number of scenes contained in one viewpoint image may be almost invisible in the street view, contributing negligibly to the final similarity. Therefore, these regions can be removed to significantly reduce computational costs. This invention employs a non-uniform focus pruning strategy in the Transformer architecture. The pruning steps are as follows: Figure 6 As shown.

[0072] 6.1 Using the attention feature map generated by the last transformer encoder in the satellite map branch This represents the contribution of each token to the final output.

[0073] 6.2 will Transform to the original image size. Important areas in the image, such as target buildings and streets, receive higher attention scores, while other occluded scenes receive lower scores. Determine the proportion β of patches to retain after cropping. Keep the patch size unchanged and increase the image resolution. This allows for a γ-fold increase in the number of patches, enabling the magnification of details.

[0074] 6.3 The feature map size is adjusted based on γ, and the feature map is binarized based on the scaling factor β. The positions of the reference map corresponding to a binary image of value 1 are retained, while the positions of the reference map corresponding to a binary image of value 0 are cropped. The cropped reference map is represented as I. RC It is divided into γβN patches.

[0075] 6.4 The cropped reference image is fed into the Transformer encoder, sharing weights with the reference image encoder from step 5. Training is performed using the same steps as in steps 5.2 to 5.4 to obtain the trained retrieval model.

[0076] Step 7: Obtain the target map localization result

[0077] After the retrieval model is trained, select a query image I from the test set. qT I qT This image is a composite image of the high elevation angle of the original query image obtained in steps 1 to 4. It is then input into... Figure 5 In the model shown, the corresponding reference image I is output. RT Based on the geographic location information contained in the retrieved satellite images, the geographic location of the target in the query image is obtained, thus achieving target positioning.

Claims

1. A large field-of-view difference image matching and localization method for navigation, characterized in that, The method includes: Step 1, acquiring real-time images from a forward-looking binocular camera on a UAV; Step 2, constructing a 4-dimensional attitude vector using a spherical coordinate system, and representing the image using the 4-dimensional attitude vector; Step 3, constructing a rendering dataset for fine-tuning the diffusion model; Step 4, constructing a stable diffusion model f using the rendering dataset from Step 3, and generating a new perspective image of the real-time image from Step 1 using the diffusion model f; In Step 4, the structure of the diffusion model includes an encoder ε, a denoising U-Net network U, and a decoder D; one branch of the denoising U-Net structure U takes the CLIP embedding of the input image as input, which is connected to the camera extrinsic parameters (R,T) to form a CLIP embedding representation containing angular information, denoted as c(x,R,T); the other branch outputs the result with semantic information in the U-Net, enabling the model to maintain scene detail information in the synthesized new perspective image; The training process of the diffusion model is represented by solving the following objective function: After the network model is trained, the inference model generates a new perspective image based on a Gaussian noise image with c(x,R,T) as input; Step 5: Train a cross-view matching model using the satellite reference image library corresponding to the real-time image in Step 1 and the new perspective image generated in Step 4; Step 5 specifically includes the following steps: 5.1 Given the new perspective image {Is} generated in Step 4 and the satellite reference image library {Ia}, each new perspective image and its corresponding aerial image pair are considered positive pairs, and other image pairs are considered negative pairs; 5.2 Train two independent Transformer encoders, with the new perspective image generated in Step 4 and the satellite reference image library representing respectively... First, the input image is transformed into multiple tokens using the Patch embedding module, which serve as the input to the Transformer encoder. The input image is divided into N1 and N2 P*P patches, flattened, and then fed into a linear projection layer to generate N1 and N2 tokens. A learnable class token is added to each of the N1 and N2 image tokens to integrate the classification information from each layer. The class token from the final layer is input into a multi-head attention module to generate the final classification vector. This output vector is used as the embedding feature, and the soft-boundary triplet loss function is used for training. 5.4 The above loss function is optimized by using adaptive sharpness-aware minimization technology to minimize the adaptive sharpness of the neural network loss function distribution, so that the loss function converges more smoothly and achieves better generalization ability; Step 6: Input the new perspective image of the real-time image in Step 1 generated in Step 4 into the cross-view matching model trained in Step 5, output the corresponding reference image, and use the reference image to perform satellite reference image library image retrieval to obtain the target image positioning result.

2. The large field-of-view difference image matching and positioning method for navigation as described in claim 1, characterized in that, In step 2, the specific process includes: when creating the image pair dataset, firstly, all content within the image is standardized to a unit cube, and then the camera viewpoints are sampled uniformly; during training, the two camera viewpoints are set as follows: and Express their relative transformations as Because the axes in the camera coordinate system always point to the center of the coordinate system. Therefore, the extrinsic parameter matrix can be uniquely determined by the camera's position in the spherical coordinate system; considering the discontinuity of the yaw angle in 2π space, we use... Come to Encoded representation; therefore, the input image is accompanied by Both are input into the model.

3. The large field-of-view difference image matching and positioning method for navigation as described in claim 1, characterized in that, In step 3, the Objaverse dataset is used to randomly sample N viewpoints for each object in the dataset to obtain N sets of camera extrinsic parameters. During training, two different viewpoints are randomly sampled from the N viewpoints for each object to form an image pair. This constitutes a fine-tuning dataset for fine-tuning the subsequent diffusion model.

4. The large field-of-view difference image matching and positioning method for navigation as described in claim 3, characterized in that, N takes the value 12.

5. The large field-of-view difference image matching and positioning method for navigation as described in claim 1, characterized in that, In step 5, a non-uniform focus cropping strategy is adopted to significantly save computational resources. The specific steps are as follows: 6.1 Use the attention feature map generated by the last Transformer encoder in the satellite reference image branch. This represents the contribution of each token to the final output; 6.2 will Transform to the original image size; important areas in the image, such as target buildings and streets, receive higher attention scores, while other occluded scenes receive lower scores; determine the proportion β of patches to retain after cropping; keep the patch size unchanged and increase the image resolution. 6.3 Adjust the feature map size based on γ, binarize the feature map based on the ratio β, retain the corresponding reference map position with a binary map of 1, and crop the corresponding reference map position with a binary map of 0; the cropped reference map is represented as IRC and is divided into γβN patches; 6.4 Send the cropped reference map into the Transformer encoder and share weights with the reference map encoder in step 5.

6. The large field-of-view difference image matching and localization method for navigation as described in claim 1, characterized in that, In step 5, the cross-view matching model adopts the ViT model.

Images