A scene matching method based on deep learning

Abstract

The application discloses a kind of scene matching methods based on deep learning, first construct scene matching model, select 1 photograph and 4 different angle template images;Then the selected image is simultaneously input to full convolution twin neural network model;Next using the two branches of full convolution twin neural network model respectively extracts feature, generates five feature maps;Afterwards, the feature map is convolved, the similarity is measured, and a score matrix is generated;Then retrieve the maximum value coordinates in score matrix, obtain the position of matching target;Again, the matching result is compared, and the template image with the best matching effect is output as the matching result;Finally, the scene matching model is deployed to the unmanned aerial vehicle platform, and the visual pose estimation of the scene matching result is performed to realize the cross-view visual positioning of the unmanned aerial vehicle. The application improves the accuracy of unmanned aerial vehicle positioning, and lays a foundation for subsequent construction of a scene matching model with higher model accuracy and faster matching speed.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) technology, specifically relating to a scene matching method based on deep learning. Background Technology

[0002] Currently, the positioning and navigation of air-to-ground missiles, loitering munitions, UAVs, and long-range bombers mainly rely on GNSS, inertial navigation, and radio navigation. GNSS and radio navigation have weak survivability in denied environments, and high-precision inertial navigation is expensive. Achieving high-precision navigation in denied environments at low cost is increasingly urgent. Vision-based navigation is an important method, as it has low optical costs and allows for the reuse of seeker images, meeting the requirements of low cost and ease of use. Image matching refers to image analysis and image processing techniques that use matching algorithms to identify a region in one image from a target region, finding corresponding points between them. Scene matching is a special type of image matching technology that mainly uses image sensors to acquire images of the area near the flight or target area and matches them with stored reference images to obtain the aircraft's position data. Scene matching algorithms are widely used in the guidance of various aircraft, possessing characteristics such as autonomy, stealth, and resistance to external interference. They have significant advantages in improving the survivability of navigation systems in denied environments and meeting navigation requirements in special mission phases.

[0003] Extensive research has been conducted both domestically and internationally on scene matching technology, yielding certain results. However, numerous difficulties and problems remain in practical applications. Scene matching technology based on UAV platforms often suffers from wide baselines due to variations in imaging conditions, such as different times, viewpoints, lighting, resolutions, platforms, and sensors. This poses a significant challenge to accurate and robust image registration. Therefore, there is a need to develop a scene matching method with higher matching accuracy, real-time performance, and faster computation speed. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a scene matching method based on deep learning. First, a scene matching model is constructed, selecting one captured image and four template images from different angles. Then, the selected images are simultaneously input into a fully convolutional Siamese neural network model. Next, features are extracted using two branches of the fully convolutional Siamese neural network model, generating five feature maps. Then, convolution operations are performed on the feature maps to measure similarity and generate a score matrix. Next, the coordinates of the maximum value in the score matrix are retrieved to obtain the location of the matching target. The matching results are then compared, and the template image with the best matching effect is output as the matching result. Finally, the scene matching model is deployed to a UAV platform, and visual pose estimation is performed on the scene matching results to achieve cross-view visual localization of the UAV. This invention improves the accuracy of UAV localization and lays the foundation for subsequently building scene matching models with higher accuracy and faster matching speed.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows:

[0006] Step 1: Construct an image matching model, select a fully convolutional Siamese neural network, and choose 1 captured image and 4 template images from different angles;

[0007] Step 2: In one matching operation, four template images from different angles and one selected captured image are input into a fully convolutional Siamese neural network model.

[0008] Step 3: Use two branches of the fully convolutional Siamese neural network model to extract features from four template images at different angles and one captured image, generating five feature maps;

[0009] Step 4: Use the feature maps of the four template images to perform convolution operations on the feature maps of the captured image, measure the similarity, and generate a score matrix;

[0010] Step 5: Retrieve the coordinates of the maximum value in the score matrix, determine the region corresponding to the maximum value coordinates as the matching region, and revert to the captured image to obtain the location of the matching target;

[0011] Step 6: Use the matching detection module to compare the matching results of the 4 template images with the selected 1 captured image, and output the template image with the best matching effect as the matching result;

[0012] Step 7: Deploy the scene matching model to the UAV platform, and obtain the UAV pose by visual pose estimation of the scene matching results, thereby realizing cross-view visual positioning of the UAV.

[0013] Preferably, the scene matching model consists of two parts: a fully convolutional Siamese neural network and a matching detection module. The fully convolutional Siamese neural network extracts features from four template images at different angles and one captured image, and performs convolution calculations on each to obtain the matching results between the four template images and the selected captured image. The matching detection module compares the matching results between the four template images and the selected captured image, and outputs the template image with the highest matching degree as the matching result.

[0014] Preferably, step 3 specifically comprises:

[0015] A fully convolutional Siamese neural network includes two channels, which extract features from the template image and the captured image, respectively. Since a fully convolutional Siamese neural network does not have fully connected layers, it can accept template images and captured images of any size as input. Each channel of a fully convolutional Siamese neural network contains convolutional layers and pooling layers.

[0016] The specific steps of the fully convolutional Siamese neural network for feature extraction from template images and captured images are as follows: four template images from different angles, merged into the same batch, and one captured image are input into the model. Convolutional layers are used to extract image features from the input images. Then, a pooling layer is used to downsample the pixel values ​​of each adjacent small region on the input feature map into a single value. Finally, the extracted high-level image features are mapped to a new feature space to generate two feature maps.

[0017] Preferably, step 6 specifically comprises:

[0018] The matching detection module calculates the Euclidean distance and cosine similarity between the score matrices of the four generated template images and the selected captured image, and outputs the template image with the highest similarity as the matching result.

[0019] Preferably, step 7 specifically comprises:

[0020] The scene matching model is deployed on the UAV platform. During flight, the UAV acquires ground images in real time. The scene matching model matches the real-time image information with the template image to obtain the image matching result. Then, the image matching result is processed by feature point extraction and sorting to obtain the image registration result and key feature matching point pair set. Then, the UAV pose is calculated based on the feature point matching points to obtain the UAV's position and attitude information. The scene matching model is deployed on the UAV platform to perform visual pose estimation on the scene matching result to obtain the UAV pose, thus realizing cross-view visual positioning of the UAV.

[0021] The beneficial effects of this invention are as follows:

[0022] 1. This invention innovatively uses a fully convolutional Siamese network to perform cross-view image matching tasks. This network has no fully connected layers and can accept template images and captured images of arbitrary sizes as input. It is no longer strictly limited by the size of the input image. After feature extraction, the template image can be used as a convolution kernel to perform convolution calculation on the feature map of the retrieved image to calculate the similarity of all sub-windows.

[0023] 2. This invention innovatively designs a cross-view image matching scheme. Addressing the issue of differing viewpoints between captured images and template images in practical use, it proposes a cross-view image matching scheme. Utilizing a fully convolutional Siamese neural network and a matching detection module, it matches four template images from different viewpoints input to the fully convolutional Siamese neural network with the captured image. The matching detection module then selects the template image with the best matching effect as the output, improving the accuracy and robustness of the matching. Furthermore, this invention merges the four template images from different angles into a single batch for simultaneous input, improving the system's real-time performance and processing speed.

[0024] 3. This invention deploys the scene matching model to the UAV platform, which improves the accuracy of UAV positioning by enhancing the precision of image matching, laying the foundation for building a scene matching model with higher accuracy and faster matching speed in the future. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the drone scene matching navigation process of the present invention.

[0026] Figure 2 This is a schematic diagram of the scene matching method of the present invention.

[0027] Figure 3 This is a schematic diagram of the vision-based UAV pose calculation and localization process of the present invention.

[0028] Figure 4 This is a matching result diagram of an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] like Figure 1 As shown, since some theories and methods of UAV scene matching visual navigation technology research are still imperfect in the existing technology, and the scene matching technology based on UAV platform has low matching accuracy due to changes in imaging conditions, such as different time, different viewing angle, different lighting, different resolution, different platform and different sensors, this invention proposes a scene matching method with higher matching accuracy, system real-time performance and faster calculation speed, and applies it to UAV platform for navigation and positioning.

[0031] like Figure 2 As shown, the implementation steps of this invention are as follows:

[0032] Step 1: Construct an image matching model, select a fully convolutional Siamese neural network, and choose 1 captured image and 4 template images from different angles;

[0033] Step 2: In one matching operation, four template images from different angles and one selected captured image are input into a fully convolutional Siamese neural network model.

[0034] Step 3: Use two branches of the fully convolutional Siamese neural network model to extract features from four template images at different angles and one captured image, generating five feature maps;

[0035] Step 4: Use the feature maps of the four template images to perform convolution operations on the feature maps of the captured image, measure the similarity, and generate a score matrix;

[0036] Step 5: Retrieve the coordinates of the maximum value in the score matrix, determine the region corresponding to the maximum value coordinates as the matching region, and revert to the captured image to obtain the location of the matching target;

[0037] Step 6: Use the matching detection module to compare the matching results of the 4 template images with the selected 1 captured image, and output the template image with the best matching effect as the matching result;

[0038] Step 7: Deploy the scene matching model to the UAV platform, and obtain the UAV pose by visual pose estimation of the scene matching results, thereby realizing cross-view visual positioning of the UAV.

[0039] Example:

[0040] The scene matching model constructed in step 1 consists of two parts: a fully convolutional Siamese neural network and a matching detection module. The selected images include one captured image and four template images from different angles.

[0041] In step 2, four template images from different angles and one selected captured image are simultaneously input into the fully convolutional Siamese neural network model.

[0042] The fully convolutional Siamese neural network used in step 3 contains two channels, one for extracting features from the template image and the other for extracting features from the captured image. Since the selected fully convolutional Siamese neural network has no fully connected layers, it can accept template and captured images of any size, eliminating strict limitations on input image dimensions. Each channel of the fully convolutional Siamese neural network contains convolutional and pooling layers. The specific steps for feature extraction from the template and captured images using the fully convolutional Siamese neural network are as follows: four template images from different angles and one selected captured image are input into the model. Convolutional layers are used to extract image features from the original images. Then, a pooling layer is used to downsample the pixel values ​​of adjacent small regions on the input feature map, merging them into a single value. Finally, the extracted high-level image features are mapped to a new feature space, generating five feature maps.

[0043] In step 4, the feature map of the template image is used to perform convolution operation on the feature map of the captured image to measure similarity and generate a score matrix.

[0044] In step 5, the coordinates of the maximum value in the score matrix are retrieved, and it is considered to be the matching region. This value is then regressed to the captured image to obtain the location of the matching target.

[0045] In step 6, the matching detection module is used to calculate the Euclidean distance and cosine similarity between the matching score matrix of the four generated template images and the selected one captured image, and the template image with the highest similarity and the best matching effect is output as the matching result.

[0046] like Figure 3 and Figure 4 As shown, in step 7, the scene matching model is deployed to the UAV platform. The UAV acquires ground images in real time during flight. The scene matching model matches the real-time image information with the template image to obtain the image matching result. Then, feature points are extracted and sorted from the image matching result to obtain the image registration result and a set of key feature matching points. Next, the UAV pose is calculated based on the feature point matching points to obtain the UAV's position and attitude information. By deploying the scene matching model to the UAV platform and performing visual pose estimation on the scene matching result, the UAV pose is obtained, achieving cross-view visual positioning of the UAV. The essence of visual positioning is to add a visual imaging module to the UAV positioning system, using the combined action of the camera imaging system and the coordinate system to calculate the UAV pose. This solution deploys the proposed scene matching model on the UAV, improving the accuracy of UAV positioning by enhancing the image matching accuracy, laying the foundation for building a scene matching model with higher accuracy and faster matching speed in the future.

Claims

1. A scene matching method based on deep learning, characterized in that, Includes the following steps: Step 1: Construct a scene matching model. Select a fully convolutional Siamese neural network (WSN), one captured image, and four template images from different angles. The scene matching model consists of two parts: a WSN and a matching detection module. The WSN extracts features from the four template images from different angles and the captured image, and performs convolution calculations on each to obtain the matching results between the four template images and the selected captured image. The matching detection module compares the matching results of the four template images with the selected captured image and outputs the template image with the highest matching degree. Step 2: In one matching operation, four template images from different angles and one selected captured image are input into a fully convolutional Siamese neural network model. Step 3: Using two branches of a fully convolutional Siamese neural network model, features are extracted from four template images at different angles and one captured image, generating five feature maps; Step 3 specifically involves: A fully convolutional Siamese neural network includes two channels, which extract features from the template image and the captured image, respectively. Since a fully convolutional Siamese neural network does not have fully connected layers, it can accept template images and captured images of any size as input. Each channel of a fully convolutional Siamese neural network contains convolutional layers and pooling layers. The specific steps of the fully convolutional Siamese neural network for feature extraction from template images and captured images are as follows: four template images from different angles, merged into the same batch, and one captured image are input into the model. The input images are convolved through convolutional layers to extract image features. Then, a pooling layer is used to downsample the pixel values ​​of each adjacent small region on the input feature map into one value. Finally, the extracted high-level image features are mapped to a new feature space to generate two feature maps. Step 4: Use the feature maps of the four template images to perform convolution operations on the feature maps of the captured image, measure the similarity, and generate a score matrix; Step 5: Retrieve the coordinates of the maximum value in the score matrix, determine the region corresponding to the maximum value coordinates as the matching region, and revert to the captured image to obtain the location of the matching target; Step 6: Use the matching detection module to compare the matching results of the 4 template images with the selected 1 captured image, and output the template image with the best matching effect as the matching result; Step 7: Deploy the scene matching model to the UAV platform, and obtain the UAV pose by visual pose estimation of the scene matching results, thereby realizing cross-view visual positioning of the UAV.

2. The scene matching method based on deep learning according to claim 1, characterized in that, Step 6 specifically involves: The matching detection module calculates the Euclidean distance and cosine similarity between the score matrices of the four generated template images and the selected captured image, and outputs the template image with the highest similarity as the matching result.

3. The scene matching method based on deep learning according to claim 2, characterized in that, Step 7 specifically involves: The scene matching model is deployed on the UAV platform. During flight, the UAV acquires ground images in real time. The scene matching model matches the real-time image information with the template image to obtain the image matching result. Then, the image matching result is processed by feature point extraction and sorting to obtain the image registration result and key feature matching point pair set. Then, the UAV pose is calculated based on the feature point matching points to obtain the UAV's position and attitude information. The scene matching model is deployed on the UAV platform to perform visual pose estimation on the scene matching result to obtain the UAV pose, thus realizing cross-view visual positioning of the UAV.

Images