A remote sensing target matching method based on a convolutional neural network of fused local features

Abstract

The application provides a remote sensing target matching method based on a convolutional neural network of fused local features, compared with a CNN feature based on an overall image, the CNN feature extracted at a region level has higher robustness and flexibility in identifying small similar regions between partial matching images, a matching region pair is accurately matched through a generating step, which makes the CNN feature extracted from the regions more robust for partial repeated image matching, because a region suggestion model in a rough division efficiently matches local features between images to select a small number of top candidate images, which can greatly reduce the target detection range of the partial repeated images, and the application can realize relatively accurate matching because the good robustness of SIFT features and the high resolution capability of CNN features are fully utilized.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image target matching technology, and in particular to a remote sensing target matching method based on convolutional neural networks that fuse local features. Background Technology

[0002] Traditional image feature extraction methods require a large number of feature experiments to find features in specific domains, which is not only very labor-intensive and complex, but also lacks robustness to complex and changing environments and conditions.

[0003] In recent years, with the development of deep learning, convolutional neural networks (CNNs) have achieved remarkable results in image pattern recognition due to their characteristics such as local feature extraction, parameter sharing, and global pooling. On the one hand, CNNs do not require manual feature engineering experiments; instead, they automatically extract effective features based on massive amounts of data and annotations. On the other hand, given sufficient training data, the model exhibits good generalization ability and strong robustness. Therefore, using CNNs to solve the problem of land cover identification and classification in remote sensing image processing has become a research hotspot in the field.

[0004] Using convolutional neural networks (CNNs) for target matching in high-resolution remote sensing images often requires a large number of samples for training. When the training dataset is limited, the features learned by the model have poor generalization ability. However, on the one hand, a large number of training samples may not be readily available for remote sensing image classification; on the other hand, collecting and labeling a large number of training samples is also very resource-intensive. Training is time-consuming and consumes a lot of disk space. Furthermore, the low-level convolutional features extracted by traditional CNN algorithms are not rotation-invariant, necessitating the rotation of a large amount of data to allow the neural network to learn different angles and enhance the model's generalization ability. This further increases the workload of data training.

[0005] Therefore, with the increasing prevalence of various deep neural networks in the field of computational intelligence, the research focus of content-based image matching and retrieval has shifted from handcrafted local features such as Scale Invariant Feature Transform (SIFT) to convolutionally derived features. However, existing image-based CNN features, extracted directly from the entire image, are not suitable for matching targets in small regions of high-resolution remote sensing images. Furthermore, region-based CNN features have limited robustness to various image modifications (such as rescaling, occlusion, and noise addition), which can affect the performance of matching partially repeated images.

[0006] In the prior art, the "Remote Sensing Image Registration Method and System Integrating SIFT Features and CNN Features" proposed in application publication number CN109035315A only makes full use of the good robustness of SIFT features and the high resolution of CNN features to achieve relatively accurate matching, but cannot solve the other technical problems mentioned above. Summary of the Invention

[0007] In view of this, the present invention proposes a remote sensing target matching method based on convolutional neural network fusion of local features. The method uses the maximum length trace generation tree algorithm to coarsely partition the image and efficiently matches the local features between images based on the region proposal model in the coarse partition to select a small number of top-ranked candidate images. This can greatly reduce the target detection range of some duplicate images and save computational resources. At the same time, by comparing multi-scale convolutional features, the need for training samples can be effectively reduced.

[0008] The technical solution of this invention is implemented as follows: This invention provides a remote sensing target matching method based on convolutional neural networks that fuse local features, comprising the following steps:

[0009] S1. Obtain the reference image and the image to be matched;

[0010] S2. Preprocess the reference image and the image to be matched;

[0011] S3. The maximum length trace generation tree algorithm divides the preprocessed reference image and the image to be matched into regions, resulting in multiple patch images of the reference image and the image to be matched.

[0012] S4. The segmentation algorithm segments the patched image to obtain multiple candidate regions of the reference image and the image to be matched;

[0013] S5. Using the CNN region selection algorithm, generate region proposals for each candidate region, and integrate the candidate regions of the reference image and the image to be matched based on similarity.

[0014] S6. Using a local feature matching detection operator, extract SIFT features from each candidate region after merging in step S5.

[0015] S7. Locate potential copies of the detected region proposals on the candidate regions, and retain only the top K region proposals with the most SIFT features.

[0016] S8. The pre-trained convolutional neural network extracts image features from the candidate regions retained in step S6 to obtain multi-scale convolutional features.

[0017] S9. Extract fast CNN features from the activation of the last convolutional layer of the CNN for each candidate region, and then use a similarity matching algorithm to match the candidate regions of the reference image and the image to be matched to measure the similarity between the reference image and the image to be matched.

[0018] S10. The multi-scale convolutional features of each candidate region of the reference image and the image to be matched are combined using the RANSAC algorithm, and the geometric constraints between the features of the fast CNN are used to remove erroneous matching pairs from the obtained matching results to obtain the final matching result.

[0019] Based on the above technical solutions, preferably, the obtained reference image and the image to be matched are downloaded from an internet dataset or acquired by drone photography.

[0020] Based on the above technical solutions, preferably, the preprocessing in step S2 includes format conversion, size cropping, data annotation, and data augmentation.

[0021] Based on the above technical solutions, preferably, the maximum length trace generation tree algorithm in step S3 specifically includes the following steps:

[0022] S31. Contour extraction algorithm extracts multiple longest trace lines from the image;

[0023] S32. Based on the distance from the endpoint of the line to the boundary and the length of the line, find the longest trace line as the Mainroot branch;

[0024] S33. Generate a maximum generation trace tree using normal detection and tangent extension methods;

[0025] S34. The image is coarsely partitioned using the maximum generated trace tree to form several patch images.

[0026] A further preferred embodiment uses the GenTree algorithm to perform the normal detection and tangent extension described in step S33. The specific steps are as follows:

[0027] S331. Find the endpoint of the longest trace line closest to the branch of the Main root;

[0028] S332. Draw a circle with the longest trace line closest to the Main root branch as the center until it is tangent to the Main root branch, forming a point of tangency.

[0029] S333, Connect the endpoints near the Main root branch with the tangent points formed on the Main root branch to form an intermediate spanning tree;

[0030] S334. Draw a circle with the end of the longest remaining trace line closest to the middle spanning tree as the center, form a tangent point on the middle spanning tree, and connect the endpoints to the tangent point to form a new middle spanning tree;

[0031] S335. Repeat step S334 until there are no longer any isolated longest trace lines, and obtain the maximum generated trace tree;

[0032] S336. Extend the other end of the longest trace line that makes up the largest trace generation tree by tangent line until the edge of the image.

[0033] Based on the above technical solutions, preferably, the region selection algorithm in step S5 is a selective search algorithm, and step S5 includes the following sub-steps:

[0034] S51. The selective search algorithm takes the candidate regions in step S4 as input to obtain a set of target candidate region suggestion boxes;

[0035] S52. Add the set of suggestion boxes for all candidate regions to the candidate region list;

[0036] S53. Merge candidate regions based on similarity and use the merged candidate region as a new candidate region.

[0037] Based on the above technical solutions, preferably, the pre-trained convolutional neural network in step S8 is a VGG16 network.

[0038] More preferably, the fifth-stage convolution operation in the VGG16 network is X-convolution.

[0039] More preferably, the specific steps of the X-convolution are as follows:

[0040] S81. Using a 3*3 convolution template, convolution is performed once every sliding window in a 5*5 image to obtain a convolution result unit.

[0041] S82. For the skipped sliding window, in the final convolution matrix, convolution result units without blanks will be formed. Find the convolution result unit with the smallest value. For the blank convolution result unit adjacent to the smallest convolution result unit, calculate the average value of the convolution result units directly adjacent to the smallest convolution result unit and the blank convolution result unit, and fill the obtained value into this blank convolution result unit.

[0042] S83. Fill the blank convolution result cell with the smallest value among the neighboring cells of the remaining blank convolution result cell.

[0043] Based on the above technical solutions, preferably, the similarity matching algorithm in step S9 defines the candidate region similarity based on the difference between the matching probability η(m) and the non-matching probability η(nm). This represents the maximum difference between the a-th candidate region in the image to be matched and the b-th candidate region in the reference image. This definition satisfies... R is the ratio threshold, set to 0.7.

[0044] The remote sensing target matching method based on convolutional neural networks that fuse local features, as described in this invention, has the following advantages over existing technologies:

[0045] (1) Compared with CNN features based on the whole image, the CNN features extracted at the region level have higher robustness and flexibility in identifying small similar regions between partially matched images. Through the potential matching region pair generation step, matching region pairs between images can be accurately matched, which makes the CNN features extracted from these regions more robust for partially repeated image matching. This is because the region proposal model based on coarse partitioning efficiently matches local features between images to select a small number of top-ranked candidate images, which can greatly reduce the target detection range of partially repeated images.

[0046] (2) By fully utilizing the good robustness of SIFT features and the high resolution of CNN features, this scheme can achieve relatively accurate matching;

[0047] (3) Due to the large size and slow computation speed of remote sensing images, a fast X-convolution method is specifically proposed, which can achieve relatively fast convolution with approximate accuracy, thereby improving the computation speed of the entire convolution network.

[0048] (4) The maximum length trace generation tree algorithm in this scheme can quickly divide the image into regions in parallel;

[0049] (5) Make full use of the multi-layer convolution of CNN to obtain multi-scale convolution features. Due to its rich detailed features, it can effectively reduce the need for training samples. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of the remote sensing target matching method based on convolutional neural networks with fused local features according to the present invention;

[0052] Figure 2 This is a schematic diagram illustrating the implementation of the maximum length trace generation tree algorithm of the remote sensing target matching method based on the fusion of local features of the present invention;

[0053] Figure 3 This is a schematic diagram of the VGG-16 network structure of the convolutional neural network remote sensing target matching method based on fused local features according to the present invention;

[0054] Figure 4 Implementation diagram of ordinary convolution in the prior art;

[0055] Figure 5 This is a schematic diagram of the convolution process of the X-convolution method for remote sensing target matching based on convolutional neural networks that fuse local features, as described in this invention.

[0056] Figure 6 for Figure 5 A schematic diagram of the convolution matrix obtained by X-convolution. Detailed Implementation

[0057] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0058] like Figure 1-6 As shown, the present invention provides a remote sensing target matching method based on convolutional neural networks with fused local features, comprising the following steps:

[0059] 1) Obtain the reference image and the image to be matched;

[0060] The reference image and the image to be matched obtained are downloaded from an internet dataset or captured by a drone.

[0061] 2) Preprocess the reference image and the image to be matched;

[0062] The preprocessing includes format conversion, size cropping, data annotation, and data augmentation.

[0063] 3) The maximum length trace generation tree algorithm divides the preprocessed reference image and the image to be matched into regions, resulting in multiple patch images of the reference image and the image to be matched.

[0064] The maximum length trace generation tree algorithm uses a contour extraction algorithm to extract multiple longest trace lines {L1,…,Ln} from the image. This step can be accelerated in parallel on a GPU. Then, FindRoot(dist{L1,…,Lm},Len{L1,…,Lm}) finds the longest trace line as the Mainroot tree based on the distance of the line endpoints from the boundary and the line length. The distance from the boundary should be as small as possible, and the line length should be as long as possible. This step can also be accelerated in parallel on a GPU. The GenTree(Lroot, {L1,…,Lm}) normal detection and tangent extension methods are run to generate a maximum generation trace tree. Finally, the image is coarsely partitioned using the maximum generation trace tree to form several patch images {P1,…,Pm}.

[0065] Specifically, GenTree's normal detection and tangent extension method first finds the endpoint of the longest trace line near the main root branch. Using this endpoint as the center, a circle is drawn until it becomes tangent to the main root branch, forming a tangent point. The endpoint near the main root branch is then connected to this tangent point to form an intermediate spanning tree. The remaining longest trace line is then used as the center of a circle near the intermediate spanning tree to form a tangent point on the intermediate spanning tree. The endpoint is then connected to this tangent point to form a new intermediate spanning tree. This process is repeated until no isolated longest trace lines exist, resulting in the maximum spanning tree. Finally, the other endpoint of the longest trace line forming the maximum spanning tree is extended tangent-wise until the edge of the image is reached.

[0066] In the specific implementation process, such as Figure 2 As shown, firstly, four curves L1, L2, L3, and L4 are extracted; then, L1 is determined as the root of the tree using FindRoot(dist{L1,…,L4},Len{L1,…,L4}); for each curve L2, L3, and L4, the endpoint closest to L1 is used for normal detection, and after detecting the nearest normal, a straight line is connected to form a spanning tree; then, for the other endpoint of each curve L2, L3, and L4, the tangent is extended to the edge of the image, and finally the image is divided into four slice images: A, B, C, D, and E.

[0067] The maximum length trace generation tree algorithm proposed in this invention can quickly divide an image into regions in parallel, and is also applicable to other image processing methods.

[0068] 4) The segmentation algorithm segments the patched image to obtain multiple candidate regions for the reference image and the image to be matched;

[0069] The segmentation algorithm described above is an edge-based segmentation algorithm. Since different regions usually have structural abrupt changes or discontinuities, these places often provide an effective basis for image segmentation. These discontinuous or structurally abrupt places are called edges. Different regions in an image usually have obvious edges. Using edge information can effectively segment different regions. Commonly used image edge detection operators include: Laplace operator, Sobel operator, Canny operator, etc.

[0070] 5) Using CNN region selection algorithms, such as Selective Search, EdgeBox, and Region Proposal Network (RPN), region proposals are generated for each candidate region, and the candidate regions of the reference image and the image to be matched are integrated based on similarity.

[0071] This embodiment uses a selective search algorithm for illustration. The candidate region is used as the input of the selective search algorithm to obtain a set of target candidate region suggestion boxes. The suggestion boxes of all candidate regions are added to the candidate region list. The candidate regions are continuously merged based on similarity, and the merged candidate region is used as a new candidate region.

[0072] 6) Use local feature matching detection operators, such as SIFT, SURF, and PCA-SIFT, to extract SIFT features in each merged candidate region;

[0073] The obtained SIFT features possess characteristics such as feature scale, dominant orientation, and coordinates.

[0074] 7) Locate potential copies of the detected region proposals on the candidate regions, and retain only the top K region proposals with the most SIFT features;

[0075] By fully utilizing the feature scale, dominant orientation, and coordinates of the matched SIFT features, potential copies of the detected region proposals are located on the image, i.e., the set of region proposal output proposal boxes. For the candidate regions used, only the top K region proposals with the most SIFT features are retained, and the region proposals after K are removed, which can effectively save computational resources, where K is a user-defined value.

[0076] 8) The pre-trained convolutional neural network extracts image features from the retained candidate regions to obtain multi-scale convolutional features;

[0077] The first K regions of each slicing image are proposed as candidate regions and input into a pre-trained convolutional neural network. Multiple convolutional layers in the neural network are used to extract image features and obtain multi-scale convolutional features.

[0078] Specifically, such as Figure 3As shown, the pre-trained convolutional neural network uses the VGG16 network. VGG-16 contains 5 convolutional computation blocks, each of which has 2-3 convolutional layers and max pooling layers. In specific implementation, this invention uses the feature vector output by the fifth layer of the VGG-16 model as the high-level feature extracted by the CNN, FP(x,y)={f3x3-xconv}.

[0079] In the improved VGG-16 network, the fifth-stage convolution operation is modified to fast X-convolution, which improves the algorithm speed while maintaining the overall matching accuracy.

[0080] like Figure 4 As shown, the existing technology uses a 3*3 convolution template to directly perform sliding convolution in a 5*5 image. Each slide yields a convolution value, and nine convolutions are required to obtain the final convolution matrix.

[0081] like Figure 5 As shown, the specific steps of X-convolution are as follows: Using a 3*3 convolution template, convolution is performed once every other sliding window in a 5*5 image. This results in the actual convolutioned value being only half the size of a normal convolution. Then, for the skipped sliding windows, in the final convolution matrix, the smallest convolution result unit is found, and the average of this unit and its direct convolution result units is calculated: c(i,j) = Avg(c(i-1,j), c(i+1,j)); c(i,j) =Avg(c(i,j-1),c(i,j+1)), where the other empty convolution result units use the convolution value of their smallest neighbor as their value, c(i,j)=Min{Adj(c(i,j))}, Adj(c(i,j))={c(i-1,j-1), c(i-1,j), c(i-1,j+1), c(i,j-1), c(i,j+1), c(i+1,j-1), c(i+1,j), c(i+1,j+1)}, as shown. Figure 6 As shown, for the skipped sliding window, in the final convolution matrix, first find the smallest convolution result unit 2, i.e., c(3,1), and then calculate the average of c(3,1) and its two directly adjacent element points c(1,1) and c(3,3) to get 3 and 3, which are used as the convolution value 3 of c(2,1) and the convolution value 3 of c(3,2) respectively. For the other empty convolution result units, c(1,2) and c(2,3), the convolution value 3 of the smallest neighbor is used as their value. This convolution method is not limited to 3*3 convolution, and its variation is also applicable to dilated convolution.

[0082] 9) Extract fast CNN features from the activation of the last convolutional layer of the CNN for each candidate region, and then use a similarity matching algorithm to match the candidate regions of the reference image and the image to be matched to measure the similarity between the reference image and the image to be matched.

[0083] Similarity matching algorithms define candidate region similarity based on the difference between the matching probability η(m) and the non-matching probability η(nm). This represents the maximum difference between the a-th candidate region in the image to be matched and the b-th candidate region in the reference image. This definition satisfies... R is the ratio threshold, set to 0.7.

[0084] 10) The multi-scale convolutional features of each candidate region of the reference image and the image to be matched are combined using the RANSAC algorithm, and the geometric constraints between the features of the fast CNN are used to remove erroneous matching pairs from the obtained matching results to obtain the final matching results.

[0085] It should be noted that directly using segmentation algorithms for image segmentation will result in inaccurate segmentation and a large difference from the logical regions in reality. Therefore, the maximum length trace spanning tree algorithm is first used for coarse division of the regions.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing target matching method based on convolutional neural networks fusing local features, characterized in that, Includes the following steps: S1. Obtain the reference image and the image to be matched; S2. Preprocess the reference image and the image to be matched; S3. The maximum length trace generation tree algorithm divides the preprocessed reference image and the image to be matched into regions, resulting in multiple patch images of the reference image and the image to be matched. The maximum length trace generation tree algorithm includes the following steps: S31. Contour extraction algorithm extracts multiple longest trace lines from the image; S32. Based on the distance from the endpoint of the line to the boundary and the length of the line, find the longest trace line as the main root branch; S33. Generate a maximum generation trace tree using normal detection and tangent extension methods; S34. The image is coarsely partitioned using the maximum generation trace tree to form several patch images; S4. The segmentation algorithm segments the patched image to obtain multiple candidate regions of the reference image and the image to be matched; S5. Using the CNN region selection algorithm, generate region proposals for each candidate region, and integrate the candidate regions of the reference image and the image to be matched based on similarity. S6. Using a local feature matching detection operator, extract SIFT features from each candidate region after merging in step S5. S7. Locate potential copies of the detected region proposals on the candidate regions, and retain only the top K region proposals with the most SIFT features. S8. The pre-trained convolutional neural network extracts image features from the candidate regions retained in step S7 to obtain multi-scale convolutional features. S9. Extract fast CNN features from the activation of the last convolutional layer of the CNN for each candidate region, and then use a similarity matching algorithm to match the candidate regions of the reference image and the image to be matched to measure the similarity between the reference image and the image to be matched. S10. The multi-scale convolutional features of each candidate region of the reference image and the image to be matched are combined using the RANSAC algorithm, and the geometric constraints between the features of the fast CNN are used to remove erroneous matching pairs from the obtained matching results to obtain the final matching result.

2. The remote sensing target matching method based on convolutional neural networks with fused local features as described in claim 1, characterized in that, The obtained reference image and the image to be matched are downloaded from an internet dataset or acquired by drone photography.

3. The remote sensing target matching method based on convolutional neural networks with fused local features as described in claim 1, characterized in that, The preprocessing described in step S2 includes format conversion, size cropping, data annotation, and data augmentation.

4. The remote sensing target matching method based on convolutional neural networks with fused local features as described in claim 1, characterized in that, The normal detection and tangent extension described in step S33 are performed using the GenTree algorithm. The specific steps are as follows: S331. Find the endpoint of the longest trace line closest to the branch of the Main root; S332. Draw a circle with the longest trace line closest to the Main root branch as the center until it is tangent to the Main root branch, forming a point of tangency. S333, Connect the endpoints near the Main root branch with the tangent points formed on the Main root branch to form an intermediate spanning tree; S334. Draw a circle with the end of the longest remaining trace line closest to the middle spanning tree as the center, form a tangent point on the middle spanning tree, and connect the endpoints to the tangent point to form a new middle spanning tree; S335. Repeat step S334 until there are no longer any isolated longest trace lines, and obtain the maximum generated trace tree; S336. Extend the other end of the longest trace line that makes up the largest trace generation tree by tangent line until the edge of the image.

5. The remote sensing target matching method based on convolutional neural networks with fused local features as described in claim 1, characterized in that, The region selection algorithm described in step S5 is a selective search algorithm, and step S5 includes the following sub-steps: S51. The selective search algorithm takes the candidate regions in step S4 as input to obtain a set of target candidate region suggestion boxes; S52. Add the set of suggestion boxes for all candidate regions to the candidate region list; S53. Merge candidate regions based on similarity and use the merged candidate region as a new candidate region.

6. The remote sensing target matching method based on convolutional neural networks with fused local features as described in claim 1, characterized in that, The pre-trained convolutional neural network mentioned in step S8 is a VGG16 network.

7. The remote sensing target matching method based on convolutional neural networks with fused local features as described in claim 6, characterized in that, The fifth-stage convolutional operation in the VGG16 network is X-convolution, and the specific steps of X-convolution are as follows: S81. Using a 3*3 convolution template, convolution is performed once every sliding window in a 5*5 image to obtain a convolution result unit. S82. For the skipped sliding window, blank convolution result units will be formed in the final convolution matrix. Find the convolution result unit with the smallest value. For the blank convolution result unit adjacent to the smallest convolution result unit, calculate the average value of the convolution result units directly adjacent to the smallest convolution result unit and the blank convolution result unit, and fill the obtained value into this blank convolution result unit. S83. Fill the blank convolution result cell with the smallest value among the neighboring cells of the remaining blank convolution result cell.

8. The remote sensing target matching method based on convolutional neural networks with fused local features as described in claim 1, characterized in that, The similarity matching algorithm described in step S9 is based on matching probability. and mismatch probability The difference is used to define the similarity of candidate regions. , representing the maximum difference between the a-th candidate region in the image to be matched and the b-th candidate region in the reference image, this definition satisfies R is the ratio threshold, set to 0.7.

Images