A feature matching method for large-scale remote sensing images

Abstract

The application belongs to the technical field of space remote sensing images, and discloses a feature matching method for large-range remote sensing images, the overlapping areas of each input image and a related image are calculated respectively, then the overlapping areas of the two input images are subjected to feature point extraction and matching by using a SAR-SIFT feature matching method, and the matched feature points are subjected to visual processing, when the feature points are extracted, according to the pyramid image layers constructed according to the SAR-SIFT feature matching method, a CPU unit follows a fork / join mode to divide sub-threads, then a GPU unit is called to calculate the response function of the pyramid image layer corresponding to each sub-thread, and after the response function is calculated, OpenMP multi-thread is used to extract feature points and construct feature point descriptors.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of space remote sensing images, and in particular to a feature matching method for large-range remote sensing images. BACKGROUND

[0002] Satellite remote sensing SAR images have wide applications in image fusion, image stitching, three-dimensional reconstruction, change detection, etc., and GF-3 satellite images have important applications in image stitching, and SAR image matching is the basis for these applications. However, there are problems of large remote sensing images and low direct matching efficiency in fast matching of satellite remote sensing SAR images. In order to improve the matching efficiency of satellite remote sensing images, the present application develops a software for fast matching of SAR images.

[0003] There are two main methods for SAR image matching: intensity-based method and feature-based method. The intensity-based matching method has high computational time complexity and low computational efficiency due to the need to calculate a large number of correlation matrices and the influence of different geometric imaging angles. The current mainstream SAR image matching is mainly based on feature matching, which mainly includes feature extraction, feature descriptor construction, similarity criterion matching, etc. This method has faster matching speed and higher accuracy than the intensity-based matching method. Among them, the representative feature matching method is SIFT (Scale-invariant feature transform). Due to the serious interference of multiplicative noise on feature point extraction of satellite remote sensing SAR images, the accuracy of SIFT in SAR image matching is not high, and SAR-SIFT is a feature matching method that can improve the influence of multiplicative noise, but general satellite SAR images are relatively large, and the existing SAR-SIFT method uses CPU serial calculation with low efficiency, which cannot meet the application scenarios of fast response after natural disasters such as earthquakes. SUMMARY

[0004] The present application provides a feature matching method for large-range remote sensing images, which extracts the overlapping area based on a block strategy, then uses the parallel feature of the SAR-SIFT feature matching algorithm to match and connect points for large-range 7104*5650 remote sensing images, and visualizes the connection points.

[0005] The present application can be implemented by the following technical solutions:

[0006] The application discloses a feature matching method for large-scale remote sensing images, and the method comprises the following steps: calculating the overlapping area of each input image and a related image respectively, extracting and matching feature points in the overlapping area of two input images by using a SAR-SIFT feature matching method, and performing visual processing on the matched feature points.

[0007] Further, all the input images are sequentially numbered, and according to a forward search method, a later input image is taken as a main image, and one or more input images in front are taken as auxiliary images, and whether the main image and the one or more auxiliary images in front have an overlap is calculated in sequence.

[0008] Further, the longitude and latitude coordinates of four corner points of the main image and the one or more auxiliary images in front are calculated in sequence by using a rational function model (RFM), and whether the area surrounded by the corner points has an overlap is judged, if yes, the main image is divided into n*n image blocks, the longitude and latitude coordinates of four corner points of each image block are calculated in sequence by using the RFM, and whether the area surrounded by the corner points has an overlap with the corresponding auxiliary image is judged, if yes, the corresponding overlapping area is marked, and finally the main image and the marked overlapping areas in the auxiliary images are segmented.

[0009] Further, eight-layer pyramid images are constructed by using the SAR-SIFT feature matching method, when the task of constructing the pyramid images is executed, the CPU unit adopts a fork / join mode, pyramid image scale space tasks of each layer are distributed to eight sub-threads one by one by using OpenMP, and then the GPU unit is called to calculate the response function of each layer of the pyramid images.

[0010] Then, the feature points are extracted by using OpenMP in layers during the feature point extraction;

[0011] During the construction of the feature descriptor, N sub-threads are divided by using OpenMP to construct the descriptor of each feature point.

[0012] Finally, the feature points are matched by using the Euclidean distance, and the RANSAC method based on a projection matrix is used to screen out the wrong matching points.

[0013] Further, the GPU unit calculates the corresponding feature scale of each layer of the pyramid images by using CUDA parallel computing.

[0014] After the pyramid image is constructed, OpenMP is used to assign the task of searching feature points and calculating the main direction of the feature points to sub-threads, and then OpenMP assigns the task of processing feature points in parallel to construct feature descriptors to the sub-threads.

[0015] Further, according to the SAR-SIFT feature matching method, first, image gradients are constructed and scale spaces are generated by Gaussian blur, and the scale space response function of each layer is calculated as follows:

[0016]

[0017] R(x,y,σ)=Det(C SH (x,y,σ)-t·Trace(C SH (x,y,σ))

[0018] wherein Det and Trace represent the determinant and trace of the matrix C SH respectively, * represents convolution, represents an image scale smoothing filter function, is a scale factor of the current image, σ is a differential scale determining the scale variable of the local region of the current corner point, G x,σ is a horizontal gradient, and G y,σ is a vertical gradient.

[0019] Feature points are extracted by searching the maximum value in a range of 3*3 in each scale space through the R(x,y,σ) response function, and the response function value is greater than 0.8, and the main direction of the feature points is obtained after the coordinates of the feature points are determined; after the feature points are extracted, the GLOH method is used to construct feature descriptors.

[0020] The present application has the beneficial technical effects that:

[0021] 1. The overlapping area block matching strategy based on the primary and secondary images is proposed, the redundant calculation amount of image matching is reduced, the calculation rate is improved, and the basis is provided for subsequent fast matching and image stitching;

[0022] 2. For the SAR-SIFT parallel computing problem, the fork / join synchronous parallel mode is used to improve the matching efficiency. First, the CPU unit main thread allocates multiple worker threads, calls the GPU unit to calculate the time-consuming response function in the worker thread, and finally the main thread collects multiple worker threads to establish a feature scale space, and OpenMP is used to calculate the feature point extraction and the construction of the feature descriptor which involves more logical calculation in parallel in the CPU unit.

[0023] 3. By visualizing the matched feature points, the distribution of the feature points can be accurately observed, and it is ensured that the user can use the extracted feature points. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 This is a schematic diagram of the system structure of the method of the present invention;

[0025] Figure 2 This is a schematic flowchart of the method of the present invention;

[0026] Figure 3 This is a schematic diagram of the visualization results of a first embodiment of feature matching using the method of the present invention;

[0027] Figure 4 This is a schematic diagram of the visualization results of Example 2 of feature matching using the method of the present invention. Detailed Implementation

[0028] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings and preferred embodiments.

[0029] Currently, most feature matching studies focus on remote sensing images with a resolution of around 1000*1000 pixels. Research on feature matching for large-scale remote sensing images is relatively limited. Therefore, such as Figures 1-2 As shown, this invention provides a feature matching method for large-scale remote sensing images. It calculates the overlapping region of each input image and related images, then uses the SAR-SIFT feature matching method to extract and match feature points in the overlapping region of the two input images. The matched feature points are then visualized. During feature point extraction, the CPU unit follows a fork / join pattern to divide into sub-threads based on the number of pyramid image layers constructed by the SAR-SIFT feature matching method. The GPU unit then calculates the response function of the pyramid image layer corresponding to each sub-thread. After calculating the response function, OpenMP multi-threaded feature point extraction and feature point descriptor construction are used.

[0030] 1) Extract overlapping regions

[0031] When the image file is very large, direct matching is inefficient. Therefore, image preprocessing is required, and a block-based strategy is used to extract the overlapping areas between the images in the image file, as follows:

[0032] All input images are sequentially numbered. Using the forward search method, the next input image is taken as the main image, and the previous one or more input images are taken as sub-images. The overlap between the main image and the previous one or more sub-images is calculated in turn. If there is overlap, the overlapping area between the main image and the corresponding sub-image is segmented out.

[0033] The rational function model RFM can be used to calculate the longitude and latitude coordinates of the four corner points of the main image and the previous one or more sub-images in turn, and judge whether the area surrounded by the corner points has overlap, if yes, the main image is divided into n*n image blocks, and then the rational function model RFM is used to calculate the longitude and latitude coordinates of the four corner points of each image block in turn, and judge whether the area surrounded by the corner points has overlap with the corresponding sub-image, if yes, the corresponding overlapping area is marked, and finally the main image and the marked overlapping area in each sub-image are segmented out.

[0034] 2) Parallel computing SAR-SIFT feature matching algorithm for feature matching

[0035] At present, feature matching algorithms include SIFT algorithm, SURF, SAR-SIFT algorithm, etc., the improved SAR-SIFT algorithm of the present application cooperates with CPU unit and GPU unit, aiming at the problems of large remote sensing image and low direct matching efficiency in fast matching of satellite remote sensing SAR image, and meets the requirement of fast matching, as follows:

[0036] According to the SAR-SIFT feature matching method, first, the image gradient is constructed and the scale space is generated by Gaussian blur, and the scale space response function of each layer is calculated as follows:

[0037]

[0038] R(x,y,σ)=Det(C SH (x,y,σ)-t·Trace(C SH (x,y,σ))

[0039] Wherein, Det and Trace represent the determinant and trace of the matrix C SH , * represents convolution, represents the image scale smoothing filter function, is the scale factor of the current image, σ is the differential scale to determine the scale variable of the local area of the current corner point, G x,σ is the horizontal gradient, G y,σ is the vertical gradient,

[0040] Then, according to the R(x,y,σ) response function, the feature points are extracted, and the maximum value within 3*3 range is found in each scale space and the response function value is greater than 0.8, and the feature main direction is obtained after determining the feature point coordinates; after extracting the feature points, the GLOH method is used to construct the feature descriptor.

[0041] In the construction of pyramid image task, SAR-SIFT collaborative parallel computing is used, the method follows fork / join mode, CPU unit uses OpenMP to distribute pyramid layer scale space task to sub-thread, GPU unit uses CUDA parallel computing feature scale; after the completion of pyramid image construction, OpenMP is used to distribute the task of finding feature points and calculating feature main direction of each layer to sub-thread, then, OpenMP distributes the task of parallel processing feature point construction feature descriptor to sub-thread, and the algorithm flow is as shown in Figure 2

[0042] The pyramid image constructed by SAR-SIFT is different from SIFT pyramid, in SAR-SIFT algorithm, the original image is taken as the bottom layer image, and eight-layer pyramid image is constructed by Gaussian blur in proportion upwards. In thread compilation, 8 independent sub-tasks can be compiled in parallel area, in each sub-task, firstly, Gaussian blur operation is performed on the image, gradient amplitude and gradient direction are calculated; then, the autocorrelation matrix of the pixel point is calculated; finally, the response function of the current layer is calculated.

[0043] Since GPU unit is more suitable for processing image, GPU unit is called to process the response function calculation of each layer of pyramid image. Since GPU unit calculation is in video memory, data in CPU unit memory needs to be transmitted to video memory, but the data transmission process needs to consume a lot of time, therefore, the image is transmitted into memory and video memory at the same time, GPU unit uses CUDA to calculate feature scale, and finally the response function result is transmitted back to memory, avoiding a large number of frequent data transmission between video memory and memory.

[0044] When extracting feature points and calculating feature main direction, eight working threads are generated by OpenMP compiler to traverse each layer of feature pyramid. In the process of constructing feature scale space, N working threads are generated by OpenMP to realize the generation of feature point descriptor, then knnmatch method based on Euclidean distance is used for matching in the matching stage, and RANSAC three times of mean error is used to screen out the wrong matching points.

[0045] 3) Visualization

[0046] The connection points obtained by using SAR-SIFT feature matching, that is, the two matched feature points, are displayed on the corresponding image, so that people can clearly see their approximate positions on the image and obtain the distribution position of the connection points. The visualization system adopts human-computer interaction mode, after the image connection points are extracted, the user manually selects the image connection points to view the distribution.

[0047] ​To verify the feasibility of the method of the application, we take the image of Yanshi County, Yunnan Province on May 26, 2021 as an example, which is a high-resolution three fine strip, and the imaging mode is a full polarization strip 1 (QPSI), covering the high-resolution three fine strip of Yanshi County, Yunnan Province in 2021 (7104*5650), the working frequency band is C band, the data resolution is 8 meters (8 meters in azimuth direction, 6-9 meters in range direction, and 30 km in width), the coverage range is 20-35 km, and the polarization mode is full polarization. The terrain of Yanshi County, Dali City, Yunnan Province is high in the north and low in the south, with the highest point at an altitude of 4122 meters and the lowest point at an altitude of 1174 meters, with a total area of 1954 square kilometers, and the mountainous area accounts for 98.4%.

[0048] (1) Extracting the overlapping area, first calculating the four corner points of each image, then setting each image as the main image and the remaining images as the secondary images, and then judging whether it intersects with other images, if it intersects, then the main image is divided into blocks, and the overlapping area between the secondary images intersecting with it is calculated, and the redundant calculation is reduced;

[0049] Then the improved SAR-SIFT algorithm is used to calculate the connection points of the main image and the secondary image overlapping area. When calculating the connection points, the pyramid image is processed in layers in parallel through the fork / join mode, and OpenMP is used in the CPU unit and CUDA is used in the GPU unit to respectively accelerate the parallel SAR-SIFT matching method. Based on the SAR-SIFT image block matching strategy, when processing 5 scenes covering the high-resolution three images of Yanshi County, Yunnan Province, the CPU serial efficiency is improved by 45 times, and the feature matching efficiency is improved.

[0050] (2) The feature matching connection point visualization can accurately show the distribution of the connection points, and under the premise of meeting the fast feature matching, the extracted connection points can be used by users, as shown in Figure 3 and Figure 4 .

[0051] Although the specific embodiments of the application are described above, those skilled in the art should understand that these are only illustrative, and various changes or modifications can be made to these embodiments without departing from the principles and essence of the application, therefore, the protection scope of the application is defined by the appended claims.

Claims

1. A feature matching method for large-scale remote sensing imagery, the method comprising: The overlapping area of each input image and the related image is calculated respectively, and then the SAR-SIFT feature matching method is used to extract and match the feature points in the overlapping area of the two input images, and the matched feature points are visualized. The eight-layer pyramid image constructed by the SAR-SIFT feature matching method is used to execute the pyramid image construction task, and the CPU unit adopts the fork / join mode to distribute the scale space tasks of each layer of the pyramid image to eight sub-threads one by one, and then the GPU unit is called to calculate the response function of each layer of the pyramid image. Then, the OpenMP is used to extract the feature points layer by layer during the feature point extraction. During the construction of the feature descriptor, the OpenMP is used to divide N sub-threads to construct the descriptor of each feature point. Finally, the Euclidean distance is used for feature point matching, and the RANSAC method based on the projection matrix is used to filter out the wrong matching points. The GPU unit uses CUDA to calculate the feature scale corresponding to each layer of the pyramid image in parallel. After the pyramid image is constructed, the OpenMP is used to distribute the tasks of finding feature points and calculating the main direction of the feature points to the sub-threads, and then the OpenMP allocates the task of constructing the feature descriptor for the sub-threads in parallel. According to the SAR-SIFT feature matching method, first, the image gradient is constructed and the scale space is generated by Gaussian blur, and the scale space response function of each layer is calculated as follows: where Det and Trace represent the determinant and trace of matrix C SH respectively, * denotes convolution, denotes the image scale-dependent smoothing function, is the scale factor of the current image, is the scale variable of the local region of the current corner point determined by differential scale, is the horizontal gradient, is the vertical gradient. 2.The method for feature matching of large-scale remote sensing images of claim 1, wherein: All input images are sequentially numbered, and the forward search method is used, with the next input image as the main image and the previous one or more input images as the secondary image, to calculate whether the main image and the previous one or more secondary images have overlap, and if so, the overlapping area of the main image and the corresponding secondary image is segmented. 3.The method for feature matching of large-scale remote sensing images of claim 2, wherein: The rational function model RFM is used to calculate the longitude and latitude coordinates of the four corner points of the main image and the previous one or more secondary images in turn, and to determine whether the area surrounded by the corner points has overlap, and if so, the main image is divided into n*n image blocks, and then the rational function model RFM is used to calculate the longitude and latitude coordinates of the four corner points of each image block in turn, and to determine whether the area surrounded by the corner points has overlap with the corresponding secondary image, and if so, the corresponding overlapping area is marked, and finally the main image and the marked overlapping areas in each secondary image are segmented.

Images