Method for quickly positioning remote sensing image at any position

Abstract

The invention provides a method for quickly positioning a remote sensing image at any position, which comprises the step of constructing a database and further comprises the following steps of: acquiring a remote sensing image needing to be positioned, and naming the remote sensing image as a test image; the test image is preprocessed; partitioning the preprocessed remote sensing image and performing feature extraction; performing feature aggregation and sorting; carrying out reordering and primary positioning; secondary sorting and accurate positioning are carried out; the accurate positioning result is evaluated; and when the positioning accuracy E is greater than an accuracy threshold, outputting a final positioning position of the test image. According to the method for rapidly positioning the remote sensing image at any position, an innovative two-stage positioning strategy is adopted, an advanced VPR model and a RERANK technology are fused, and rapid and accurate positioning of the remote sensing image is effectively improved under the condition that no fine adjustment is carried out.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image positioning, in particular to a method for rapid positioning of remote sensing images at any position. Background Art

[0002] Research on visual place recognition (VPR) technology has traditionally focused on the field of street view positioning, while its application in multi-source optical remote sensing images is relatively rare. Although the visual model pre-trained based on large-scale street view data can achieve a TOP1 accuracy of more than 0.95 after fine-tuning, there is a significant semantic gap between optical remote sensing images and natural images due to their unique bird's-eye view shooting perspective. This difference leads to a significant decrease in the TOP1 and TOP5 accuracy of the VPR model fine-tuned directly using street view data when processing optical remote sensing images.

[0003] The Chinese invention patent application with application number CN115630236A discloses a global rapid retrieval and positioning method, storage medium and equipment for passive remote sensing images, wherein the positioning method includes: establishing a global reference image tile database, extracting local feature points and feature descriptors of tiles, performing feature aggregation on local feature point features, generating a global feature vector of tiles as a first global feature vector, establishing a global reference image tile data feature library, and generating a second global feature vector to be matched by the same method for the passive remote sensing image to be matched, performing feature comparison and similarity check on the second global feature vector and the first global feature vector, calculating the image with the highest similarity as the matching image, and taking the geographic coordinates of the matching image as the geographic coordinates of the passive image to be matched. The disadvantages of this method are: 1) The dimension of the generated global feature vector is very high, resulting in a very high storage space required to establish a global benchmark image tile database and a very high amount of computational effort to subsequently calculate the feature vector similarity; 2) It relies on the extraction and feature aggregation of local feature points, which results in a strong dependence on low-level features, making it easily affected by changes in lighting and weather, less adaptable to long-term changes, and performing poorly in low-texture or low-feature scenes; 3) There is a lack of verification mechanism, and global remote sensing image positioning only relies on the distance between two 256-dimensional vectors to determine whether the match is successful, resulting in a low success rate. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a method for rapid positioning of remote sensing images at any position, which adopts an innovative two-stage positioning strategy and integrates the advanced VPR model and RERANK technology to effectively improve the rapid and accurate positioning of remote sensing images without any fine-tuning.

[0005] The purpose of the present invention is to provide a method for quickly locating remote sensing images at any position, comprising building a database and further comprising the following steps: Step 1: Obtain the remote sensing image that needs to be located and name it as a test image; Step 2: preprocessing the test image; Step 3: Divide the preprocessed remote sensing image into blocks and perform feature extraction; Step 4: Perform feature aggregation and sorting; Step 5: Re-sort and perform initial positioning; Step 6: Perform secondary sorting and precise positioning; Step 7: Evaluate the quasi-positioning result; Step 8: When the positioning accuracy E When the accuracy is greater than the threshold, the final positioning position of the test image is output.

[0006] Preferably, the database includes an offline benchmark image data set, a test data set and a FAISS vector feature library.

[0007] In any of the above solutions, preferably, the first i Baseline images T i Randomly crop according to BLOCK_SIZE*BLOCK_SIZE pixels m sub-region image blocks, generate a sub-region image block set T i,j ={ T i,1 , T i,2 , …, T i,m},in, j Indicates j sub-regions, 1≤ j ≤ m .

[0008] In any of the above solutions, preferably, the sub-region image set T i,j Each image in is rotated, scaled and denoised and stored in the test dataset.

[0009] In any of the above schemes, preferably, an overlapping area of ​​overlap pixels is provided between each of the sub-region image blocks, global features are extracted from each sub-region image block, and these features are stored in the FAISS vector feature library.

[0010] In any of the above schemes, preferably, the preprocessing includes calculating the minimum circumscribed quadrilateral of the effective area of ​​the test image, and affine transforming the test image into a "vertical" state accordingly.

[0011] In any of the above solutions, preferably, the blocking includes cutting the test image according to BLOCK_SIZE*BLOCK_SIZE pixels to generate N test image blocks.

[0012] In any of the above schemes, preferably, the feature extraction includes extracting global features for each of the test image blocks, retrieving TOPK nearest features in the FAISS vector feature library, and obtaining the sub-region image blocks corresponding to these features, the name of the benchmark image and the position of the benchmark image.

[0013] In any of the above schemes, preferably, step 4 includes the following sub-steps: Step 41: Count all retrieved N*TOPK features; Step 42: for each global feature, check whether there is an adjacent global feature, and when there is an adjacent feature, the count of the corresponding global feature is increased; Step 43: sorting according to the counts of the global features, and if the counts are the same, performing secondary sorting according to the order of TOPK; Step 44: Record the test video block corresponding to each global feature in the sorting result.

[0014] In any of the above schemes, preferably, step 5 includes performing RERANK processing on the test image and the reference image based on the sorting result, and using LOFTR to perform fine image matching. When the number of matched key points exceeds a given threshold and the number of internal points after RANSAC affine transformation meets the requirements, the preliminary positioning is deemed to be successful.

[0015] In any of the above schemes, preferably, step 6 includes performing secondary LOFTR matching on the roughly matching area between the vertical image and the reference image after the initial determination is successful, and obtaining the precise position of the reference image by matching the four corners of the reference image.

[0016] In any of the above schemes, preferably, the evaluation method is: Assume that Y Test image, for y A test image, assuming its prediction box is B yp , the real frame is B yg , the predicted box and the real box are determined by the coordinates of the two vertices of the upper left corner and the lower right corner. IoU y Value is B yp and B yg The area of ​​their intersection divided by their union area is calculated as: IoU y = ( B yp ∩ B yg ) / ( B yp ∪ B yg ) exist Y The positioning accuracy on the test images E The calculation formula is as follows: .

[0017] The present invention proposes a method for quickly locating remote sensing images at any position, thereby improving the effectiveness of the visual position recognition capability of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a flowchart of a preferred embodiment of the method for rapid positioning of remote sensing images at any position according to the present invention.

[0019] Figure 2 The present invention is a flow chart of another embodiment of the satellite image visual positioning method for rapid positioning of remote sensing images at any position according to the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0021] Example 1 like Figure 1 As shown, a method for rapid positioning of remote sensing images at any position, executing step 100, constructing a database, the database includes an offline benchmark image data set, a test data set and a FAISS vector feature library.

[0022] The first i Baseline images T i Randomly crop according to BLOCK_SIZE*BLOCK_SIZE pixels m sub-region image blocks, generate a sub-region image block set T i,j ={ T i,1 , T i,2 , …, T i,m},in, j Indicates j sub-regions, 1≤ j ≤m .

[0023] Sub-region image collection T i,j Each image in the image block is rotated, scaled and denoised, and stored in the test data set. An overlapping area of ​​overlap pixels is set between each sub-region image block, and global features are extracted from each sub-region image block, and these features are stored in the FAISS vector feature library.

[0024] Step 110 is executed to obtain a remote sensing image to be positioned and named as a test image.

[0025] Execute step 120 to pre-process the test image. The pre-processing includes calculating the minimum circumscribed quadrilateral of the effective area of ​​the test image and affine transforming the test image into a "vertical" state accordingly.

[0026] Execute step 130 to divide the preprocessed remote sensing image into blocks and perform feature extraction. The block division includes cutting the test image according to BLOCK_SIZE*BLOCK_SIZE pixels to generate N test image blocks.

[0027] The feature extraction includes extracting global features from each of the test image blocks, retrieving TOPK nearest features in the FAISS vector feature library, and obtaining the sub-region image blocks corresponding to these features, the name of the reference image and the position of the reference image.

[0028] Executing step 140 to aggregate and sort features includes the following sub-steps: Execute step 141 to count all retrieved N*TOPK features; Execute step 142, for each global feature, check whether there is an adjacent global feature, when there is an adjacent feature, the count of the corresponding global feature is increased; Execute step 143, sort according to the counts of the global features, and if the counts are the same, perform secondary sorting according to the order of TOPK; Execute step 144 to record the test video block corresponding to each global feature in the sorting result.

[0029] Execute step 150 to perform reordering and initial positioning, including performing RERANK processing on the test image and the reference image based on the sorting result, and using LOFTR to perform fine image matching. When the number of matched key points exceeds a given threshold and the number of inner points after RANSAC affine transformation meets the requirements, the initial positioning is considered to be successful.

[0030] Execute step 160 to perform secondary sorting and precise positioning, including performing secondary LOFTR matching on the roughly matching area between the vertical image and the reference image after the initial positioning is successful, and obtaining the precise position of the reference image by matching the four corners of the reference image.

[0031] Execute step 170 to evaluate the quasi-positioning result, wherein the evaluation method is: Assume that Y Test image, for y A test image, assuming its prediction box is B yp , the real frame is B yg , the predicted box and the real box are determined by the coordinates of the two vertices of the upper left corner and the lower right corner. IoU y Value is B yp and B yg The area of ​​their intersection divided by their union area is calculated as: IoU y = ( B yp ∩ B yg ) / ( B yp ∪ B yg ) exist Y The positioning accuracy on the test images E The calculation formula is as follows:

[0032] .

[0033] Execute step 180, when the positioning accuracy E When the accuracy is greater than the threshold, the final positioning position of the test image is output.

[0034] Example 2 In the existing positioning methods, the entire remote sensing image is compared with all the images in the database one by one. This method is slow on the one hand, and on the other hand, it takes up a lot of computing resources, which slows down the recognition and positioning speed.

[0035] This paper creatively proposes a two-stage VPR strategy, which aims to improve the positioning accuracy of remote sensing images through efficient global feature retrieval and fine local feature reordering. In the first stage, we use the SELAVPR model for global feature search. SELAVPR realizes the hybrid extraction of global and local features by inserting a lightweight adapter into the pre-trained model, where the global feature dimension is only 1024 dimensions, which is conducive to saving storage space and is particularly suitable for large-scale remote sensing image databases.

[0036] Among many VPR models, SELAVPR was selected for its compact global feature representation. This model not only retains strong feature extraction capabilities, but also significantly reduces storage requirements by optimizing feature dimensions, which is critical for remote sensing image positioning applications that require offline storage of a large number of global features. We only use the global features of SELAVPR to build a feature database to support efficient online retrieval.

[0037] For the global retrieval results, the second stage uses LOFTR (Transformer-based Local Feature Matcher) for fine local feature matching and re-ranking. LOFTR abandons the traditional feature detection step and directly processes the dense feature map, using the Transformer's self-attention and cross-attention mechanism to capture global context information, achieving the ability to generate high-quality matches even in low-texture, blurred or repeated patterns.

[0038] To further improve positioning accuracy, we proposed a spatial mutual nearest neighbor global TOPK ranking method. This method effectively compensates for the low accuracy of TOP1 and TOP5 in remote sensing images based on the VPR model TOPK retrieval, and achieves more accurate location estimation by comprehensively considering the spatial distribution and feature similarity of candidate images.

[0039] Example 3 A method for quickly positioning remote sensing images at any position, comprising: 1. Offline stage: database construction In the process of implementing VPR technology, the core task of the offline stage is to build a comprehensive and efficient database. The database focuses on storing a series of pre-collected images that cover a wide range of locations and perspectives of the target environment, providing a solid data foundation for subsequent online processing. For VPR systems based on remote sensing satellite images, this stage is particularly important because it requires not only capturing a broad geographical field of view, but also accurately processing image scaling and overlap to ensure that the database can reflect the macro characteristics of the terrain without losing the accuracy of details.

[0040] 2. Online stage: image retrieval and image registration Entering the online stage, the system needs to have real-time or near real-time image processing capabilities to cope with dynamically changing query requirements. This stage includes two core links: Image retrieval: When faced with a new query image, the system first starts the image retrieval mechanism to quickly search for candidate images that best match the query image in the built database. This process relies on efficient feature matching algorithms or deep learning technology to quickly narrow the search scope, improve recognition efficiency, and ensure that possible target locations can be quickly located in massive data.

[0041] Image registration: Once the candidate image is selected, the system will enter the image registration stage. This stage deeply analyzes the subtle differences between the query image and the candidate image, and accurately calculates the spatial correspondence between the two through advanced technologies such as feature point matching and affine transformation estimation, thereby achieving accurate positioning of the query image in the target environment.

[0042] 3. Dataset To support the verification and optimization of the above technical process, this paper uses the "Tianzhi Cup Arbitrary Area Optical Image Rapid Positioning Dataset" as offline benchmark image data. The dataset consists of 980 high-resolution (1 meter) optical satellite images, each of which is a 5000×5000 pixel RGB three-channel image in a unified TIF format. This dataset is not only large in scale but also rich in content, providing valuable experimental resources for the research and development and application of VPR technology.

[0043] The test dataset of this article consists of two parts: Online unpublished test set: This part contains 200 specially processed datasets, which are derived from the benchmark datasets but have undergone a series of complex transformations to enhance their diversity and challenge. Specifically, the images in the benchmark dataset are randomly cropped into 200 sub-region images, and the original size of each sub-region ranges from 300×300 pixels to 1000×1000 pixels. After cropping, these sub-region images further undergo a variety of image processing operations such as rotation, scaling, and noise addition, and are randomly saved in JPG, PNG, or TIF format. It is worth noting that since these datasets are not publicly available online, they are only used for accuracy verification through algorithm mirroring to evaluate the generalization ability of the model on unknown data.

[0044] Internally generated simulated test sets: In order to supplement the online test sets and increase the diversity of test samples, we randomly generated N test sets based on the processing methods of the online test sets. These simulated data sets strictly follow the preprocessing process of the online test sets, including cropping, rotation, scaling, noise addition, and random format conversion steps, aiming to simulate complex scenes and noise conditions in the real world, so as to more comprehensively evaluate the performance and robustness of the model.

[0045] By combining these two test data sets, we can more comprehensively and accurately evaluate the performance of the proposed algorithm under different conditions, providing strong support for the optimization and improvement of the model.

[0046] 4. Evaluation Method In terms of evaluation indicators, the positioning accuracy and positioning efficiency are used to measure the performance of the algorithm. The positioning accuracy is obtained by accumulating and averaging the positioning accuracy of each test image, and the positioning accuracy of each test image is evaluated by the IOU (intersection over union) indicator.

[0047] Assume that Y Test image, for y A test image, assuming its prediction box is B yp , the real frame is B yg , the predicted box and the real box are determined by the coordinates of the two vertices of the upper left corner and the lower right corner. IoU y Value is B yp and B yg The area of ​​their intersection divided by their union area is calculated as: IoU y = ( B yp ∩ B yg ) / ( B yp ∪ B yg ) exist Y The positioning accuracy on the test images E The calculation formula is as follows:

[0048] .

[0049] Example 4 like Figure 2 As shown in the figure, the experimental process is mainly divided into two stages: offline and online.

[0050] Offline stage In the offline stage, the reference image is first segmented into image blocks of size BLOCK_SIZE × BLOCK_SIZE pixels, with overlapped areas of overlap pixels between adjacent image blocks. Subsequently, the SELAVPR technique is used to extract global features from each image block and store these features in the FAISS vector feature library for subsequent rapid retrieval.

[0051] Online Stage The online stage involves the processing and positioning of the test image. The specific steps are as follows: Preprocessing: Since the test image may have been rotated and stretched asymmetrically in width and height, resulting in a large invalid area, the minimum circumscribed quadrilateral of the valid area is first calculated, and the test image is affine transformed into a "vertical" state based on it to maximize information utilization.

[0052] Block segmentation and feature extraction: The processed test image is cut according to the BLOCK_SIZE size and a certain degree of overlap is considered to generate N test image blocks. Then, the global features of each test image block are extracted using SELAVPR, and the TOPK nearest features are retrieved in the FAISS database, and the corresponding reference image names and image positions (IMAGENAME_LEFT_TOP) of these features are obtained.

[0053] Feature aggregation and sorting: Count all retrieved N*TOPK features, and for each benchmark feature (a benchmark feature is a feature of each record in the benchmark image dataset, that is, a global feature, and a global feature is related to the image), check whether there is an adjacent benchmark feature (based on IMAGENAME consistency and position proximity). If there is an adjacent feature, the count of the corresponding benchmark feature is increased; otherwise, the count is 0. Then, sort according to the count. If the counts are the same, perform a secondary sort according to the TOPK order. At the same time, record the test image corresponding to each benchmark feature.

[0054] Reranking and preliminary positioning: Based on the ranking results, the test image and the reference image are processed by RERANK, and LOFTR is used for fine image matching. If the number of matched key points exceeds the given threshold and the number of inliers after RANSAC affine transformation meets the requirements, the preliminary positioning is considered successful.

[0055] Secondary matching and precise positioning: After the initial positioning is successful, a secondary LOFTR matching is performed on the approximate matching area between the “vertical image” and the reference image to further narrow the positioning range and achieve more accurate position identification.

[0056] Experiment 1 was conducted to evaluate the performance of the image matching and location recognition algorithms. First, the first experiment focused on the internal test set, which was preprocessed to remove images lacking significant features (such as images covered by full vegetation) and uniformly adjusted to a "vertical" state. The experiment used BLOCK_SIZE=256 to block the image and generate N test image blocks. Subsequently, SELAVPR was used to extract features, and global features with a sort count greater than 1 and in the top 60% were selected through feature aggregation and sorting strategies to reduce the amount of calculation. On this basis, RERANK processing and LOFTR fine matching were performed on the test image and the reference image. If the number of matched key points and the number of inliers after RANSAC affine transformation meet the preset conditions, the preliminary positioning is considered successful. Subsequently, a secondary LOFTR match is performed on the roughly matched areas of the "vertical" image and the reference image to further improve the positioning accuracy. This experiment generated a total of 5 groups of 200 test data each. As shown in the statistical results of Table 1, the success rates and time consumption of TOP1, TOP5, TOP10 and TOP128 are excellent. In particular, TOP128 detects almost all images, and the average single image processing time is less than 2 seconds. The average accuracy of 0.99638 reflects high efficiency and accuracy.

[0057] Table 1 Evaluation results of optical satellite positioning on the internal test set

[0058] Experiment 2 was conducted for the online test set, with the goal of fully locating all images. The experimental process was similar to the first experiment, but a loop processing step was added to meet the competition requirements, and two adjustments were made to BLOCK_SIZE (256 and 386) to accommodate different image features. First, the TOP128 strategy successfully located most of the images (197), and only three complex images failed to be recognized. Subsequently, the TOP384 strategy was used to successfully locate these three difficult images, as shown in the statistical results in Table 2.

[0059] Table 2 Evaluation results of optical satellite positioning on the online test set

[0060]

[0061] In summary, the experimental process effectively improved the efficiency and accuracy of visual position recognition. By recording the initial positions of the four corner points of the test image and the relative positions after transformation, the accuracy of positioning is ensured. In particular, by giving priority to features with the highest counts and counts greater than 0 after sorting, the speed of local matching is significantly accelerated, and the success rate of positioning is improved. In the Tianzhi Cup competition, the solution achieved an accuracy of 0.995846 based on an online undisclosed test set, ranking first in the Tianzhi Cup competition, demonstrating its efficiency and accuracy. It has been calculated that the average matching accuracy of each image has reached the sub-pixel level (0.6-0.7 pixels), further verifying its excellent performance.

[0062] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A method for rapid positioning of remote sensing images at any position, comprising building a database, characterized in that: The following steps are also included: Step 1: Obtain the remote sensing image that needs to be located and name it as a test image; Step 2: preprocessing the test image; Step 3: Divide the preprocessed remote sensing image into blocks and perform feature extraction, wherein the block division includes cutting the test image according to BLOCK_SIZE*BLOCK_SIZE pixels to generate N test image blocks; the feature extraction includes extracting global features for each of the test image blocks, retrieving TOPK nearest features in the FAISS vector feature library, and obtaining the sub-region image blocks corresponding to these features, the name of the benchmark image and the location of the benchmark image; Step 4: Perform feature aggregation and sorting; Step 5: Re-sort and perform initial positioning; Step 6: Perform secondary sorting and precise positioning; Step 7: Evaluate the quasi-positioning results; Step 8: When the positioning accuracy E When the accuracy is greater than the threshold, the final positioning position of the test image is output.

2. The method for rapid positioning of remote sensing images at any position as claimed in claim 1, characterized in that: The database includes an offline benchmark image data set, a test data set and a FAISS vector feature library.

3. The method for rapid positioning of remote sensing images at any position as claimed in claim 2, characterized in that: The first i Baseline images T i Randomly crop according to BLOCK_SIZE*BLOCK_SIZE pixels m sub-region image blocks, generate a sub-region image block set T i,j ={ T i,1 , T i,2 , …, T i,m },in, j Indicates j sub-regions, 1≤ j ≤ m .

4. The method for rapid positioning of remote sensing images at any position as claimed in claim 3, characterized in that: Sub-region image collection T i,j Each image in is rotated, scaled and denoised and stored in the test dataset.

5. The method for rapid positioning of remote sensing images at any position as claimed in claim 4, characterized in that: An overlapping area of ​​overlap pixels is set between each of the sub-region image blocks, global features are extracted from each sub-region image block, and the features are stored in the FAISS vector feature library.

6. The method for rapid positioning of remote sensing images at any position as claimed in claim 5, characterized in that: The preprocessing includes calculating the minimum circumscribed quadrilateral of the effective area of ​​the test image, and affine transforming the test image into a "vertical" state accordingly.

7. The method for rapid positioning of remote sensing images at any position as claimed in claim 6, characterized in that: The step 4 includes the following sub-steps: Step 41: Count all retrieved N*TOPK features; Step 42: for each reference feature, checking whether there is an adjacent reference feature, when there is an adjacent feature, the count of the corresponding reference feature is increased; Step 43: sorting according to the counts of the benchmark features, and if the counts are the same, performing secondary sorting according to the order of TOPK; Step 44: Record the test video block corresponding to each benchmark feature in the sorting result.

8. The method for rapid positioning of remote sensing images at any position as claimed in claim 7, characterized in that: The step 5 includes performing RERANK processing on the test image and the reference image based on the sorting result, and performing fine image matching using LOFTR. When the number of matched key points exceeds a given threshold and the number of inner points after RANSAC affine transformation meets the requirements, the preliminary positioning is considered to be successful.

9. The method for rapid positioning of remote sensing images at any position as claimed in claim 8, characterized in that: The step 6 includes performing secondary LOFTR matching on the roughly matching area between the vertical image and the reference image after the initial determination is successful, and obtaining the precise position of the reference image by matching the four corners of the reference image.

10. The method for rapid positioning of remote sensing images at any position as claimed in claim 9, characterized in that: The evaluation method is: Assume that Y Test image, for y A test image, assuming its prediction box is B yp , the real frame is B yg , the predicted box and the real box are determined by the coordinates of the two vertices of the upper left corner and the lower right corner. IoU y Value is B yp and B yg The area of ​​their intersection divided by their union area is calculated as: IoU y = ( B yp ∩ B yg ) / ( B yp ∪ B yg ) exist Y The positioning accuracy on the test images E The calculation formula is as follows: 。

Images