Method, device and equipment for matching points of interest in remote sensing images and medium

Abstract

The present disclosure provides a method, device, equipment and medium for matching interest points of remote sensing images; the method comprises: obtaining a target remote sensing image pair to be processed; the target remote sensing image pair comprises a reference remote sensing image and a remote sensing image to be matched; inputting the target remote sensing image pair into a pre-trained remote sensing image interest point matching model, and outputting at least one group of matched interest points through the pre-trained remote sensing image interest point matching model; the pre-trained remote sensing image interest point matching model is trained based on at least one group of training data pairs, and each group of training data pairs in the at least one group of training data pairs comprises a reference remote sensing sample image and a remote sensing sample image to be matched. The method provided by the embodiment of the present disclosure can effectively process the remote sensing image interest point matching task, and can perform excellent performance in various complex scenes.

Description

Technical Field

[0001] This disclosure relates to the field of satellite remote sensing image feature matching technology in computer vision, and in particular to a method, apparatus, electronic device, non-transitory computer-readable storage medium, and computer program product for matching points of interest in remote sensing images. Background Technology

[0002] With the rapid development of remote sensing technology, remote sensing image acquisition technology has become increasingly advanced, and it is now possible to acquire remote sensing images with higher spatial and spectral resolutions. Furthermore, the fusion and composite application of multiple platforms, multiple layers, multiple sensors, multiple temporal phases, multiple spectra, multiple angles, and multiple spatial resolutions has become an important development direction for remote sensing technology.

[0003] Image matching is a crucial step in remote sensing impact processing within the aforementioned fusion and composite application scenarios. Image matching refers to the process of comparing and analyzing two or more images to find points of interest (e.g., commonalities or feature points) between them. Its purpose is to establish correspondences between the two or more images and use this as a basis for further measurement and analysis. Since the matched images often originate from different sensors and exhibit differences in viewpoint, lighting, scale, and other aspects, research on accurate, efficient, and robust feature matching techniques remains a key focus in the field of remote sensing. This research can serve as fundamental research for applications such as image registration, image stitching, image fusion, change detection, target recognition and tracking, and 3D reconstruction, driving the advancement and development of related technologies.

[0004] Currently, existing technologies for point-of-interest (POI) matching in remote sensing images mainly include sparse feature point matching and dense feature point matching. However, sparse feature point matching suffers from poor robustness, while dense feature point matching suffers from difficulties in matching, mismatches, low matching efficiency, and high computational complexity. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, non-transitory computer-readable storage medium, and computer program product for matching points of interest in remote sensing images, in order to address at least one of the deficiencies in the prior art.

[0006] This disclosure provides a method for matching points of interest in remote sensing images, including:

[0007] Acquire a target remote sensing image pair to be processed; wherein, the target remote sensing image pair includes a reference remote sensing image and a remote sensing image to be matched;

[0008] The target remote sensing image is input into a pre-trained remote sensing image interest point matching model, and the pre-trained remote sensing image interest point matching model outputs at least one set of matching interest points.

[0009] The pre-trained remote sensing image point of interest matching model is trained based on at least one set of training data pairs, each of which includes a reference remote sensing sample image and a remote sensing sample image to be matched.

[0010] This disclosure also provides an apparatus for matching points of interest in remote sensing images, comprising:

[0011] The data acquisition module is configured to acquire a pair of target remote sensing images to be processed; wherein the target remote sensing image pair includes a reference remote sensing image and a remote sensing image to be matched.

[0012] The matching inference module is configured to: input the target remote sensing image pair into a pre-trained remote sensing image interest point matching model, and output at least one set of matching interest points through the pre-trained remote sensing image interest point matching model;

[0013] The pre-trained remote sensing image point of interest matching model is trained based on at least one set of training data pairs, each of which includes a reference remote sensing sample image and a remote sensing sample image to be matched.

[0014] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for matching points of interest in remote sensing imagery as described above.

[0015] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for matching points of interest in remote sensing images as described above.

[0016] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method for matching points of interest in remote sensing imagery as described above.

[0017] As described above, the method for matching points of interest in remote sensing images provided in this disclosure, as a first aspect, firstly, through multi-scale feature extraction, can not only capture local detailed features in remote sensing images but also obtain global contextual information. This multi-level feature representation provides richer options for subsequent matching. Secondly, by employing a layer-by-layer downsampling method, the dimensionality of the data can be significantly reduced when processing high-resolution remote sensing images, thereby reducing the computational load. In addition, through a high-scale constrained matching strategy, each level of matching can be performed based on the previous level, which also greatly reduces the computational load and improves matching efficiency. As a second aspect, by introducing a location encoding mechanism, features can be distinguished based on location information when facing scenes with similar textures (such as different buildings with similar shapes), thereby avoiding duplicate matching and mismatches. Furthermore, by introducing location information, more contextual information can be considered during feature matching, thus making the matching results more accurate. Thirdly, by introducing a spatial attention mechanism, the model can focus on key areas to extract deeper features when dealing with sparse textured scenes (such as deserts and oceans), thereby significantly improving the matching accuracy of low-texture points and / or textureless points, and achieving good matching results even in sparse textured scenes. Fourthly, by introducing a self-attention mechanism, the pre-trained remote sensing image point of interest matching model can ensure that it considers the dependencies between features when processing features, thus generating richer feature representations. By introducing a mutual attention mechanism, feature similarity calculation can be optimized, thereby further improving matching accuracy. In addition, for densely textured regions (numerous feature points and complex textures) in remote sensing images, the self-attention and mutual attention mechanisms described above can increase the robustness of the pre-trained remote sensing image point of interest matching model in such complex scenes.

[0018] In summary, this disclosure, through the introduction of multi-scale feature extraction, location encoding mechanism, spatial attention mechanism, and self-attention and mutual attention mechanisms, can not only effectively handle the task of matching points of interest in remote sensing images, but also demonstrate excellent performance in various complex scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a method for matching points of interest in remote sensing images according to an embodiment of this disclosure.

[0021] Figure 2 This is a schematic diagram of the structure of the pre-trained remote sensing image point of interest matching model provided in the embodiments of this disclosure;

[0022] Figure 3 This is a schematic diagram of the structure of the spatial attention subunit (i.e., spatial attention) provided in the embodiments of this disclosure;

[0023] Figure 4 This is a schematic diagram of the structure of the matching subunit (i.e., Match Module) provided in the embodiments of this disclosure;

[0024] Figure 5 This is a flowchart illustrating the training process of the pre-trained remote sensing image point of interest matching model provided in this embodiment of the disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of the apparatus for matching points of interest in remote sensing images provided in the embodiments of this disclosure;

[0026] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] Overview of the inventive concept:

[0029] The inventors of this disclosure have discovered through repeated research that the cause of the problems in the prior art is as follows:

[0030] First, for sparse feature point matching schemes, the matching results obtained by this scheme are not robust because remote sensing images often have a lack of texture, changes in illumination, and changes in viewpoint.

[0031] Secondly, for schemes based on dense feature point matching: On the one hand, remote sensing images often contain single homogeneous image regions (e.g., deserts, water surfaces, etc.), leading to matching difficulties; remote sensing images may contain regions with repetitive textures (e.g., farmland, woodlands, etc.), leading to mismatches; remote sensing images may contain occlusion, leading to matching failures; and the increasing number of matching observations for dense feature point matching in remote sensing images often results in exponentially increasing computational complexity; that is, high computational complexity. On the other hand, since the imaging mode of remote sensing images (especially satellite images) is often linear array, it cannot satisfy epipolar relationships. When performing multi-view stereo reconstruction, a large-scale search is required, leading to low efficiency; and since the image size of remote sensing images is much larger than that of images taken by ordinary cameras, it also causes the problem of low matching efficiency. Finally, since the number of bands in remote sensing images often differs from that of ordinary RGB or monochrome cameras, and the resolution between different bands is also different; in addition, when using remote sensing satellites for stereo observation, it may be stereo imaging on the same orbit, stereo imaging on different orbits, or imaging from different satellite sensors, all of which can further increase the difficulty of matching.

[0032] In view of this, the inventors of this disclosure have further discovered that, with the continuous development of deep learning technology, it is possible to combine deep learning methods with interest point matching of remote sensing images, thereby providing a deep learning-based remote sensing image matching scheme (see the embodiments below for details), which can be used to at least solve one of the above-mentioned prior art problems.

[0033] Example:

[0034] The scheme for matching points of interest in remote sensing images disclosed herein will be described below with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic flowchart illustrating a method for matching points of interest in remote sensing imagery, provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices (e.g., servers or cloud computing platforms), such as... Figure 1 As shown, the method for matching points of interest in remote sensing images includes the following steps:

[0036] S110. Acquire the target remote sensing image pair to be processed.

[0037] The target remote sensing image pair includes a reference remote sensing image and a remote sensing image to be matched.

[0038] This disclosure does not limit the specific method of "acquiring". For example, an electronic device (e.g., a server or cloud computing platform) that performs step S110 may communicate with a storage server that stores the "target remote sensing image pair to be processed" to obtain the "target remote sensing image pair to be processed".

[0039] Remote sensing image (RS) refers to film or photographs that record the magnitude of electromagnetic waves emitted by various ground features, and can mainly include aerial photographs and satellite photographs.

[0040] S120. Input the target remote sensing image pair into the pre-trained remote sensing image interest point matching model, and output at least one set of matching interest points through the pre-trained remote sensing image interest point matching model.

[0041] The pre-trained remote sensing image point of interest matching model is trained based on at least one set of training data pairs, each of which includes a reference remote sensing sample image and a remote sensing sample image to be matched.

[0042] The pre-trained remote sensing image point of interest matching model may include network / unit structures capable of multi-scale feature extraction and computational optimization under high-scale constraints, anti-repetition matching under location encoding mechanisms, low-texture / textureless point matching under spatial attention mechanisms, and self-attention and mutual attention mechanisms. Specific implementation methods will be described below and will not be elaborated here.

[0043] As described above, the method for matching points of interest in remote sensing images provided in this disclosure, as a first aspect, firstly, through multi-scale feature extraction, can not only capture local detailed features in remote sensing images but also obtain global contextual information. This multi-level feature representation provides richer options for subsequent matching. Secondly, by employing a layer-by-layer downsampling method, the dimensionality of the data can be significantly reduced when processing high-resolution remote sensing images, thereby reducing the computational load. In addition, through a high-scale constrained matching strategy, each level of matching can be performed based on the previous level, which also greatly reduces the computational load and improves matching efficiency. As a second aspect, by introducing a location encoding mechanism, features can be distinguished based on location information when facing scenes with similar textures (such as different buildings with similar shapes), thereby avoiding duplicate matching and mismatches. Furthermore, by introducing location information, more contextual information can be considered during feature matching, thus making the matching results more accurate. Thirdly, by introducing a spatial attention mechanism, the model can focus on key areas to extract deeper features when dealing with sparse textured scenes (such as deserts and oceans), thereby significantly improving the matching accuracy of low-texture points and / or textureless points, and achieving good matching results even in sparse textured scenes. Fourthly, by introducing a self-attention mechanism, the pre-trained remote sensing image point of interest matching model can ensure that it considers the dependencies between features when processing features, thus generating richer feature representations. By introducing a mutual attention mechanism, feature similarity calculation can be optimized, thereby further improving matching accuracy. In addition, for densely textured regions (numerous feature points and complex textures) in remote sensing images, the self-attention and mutual attention mechanisms described above can increase the robustness of the pre-trained remote sensing image point of interest matching model in such complex scenes.

[0044] In summary, this disclosure, through the introduction of multi-scale feature extraction, location encoding mechanism, spatial attention mechanism, and self-attention and mutual attention mechanisms, can not only effectively handle the task of matching points of interest in remote sensing images, but also demonstrate excellent performance in various complex scenarios.

[0045] exist Figure 1 Based on the embodiments, as an optional implementation method, refer to Figure 2 The pre-trained remote sensing image point of interest matching model uses a U-shaped network with a twin encoder structure as the main network. Specifically, the pre-trained remote sensing image point of interest matching model includes an encoder and a decoder.

[0046] The twin encoder includes a first network branch connected in parallel (i.e., Figure 2 In the middle, reference image X ref (as input to the corresponding network) and the second network branch (the image to be matched X) match(The network corresponding to the input). The first network branch includes a cascaded first multi-scale feature extraction unit and a first fusion position coding unit; the second network branch includes a cascaded second multi-scale feature extraction unit and a second fusion position coding unit.

[0047] It should be noted that both the first multi-scale feature extraction unit and the second multi-scale feature extraction unit include low-, medium-, and high-order feature extraction. Specifically, refer to... Figure 2 The third-order feature extraction of the first network branch corresponds to the outputs F_ref1, F_ref2, and F_ref3, respectively; the third-order feature extraction of the second network branch corresponds to the outputs F_match1, F_match2, and F_match3, respectively. Low-order feature extraction corresponds to Low-scale, medium-scale, and high-scale.

[0048] Among them, the most important role of multi-scale feature extraction is to refine the matching results through downsampling, while reducing the amount of computation by combining it with upsampling constraints in the decoder; and the calculation process of these three scales is basically the same.

[0049] The decoder includes a third network branch (i.e., Figure 2 In the output network, the third network branch includes a cascaded first matching subunit (i.e., Figure 2 In the Decoder network shown, from bottom to top according to the data flow, the first Match Module... e ), the first remodeling subunit (i.e., Figure 2 The Decoder network shown consists of three units arranged from bottom to top according to the data flow: the first Reshape unit, the first spatial attention matching unit, and the second spatial attention matching unit.

[0050] Based on the above-described implementation method, as an optional example, please continue to refer to... Figure 2 The first multi-scale feature extraction unit includes a cascaded (based on the input reference image X) ref The first image block subunit (i.e., data flow direction) Figure 2 The Patch Partition shown), the first linear embedding subunit (i.e. Figure 2 The Linear Embedding shown), the first image block downsampling subunit (i.e., Figure 2 According to the input reference image X ref The data flow direction, the first Patch Merging), the first normalized subunit (i.e., Figure 2 According to the input reference image X refThe data flow direction, the first Layer Norm), the second image block downsampling subunit (i.e., Figure 2 According to the input reference image X ref The data flow direction, the second Patch Merging) and the second normalization subunit (i.e., Figure 2 According to the input reference image X ref The data flow direction, the second LayerNorm).

[0051] Based on the above implementation method, as an optional example, please continue to refer to... Figure 2 The first fusion position encoding unit includes a cascaded first flattening subunit (i.e., Figure 2 According to the input reference image X ref The data flow direction, the first Flatten) and the first position coding subunit (i.e., Figure 2 According to the input reference image X ref The data flow direction, the first Positional Encoding).

[0052] Based on the above implementation method, as an optional example, please continue to refer to... Figure 2 The second multi-scale feature extraction unit includes a cascaded ((according to the input image to be matched X)) match The data flow direction) is the second image block subunit (i.e. Figure 2 The Patch Partition shown), the second linear embedding subunit (i.e. Figure 2 The Linear Embedding shown), the third image block downsampling subunit (i.e., Figure 2 According to the input image X to be matched match The data flow direction, the first PatchMerging), the third normalization subunit (i.e., Figure 2 According to the input image X to be matched mutch The data flow direction, the first Layer Norm), the fourth image block downsampling subunit (i.e., Figure 2 According to the input image X to be matched match The data flow direction, the second Patch Merging) and the fourth normalization subunit (i.e., Figure 2 According to the input image X to be matched match The data flow direction, the second Layer Norm).

[0053] Based on the above implementation method, as an optional example, please continue to refer to... Figure 2 The second fusion position encoding unit includes a cascaded second flattening subunit (i.e., Figure 2 According to the input image X to be matchedmatch The data flow direction, the first Flatten) and the second position encoding subunit (i.e., Figure 2 According to the input image X to be matched match The data flow direction, the first Positional Encoding).

[0054] The first spatial attention matching unit includes cascaded (i.e., Figure 2 The first upsampling subunit in the Decoder network shown (from bottom to top according to the data flow) (i.e., Figure 2 In the Decoder network shown, from bottom to top according to the data flow, the first Up Sampling unit and the first spatial attention subunit (i.e., Figure 2 In the decoder network shown, from bottom to top according to the data flow, the first Spatial Attention unit and the third flattening subunit (i.e., Figure 2 In the decoder network shown, from bottom to top according to the data flow direction, the first Flatten), the second matching subunit (i.e., Figure 2 The Decoder network shown is arranged from bottom to top according to the data flow direction, including the second Match Module and the second reshaping subunit (i.e., Figure 2 The Decoder network shown is arranged from bottom to top according to the data flow direction (second Reshape).

[0055] Wherein, the first normalized subunit ((i.e., Figure 2 According to the input reference image X ref The data flow direction, the output of the first Layer Norm) and the first spatial attention subunit (i.e., Figure 2 In the Decoder network shown, following the data flow from bottom to top, the first Spatial Attention input is connected; the third normalization subunit (i.e., Figure 2 According to the input image X to be matched match The data flow direction, the output of the first Layer Norm) and the first spatial attention subunit (i.e., Figure 2 The Decoder network shown is connected from bottom to top according to the data flow direction, with the first SpatialAttention input connected.

[0056] The second spatial attention matching unit includes cascaded (i.e., Figure 2 The second upsampling subunit in the Decoder network shown (from bottom to top according to the data flow) (i.e., Figure 2 In the Decoder network shown, following the data flow from bottom to top, the second Up Sampling and the second spatial attention subunit (i.e., Figure 2 In the Decoder network shown, from bottom to top according to the data flow, the second Spatial Attention unit and the fourth flattening subunit (i.e., Figure 2 In the decoder network shown, from bottom to top according to the data flow direction, the second Flatten), the third matching subunit (i.e., Figure 2 The Decoder network shown is arranged from bottom to top according to the data flow, including the third Match Module and the third reshaping subunit (i.e., Figure 2 In the Decoder network shown, the data flow is from bottom to top, with the third Reshape.

[0057] Wherein, the first linear embedded subunit (i.e. Figure 2 The input reference image X is shown. ref The output of the Linear Embedding in the network is connected to the second spatial attention subunit (i.e., Figure 2 In the decoder network shown, the data flow is from bottom to top, with the second Spatial Attention input connected; the second linear embedding subunit (i.e. Figure 2 The input image X to be matched is shown. match The output of the Linear Embedding in the network is connected to the second spatial attention subunit (i.e., Figure 2 The Decoder network shown is connected from bottom to top according to the data flow direction, with the second SpatialAttention input connected.

[0058] In the above Figure 1 , 2 Based on the embodiments, as an optional implementation method, refer to Figure 3 The first spatial attention subunit (i.e., Figure 3 The Spatial Attention Module and the second Spatial Attention Subunit shown include cascaded max pooling layers (i.e., Figure 3 MaxPool and average pooling layer (i.e., MaxPool) are shown. Figure 3 The diagram shows an AvgPool layer, a first convolutional layer, and an activation function layer; the first convolutional layer has 1 channel.

[0059] In the above Figure 1 , 2 Based on the embodiments, as an optional implementation method, refer to Figure 4 The first matching subunit (i.e., Figure 4 The Match Module shown), the second Match Sub-unit, and the third Match Sub-unit include a self-attention layer (i.e., Figure 4The self-attention layer and mutual attention layer shown are shown. Figure 4 (as shown in the Cross-Attention diagram).

[0060] Below, referring to the above Figures 2-4 The working mechanism of each specific network unit involved in the pre-trained remote sensing image point of interest matching model provided in this embodiment is briefly explained:

[0061] l. About the encoder

[0062] l1) Patch Partition unit, mainly used to divide remote sensing images into blocks and flatten them on the channel.

[0063] As an optional example, adjacent pixels of size 4x4 can be considered as a patch. Specifically, taking a three-channel RGB image as an example, after flattening, each patch changes from the original 16 pixels, 3 channels, and 48 values ​​(4x4x3) to a single pixel with 48 channels and 48 values ​​(1x1x48). For a W*H size C channel (i.e., Figure 2 The reference image X (H×W×C) shown ref and the image to be matched X match After passing through the Patch Partition module, the image becomes W / 4*H / 4*16C in size. The total number of pixel values ​​does not change in this step; only the image structure changes. The image size is reduced by increasing the number of channels, and the amount of data is not lost.

[0064] I2) Linear Embedding unit, mainly used for data dimensionality reduction.

[0065] Specifically, the flattened data is fed into the Linear Embedding unit for linear transformation along the channels, with the output channel size specified as C′. The final image becomes W / 4*H / 4*C′. It should be noted that although PatchPartition and Linear Embedding belong to different network units, they can be implemented using a single fully convolutional layer.

[0066] I3) Patch Merging unit, mainly used for downsampling feature maps, according to... Figure 2 The data processing flow shown indicates that downsampling is required in all three stages except for the first stage.

[0067] Specifically, the Patch Merging unit uses a 2x2 kernel with a stride of 2 for convolution. For an input 4x4 feature map, the Patch Merging unit divides it into four 2x2 regions, concatenates pixels at the same location within each region, and transforms the original feature map into four 2x2 feature maps. Then, these four generated feature maps are concatenated along the channel plane, and the input fully connected layer undergoes linear transformation along the channel plane, ultimately resulting in a feature map with two channels.

[0068] The I4) Layer Norm unit primarily reduces internal covariate bias by normalizing the activation values ​​of each layer to a distribution with a mean of zero and a standard deviation of one, thus helping to accelerate the training process.

[0069] Based on the units 1) to 4) above, the input reference image X is... ref and the image to be matched X match Through multi-stage processing, feature maps F_ref1 and F_match1, F_ref2 and F_match2, and F_ref3 and F_match3 at three different scales (1 / 4, 1 / 8, and 1 / 16 of the original image size) can be extracted.

[0070] The I5) Flatten unit and Positional Encoding unit are mainly used to eliminate similar background interference and reduce false matches.

[0071] The Flatten unit is mainly used to flatten the last extracted feature maps F_ref3 and F_match3, converting the feature maps into an encodeable sequence vector F_ref. flatten and F_match flatten Then the encodeable sequence vector F_ref flatten and F_match flatten Input the Positional Encoding unit.

[0072] The Positional Encoding unit can use sine and cosine positional encoding to encode the flattened feature sequence (i.e., the sequence vector F_ref). flatten and F_match flatten Position encoding is performed to obtain the position-encoded feature F-ref. pos and F_match pos Specifically, the following calculation formulas (1) to (2) can be used:

[0073]

[0074] Among them, PE(pos,i) d represents the value of the pos-th position and the i-th dimension in the position coding matrix. model This represents the dimension of the model embedding vector. This encoding method ensures the positional uniqueness of the feature vector, making it highly effective for eliminating false matches caused by similar backgrounds.

[0075] Furthermore, the encoder also includes an output layer, which sums the outputs of the Flatten unit and the PositionalEncoding unit respectively before outputting the result. Specifically, this can be expressed using the following calculation formulas (3) to (4):

[0076]

[0077] in, Represents matrix addition; F_ref out F_match represents the output features of the reference image after processing by the encoder. out This represents the output features of the image to be matched after processing by the encoder.

[0078] II. Regarding the decoder, we will mainly introduce two units: Spatial Attention and Match Module.

[0079] II1) The Spatial Attention unit takes the maximum and average values ​​of each channel at the input feature layer, stacks these two results, adjusts the number of channels using a single convolution with 1 channel, and then applies a sigmoid activation function. This yields the weights for each feature point in the input feature layer (these weights are between 0 and 1), completing the training of the spatial weights. After obtaining these weights, they are multiplied by the original input feature layer.

[0080] Reference Figure 2 It can be seen that, apart from the feature maps F_ref3 and F_match3 in the third stage, the feature maps F_ref2 and F_match2, F_ref1 and F_match1 extracted in the second and first stages must first enter the corresponding SpatialAttention units for matching in the decoding stage.

[0081] II2) The Match Module unit is mainly used to perform similarity matching calculations. Specifically, it can use a parallel computation method that combines self-attention and mutual attention mechanisms. During the execution phase, the feature map of the last layer in the decoder can undergo N attention calculations based on the aforementioned "self-attention and mutual attention mechanisms," and then output a high-scale feature matching result.

[0082] It should be noted that the purpose of self-attention calculation is to enhance global representation capabilities by calculating the relationships between pixel feature descriptors within an image, ensuring that no multiple points in the reference image are matched with a single point in the image to be matched. The purpose of mutual attention calculation is to select an optimal matching point for the reference image's descriptor by calculating the relationships between pixel feature descriptors between two images.

[0083] As an alternative example, self-attention and mutual attention calculations can be performed using the following formulas (5) to (14):

[0084]

[0085]

[0086] Where d represents the number of channels in each network layer of the Match Module unit; x and y represent feature descriptors; A hw×C This represents the matrix obtained by concatenating all pixels in the feature descriptor; This represents the transformation matrix required to generate the corresponding query vector from the pixels in the feature descriptor x; This represents the transformation matrix required to generate the corresponding query vector from the pixels in the feature descriptor y; This represents the transformation matrix required to generate the corresponding key vector from the pixels in the feature descriptor x; This represents the transformation matrix required to generate the corresponding key vector from the pixels in the feature descriptor y; This represents the transformation matrix required to generate the corresponding value vector from the pixels in the feature descriptor x; This represents the transformation matrix required to generate the corresponding value vector from the pixels in the feature descriptor y; SelfAttention x SelfAttention represents the self-attention estimate of the corresponding feature descriptor x. y CrossAttention represents the self-attention estimate of the corresponding feature descriptor y. x CrossAttention represents the estimated cross-attention value for the corresponding feature descriptor x. y This represents the mutual attention estimate of the corresponding feature descriptor y.

[0087] The following is an overview of the "matching optimization based on high-scale constraints" process implemented in the decoder:

[0088] Reference image X ref and the image to be matched X matchAfter processing by the encoder, three feature maps of different scales can be generated (i.e., feature maps F_ref1 and F_match1, F_ref2 and F_match2, and F_ref3 and F_match3). The high-scale feature maps (i.e., F_ref3 and F_match3) undergo positional encoding and spatial attention similarity matching to output the high-scale matching result (i.e., F_ref1 and F_match1). out and F_match out At this point, the matching result is still a flattened sequence vector.

[0089] Next, after the flattened sequence vector in the decoder is processed by the Match Module unit, it can be further reshaped by the Reshape unit to resemble the feature vector. Figure 1 Two-dimensional feature maps M_ref3 and M_match3 of the same size are generated; where the high-scale pixels matched in the two-dimensional feature maps are M_ref3(x,y) and M_match3(x,y); here, only the feature vectors of the matched pixels with high confidence can be retained, and the other vectors can be zeroed out, thereby reducing the amount of computation in subsequent processing steps.

[0090] In this embodiment, only the matched regions between the two images in the high-scale result are upsampled and added to the mid-scale feature map of the previous level to constrain the matching region. The feature map with high-scale constraints can then be processed by the matching sub-unit in the next spatial attention matching unit (i.e., Figure 2 The Match Module shown in the image performs similarity matching to obtain matching results M_ref2 and M_match2, and then upsamples them. This method is also used for the feature map of the next higher level until the resolution of the upsampled image is consistent with that of the original image. Finally, the matching results M_ref1 and M_match1 are output.

[0091] Based on the above embodiments, as an optional implementation method, refer to Figure 5 The training process of the pre-trained remote sensing image interest point matching model includes:

[0092] S510. Obtain the at least one set of training data pairs.

[0093] As an optional example, S510 may include the following steps:

[0094] 1) Prepare a large number of satellite remote sensing images of the same area from different perspectives;

[0095] 2) From satellite remote sensing imagery, select multiple matching satellite remote sensing image pairs according to preset criteria. Each satellite remote sensing image pair is recorded as a training data pair, including one remote sensing sample image to be matched and one corresponding reference remote sensing sample image. The preset criterion is that the two images in each satellite remote sensing image pair have a certain degree of overlap.

[0096] Optionally, multiple matched satellite remote sensing image pairs can be divided into training and testing datasets in a 9:1 ratio. The testing dataset can be used to test the "remote sensing image point of interest matching model" after it has been trained using the training dataset, in order to further optimize the model parameters.

[0097] S520. Input the reference remote sensing sample image and the remote sensing sample image to be matched in each training data pair into the remote sensing image interest point matching model to be trained, and output at least one set of predicted matching interest points through the remote sensing image interest point matching model to be trained.

[0098] Each of the at least one set of predicted matching points of interest includes one point of interest belonging to the reference remote sensing sample image and one point of interest belonging to the remote sensing sample image to be matched.

[0099] S530. Using a preset loss function, based on the at least one set of predicted matching interest points, obtain the function value of the loss function.

[0100] As an optional example, the Euclidean geometric distance between interest points belonging to the reference remote sensing sample image and interest points belonging to the remote sensing sample image to be matched can be calculated based on each set of predicted matching interest points, and used as the function value of the loss function.

[0101] Specifically, the function value of the loss function can be calculated using the following formulas (15) to (16):

[0102]

[0103] Where L3 represents the function value of the loss function; d (i，j) M represents the Euclidean geometric distance between points of interest belonging to the reference remote sensing image and points of interest belonging to the remote sensing image to be matched; g The true matching space is determined based on the camera pose of the camera that acquired the image to be matched; (i, j) represents the position number corresponding to the predicted matching interest point.

[0104] Among them, (X_ref i Y_ref j) represents the two-dimensional coordinates in the real world of the point of interest belonging to the reference remote sensing sample image, obtained based on rigorous pose estimation.

[0105] Among them, (X_match) i Y_match j ) represents the two-dimensional coordinates in the real world of the point of interest belonging to the remote sensing sample image to be matched, obtained based on rigorous pose estimation.

[0106] S540. Based on the function value of the loss function, train the remote sensing image interest point matching model to be trained until the preset training completion condition is met, and obtain the pre-trained remote sensing image interest point matching model from the remote sensing image interest point matching model to be trained.

[0107] As an optional example, step S540 can be implemented as follows: the function value of the loss function is back-transmitted to the remote sensing image interest point matching model to be trained; the remote sensing image interest point matching model to be trained adjusts the network parameters of each network layer according to the function value of the loss function; the step of obtaining the function value of the loss function is iteratively executed up to the step of adjusting the network parameters of each network layer until the preset training completion condition is met.

[0108] Among them, the preset training completion condition can be that the number of iterations is greater than a preset threshold; or, the function value of the loss function is less than a preset loss threshold.

[0109] As described above, the method for matching points of interest in remote sensing images provided in this disclosure, as a first aspect, firstly, through multi-scale feature extraction, can not only capture local detailed features in remote sensing images but also obtain global contextual information. This multi-level feature representation provides richer options for subsequent matching. Secondly, by employing a layer-by-layer downsampling method, the dimensionality of the data can be significantly reduced when processing high-resolution remote sensing images, thereby reducing the computational load. In addition, through a high-scale constrained matching strategy, each level of matching can be performed based on the previous level, which also greatly reduces the computational load and improves matching efficiency. As a second aspect, by introducing a location encoding mechanism, features can be distinguished based on location information when facing scenes with similar textures (such as different buildings with similar shapes), thereby avoiding duplicate matching and mismatches. Furthermore, by introducing location information, more contextual information can be considered during feature matching, thus making the matching results more accurate. Thirdly, by introducing a spatial attention mechanism, the model can focus on key areas to extract deeper features when dealing with sparse textured scenes (such as deserts and oceans), thereby significantly improving the matching accuracy of low-texture points and / or textureless points, and achieving good matching results even in sparse textured scenes. Fourthly, by introducing a self-attention mechanism, the pre-trained remote sensing image point of interest matching model can ensure that it considers the dependencies between features when processing features, thus generating richer feature representations. By introducing a mutual attention mechanism, feature similarity calculation can be optimized, thereby further improving matching accuracy. In addition, for densely textured regions (numerous feature points and complex textures) in remote sensing images, the self-attention and mutual attention mechanisms described above can increase the robustness of the pre-trained remote sensing image point of interest matching model in such complex scenes.

[0110] In summary, this disclosure, through the introduction of multi-scale feature extraction, location encoding mechanism, spatial attention mechanism, and self-attention and mutual attention mechanisms, can not only effectively handle the task of matching points of interest in remote sensing images, but also demonstrate excellent performance in various complex scenarios.

[0111] The apparatus for matching points of interest in remote sensing images provided in this disclosure is described below. The apparatus for matching points of interest in remote sensing images described below can be referred to in correspondence with the method for matching points of interest in remote sensing images described above.

[0112] Figure 6 This is a schematic diagram of the structure of an apparatus for matching points of interest in remotely sensed images, provided in an exemplary embodiment of this disclosure. Figure 6 As shown, the apparatus for matching points of interest in remote sensing images includes:

[0113] The data acquisition module 610 is configured to: acquire a pair of target remote sensing images to be processed; wherein the pair of target remote sensing images includes a reference remote sensing image and a remote sensing image to be matched; the matching inference module 620 is configured to: input the pair of target remote sensing images into a pre-trained remote sensing image interest point matching model, and output at least one set of matching interest points through the pre-trained remote sensing image interest point matching model; wherein the pre-trained remote sensing image interest point matching model is trained based on at least one set of training data pairs, and each set of training data pairs includes a reference remote sensing sample image and a remote sensing sample image to be matched.

[0114] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a method for matching points of interest (POIs) in remote sensing images. This method includes: acquiring a pair of target remote sensing images to be processed; wherein the target remote sensing image pair includes a reference remote sensing image and a remote sensing image to be matched; inputting the target remote sensing image pair into a pre-trained remote sensing image POI matching model, and outputting at least one set of matched POIs through the pre-trained remote sensing image POI matching model; wherein the pre-trained remote sensing image POI matching model is trained based on at least one set of training data pairs, each of the at least one set of training data pairs including a reference remote sensing sample image and a remote sensing sample image to be matched.

[0115] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] On the other hand, this disclosure also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the method for matching points of interest in remote sensing images provided by the above methods. The method includes: acquiring a pair of target remote sensing images to be processed; wherein the pair of target remote sensing images includes a reference remote sensing image and a remote sensing image to be matched; inputting the pair of target remote sensing images into a pre-trained remote sensing image point of interest matching model, and outputting at least one set of matched points of interest through the pre-trained remote sensing image point of interest matching model; wherein the pre-trained remote sensing image point of interest matching model is trained based on at least one set of training data pairs, and each set of training data pairs includes a reference remote sensing sample image and a remote sensing sample image to be matched.

[0117] In another aspect, this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for matching points of interest in remote sensing images provided by the methods described above. This method includes: acquiring a pair of target remote sensing images to be processed; wherein the target remote sensing image pair includes a reference remote sensing image and a remote sensing image to be matched; inputting the target remote sensing image pair into a pre-trained remote sensing image point of interest matching model, and outputting at least one set of matched points of interest through the pre-trained remote sensing image point of interest matching model; wherein the pre-trained remote sensing image point of interest matching model is trained based on at least one set of training data pairs, each of the at least one set of training data pairs including a reference remote sensing sample image and a remote sensing sample image to be matched.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A method for matching points of interest in remote sensing imagery, comprising: Acquire a target remote sensing image pair to be processed; wherein, the target remote sensing image pair includes a reference remote sensing image and a remote sensing image to be matched; The target remote sensing image is input into a pre-trained remote sensing image interest point matching model, and the pre-trained remote sensing image interest point matching model outputs at least one set of matching interest points. The pre-trained remote sensing image point of interest matching model is trained based on at least one set of training data pairs, and each set of training data pairs includes a reference remote sensing sample image and a remote sensing sample image to be matched. The pre-trained remote sensing image interest point matching model includes an encoder and a decoder; The encoder includes a first network branch and a second network branch connected in parallel; wherein, the first network branch includes a first multi-scale feature extraction unit and a first fusion position coding unit connected in cascade; the second network branch includes a second multi-scale feature extraction unit and a second fusion position coding unit connected in cascade. The decoder includes a third network branch; wherein the third network branch includes a cascaded first matching subunit, a first reshaping subunit, a first spatial attention matching unit, and a second spatial attention matching unit; the first multi-scale feature extraction unit includes a cascaded first image block subunit, a first linear embedding subunit, a first image block downsampling subunit, a first normalization subunit, a second image block downsampling subunit, and a second normalization subunit. The first fusion position coding unit includes a cascaded first flattening subunit and a first position coding subunit; The second multi-scale feature extraction unit includes a cascaded second image block subunit, a second linear embedding subunit, a third image block downsampling subunit, a third normalization subunit, a fourth image block downsampling subunit, and a fourth normalization subunit; The second fusion position coding unit includes a cascaded second flattening subunit and a second position coding subunit; The first spatial attention matching unit includes a cascaded first upsampling subunit, a first spatial attention subunit, a third flattening subunit, a second matching subunit, and a second reshaping subunit; wherein, the output of the first normalization subunit is connected to the input of the first spatial attention subunit; the output of the third normalization subunit is connected to the input of the first spatial attention subunit; the first spatial attention subunit and the second spatial attention subunit include a cascaded max pooling layer, a flat pooling layer, a first convolutional layer, and an activation function layer; wherein, the first convolutional layer has 1 channel; the first matching subunit, the second matching subunit, and the third matching subunit include a self-attention layer and a mutual attention layer; The second spatial attention matching unit includes a cascaded second upsampling subunit, a second spatial attention subunit, a fourth flattening subunit, a third matching subunit, and a third reshaping subunit; wherein, the output of the first linear embedding subunit is connected to the input of the second spatial attention subunit; and the output of the second linear embedding subunit is connected to the input of the second spatial attention subunit.

2. The method according to claim 1, characterized in that, The training process of the pre-trained remote sensing image point of interest matching model includes: Obtain the at least one set of training data pairs; The reference remote sensing sample image and the remote sensing sample image to be matched in each training data pair are respectively input into the remote sensing image interest point matching model to be trained, and the remote sensing image interest point matching model to be trained outputs at least one set of predicted matching interest points; wherein, each set of predicted matching interest points in the at least one set includes one interest point belonging to the reference remote sensing sample image and one interest point belonging to the remote sensing sample image to be matched. Using a preset loss function, the function value of the loss function is obtained based on the at least one set of predicted matching interest points; Based on the function value of the loss function, the remote sensing image interest point matching model to be trained is trained until the preset training completion condition is met, and the pre-trained remote sensing image interest point matching model is obtained from the remote sensing image interest point matching model to be trained.

3. The method according to claim 2, characterized in that, The step of obtaining the function value of the loss function based on the at least one set of predicted matching interest points using a preset loss function includes: Based on each set of predicted matching interest points, the Euclidean geometric distance between interest points belonging to the reference remote sensing sample image and interest points belonging to the remote sensing sample image to be matched is calculated and used as the function value of the loss function.

4. The method according to claim 2, characterized in that, The process of training the remote sensing image interest point matching model based on the function value of the loss function until the preset training completion conditions are met includes: The value of the loss function is then passed back to the remote sensing image interest point matching model to be trained. The remote sensing image interest point matching model to be trained adjusts the network parameters of each network layer according to the function value of the loss function; The process iteratively executes the steps from obtaining the function value of the loss function to adjusting the network parameters of each network layer until the preset training completion conditions are met.

5. An apparatus for matching points of interest in remote sensing imagery, comprising: The data acquisition module is configured to acquire a pair of target remote sensing images to be processed; wherein the target remote sensing image pair includes a reference remote sensing image and a remote sensing image to be matched. The matching inference module is configured to: input the target remote sensing image pair into a pre-trained remote sensing image interest point matching model, and output at least one set of matching interest points through the pre-trained remote sensing image interest point matching model; The pre-trained remote sensing image point of interest matching model is trained based on at least one set of training data pairs, and each set of training data pairs includes a reference remote sensing sample image and a remote sensing sample image to be matched. The pre-trained remote sensing image interest point matching model includes an encoder and a decoder; The encoder includes a first network branch and a second network branch connected in parallel; wherein, the first network branch includes a first multi-scale feature extraction unit and a first fusion position coding unit connected in cascade; the second network branch includes a second multi-scale feature extraction unit and a second fusion position coding unit connected in cascade. The decoder includes a third network branch; wherein the third network branch includes a cascaded first matching subunit, a first reshaping subunit, a first spatial attention matching unit, and a second spatial attention matching unit; the first multi-scale feature extraction unit includes a cascaded first image block subunit, a first linear embedding subunit, a first image block downsampling subunit, a first normalization subunit, a second image block downsampling subunit, and a second normalization subunit. The first fusion position coding unit includes a cascaded first flattening subunit and a first position coding subunit; The second multi-scale feature extraction unit includes a cascaded second image block subunit, a second linear embedding subunit, a third image block downsampling subunit, a third normalization subunit, a fourth image block downsampling subunit, and a fourth normalization subunit; The second fusion position coding unit includes a cascaded second flattening subunit and a second position coding subunit; The first spatial attention matching unit includes a cascaded first upsampling subunit, a first spatial attention subunit, a third flattening subunit, a second matching subunit, and a second reshaping subunit; wherein, the output of the first normalization subunit is connected to the input of the first spatial attention subunit; the output of the third normalization subunit is connected to the input of the first spatial attention subunit; the first spatial attention subunit and the second spatial attention subunit include a cascaded max pooling layer, a flat pooling layer, a first convolutional layer, and an activation function layer; wherein, the first convolutional layer has 1 channel; the first matching subunit, the second matching subunit, and the third matching subunit include a self-attention layer and a mutual attention layer; The second spatial attention matching unit includes a cascaded second upsampling subunit, a second spatial attention subunit, a fourth flattening subunit, a third matching subunit, and a third reshaping subunit; wherein, the output of the first linear embedding subunit is connected to the input of the second spatial attention subunit; and the output of the second linear embedding subunit is connected to the input of the second spatial attention subunit.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for matching points of interest in remote sensing images as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for matching points of interest in remote sensing images as described in any one of claims 1 to 4.

Images