A Multimodal Remote Sensing Image Keypoint Detection Method Based on a Dual-Branch Network Architecture

Abstract

This invention discloses a multimodal remote sensing image keypoint detection method based on a dual-branch network architecture, comprising the following steps: RGB and SAR images are input into two branch networks with identical structures, each branch including: an encoder, a multi-scale locally shared convolutional module, a global information extraction module, and a lightweight detection head; in each branch, the image is first enlarged by passing six standard convolutional layers with non-shared parameters to quickly obtain dense feature descriptors at minimal cost; local features of different receptive field regions of the image are obtained through parameter-shared serial multi-scale dilated convolution; then, the multi-head attention architecture in Transformer is used to interactively combine global information and local features to obtain the feature descriptor of each pixel; subsequently, average pooling and maximum value operations are used to obtain the relative saliency of each point with local region points; finally, a keypoint detection score map is obtained through softmax.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method for detecting and describing key points in multimodal remote sensing images based on a dual-branch network. Background Technology

[0002] With the continuous advancement of sensor technology and the widespread application of multi-source data, multimodal remote sensing data plays an increasingly important role in geographic information extraction and analysis. In recent years, the rapid development of deep learning has provided strong technical support for feature extraction and information fusion of multimodal remote sensing images, particularly achieving significant progress in keypoint detection and description. In the field of remote sensing, keypoint detection refers to identifying salient and stable pixels in an image, while keypoint description further extracts the local feature information of these points for matching, recognition, and analysis. This task is of great significance for the geometric correction, target recognition, and change detection of remote sensing images.

[0003] Compared to single-modal images, multimodal remote sensing data (such as optical and SAR data) offers significant advantages. Multimodal data not only provides more comprehensive information on target characteristics but also compensates for information gaps in single-modal data caused by lighting conditions, weather variations, or terrain complexity. For example, optical data excels at providing rich texture and color information, while SAR data is robust to weather and lighting conditions. Effective fusion of multimodal data can more comprehensively and accurately characterize the key features of ground targets, greatly improving the reliability and accuracy of key point detection and description.

[0004] Therefore, deep dual-branch networks, proposed in recent years, have become an effective method for keypoint detection and description in multimodal remote sensing images. By extracting features from each modality's data through shared and independent branches, and fully utilizing the complementarity of different modalities in the feature fusion stage, this method can avoid information confusion and feature loss, thereby enhancing the model's robustness and generalization ability.

[0005] Therefore, how to utilize the complementarity and spatial information of multimodal remote sensing images to organically combine the characteristics of different modalities and construct a high-precision, robust keypoint detection and description model is an important problem that urgently needs to be solved in the field of remote sensing image analysis. Summary of the Invention

[0006] To address the problems mentioned above, this invention aims to provide a method for key point detection and description in multimodal remote sensing images using a dual-branch network architecture. This method can fully utilize the multimodal remote sensing images in the training dataset to train the model, thereby enabling the model to automatically detect the location of key points and their feature descriptors.

[0007] To address the aforementioned technical problems, this invention provides a method for detecting key points in multimodal remote sensing images based on a dual-branch network architecture. The method includes the following steps:

[0008] Step 1: Obtain a multimodal remote sensing image dataset, which includes RGB images and SAR images;

[0009] Step 2: Divide the multimodal remote sensing image dataset to form a training database;

[0010] Step 3: Construct a deep neural network model for keypoint detection and description of multimodal remote sensing images. The deep neural network model contains two branches, each consisting of an encoder network, a multi-scale local shared convolutional module, a global information extraction module, and a lightweight detection head. The inputs to the model are two registered RGB images and a SAR image. The output of the global information extraction module is the feature descriptor of each of the two images. The output of the lightweight detection head is the keypoint score of each of the two modal images.

[0011] Step 4: Train the deep neural network model from Step 3 using the training database to obtain the trained deep neural network model for multimodal remote sensing image key point detection and description.

[0012] Step 5: Input the multimodal image to be detected into the deep neural network model trained in Step 4. The global information extraction module outputs the feature descriptor of each pixel. The output value of the last layer of the lightweight detection head is used to calculate the key point score of each pixel through the softmax function, thereby realizing the key point detection and description of multimodal images.

[0013] Furthermore, step 3 specifically includes:

[0014] Step 3.1: Input the RGB image and SAR image into the encoder network. Use a lightweight encoder network to increase the channel dimension to 128 dimensions through 6 standard convolutional layers with non-shared parameters to quickly obtain the first feature map f.

[0015] Step 3.2, establish a multi-scale locally shared convolutional module: For the first feature map f obtained from the encoder network, the output after dilated convolution is represented by the following formula:

[0016] ;

[0017] In the formula, Represents the convolution kernel. Indicates the position on the first feature map f. This indicates the position on the corresponding convolution kernel. This represents the dilation rate; the size of the feature map remains unchanged before and after dilated convolution.

[0018] In each modality, three consecutive dilated convolutions are used to obtain multi-scale features, resulting in a second feature map. The following formula represents:

[0019] ;

[0020] In the formula, It is a 3×3 dilated convolution. For batch standardization layer;

[0021] Step 3.3, establish a global information extraction module: use the output of the obtained multi-scale local shared convolution module as the input of the global information extraction module, stretch it to a flat 2D image block, and then focus on the positional relationship between elements by superimposing relative position encoding;

[0022] Step 3.4, Establish a lightweight detection head: Obtain the saliency score and the maximum channel score through training. The saliency score is calculated using the following formula:

[0023] ;

[0024] In the formula, This indicates that average pooling is performed over a 3×3 area.

[0025] The optimal channel is selected for each point by calculating the maximum score of the channel, as shown in the following formula:

[0026] ;

[0027] Finally, by multiplying the saliency score by the maximum score of each channel and then applying a softmax operation, the keypoint detection score map is obtained, as shown in the following formula:

[0028] .

[0029] Furthermore, skip connections are used to connect the first feature map. Output features after dilated convolution By performing a connection, a third feature map is obtained. As shown in the formula below:

[0030] ;

[0031] This represents a 1×1 pointwise convolution operation, followed by a BN layer and a ReLU activation function.

[0032] Furthermore, the third feature map is in Each position in both dimensions is encoded into a vector of length 128. Finally, the modal vectors are concatted to obtain the final encoded vector. Specifically,

[0033] When encoding each dimension, the sine function is used for even positions and the cosine function is used for odd positions. The encoding is calculated as shown in the following formula:

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] Then concatenate them to obtain a vector of length 256, in order to match the third feature map. The shape remains consistent, and its length is reduced to 128 by a 1×1 convolution;

[0039] Then, it passes through three linear mapping layers, with the weight matrix being... , , , This represents the feature dimension of each pixel in different mapping layers;

[0040] The third feature map Mapped to queries with the same dimensions Keywords Sum : ;

[0041] A self-attention function based on cross-covariance is used to calculate the attention coefficients along the channel dimension, as shown in the following formula:

[0042] ;

[0043] The features are remapped using a residual structure and a linear layer to obtain the final output, as shown in the following formula:

[0044] .

[0045] The beneficial effects of this invention are as follows: This invention can train the model using images with a relatively small amount of data. After registering multimodal remote sensing image data, it can automatically learn the features required for different modalities from the training database and perform key point detection and description through deep learning. During the training process, it continuously corrects the data features used for judgment and adjusts the hyperparameters in the deep neural network, thereby realizing key point detection and description of multimodal images. Attached Figure Description

[0046] Figure 1 This is a structural diagram of the multimodal remote sensing image key point detection and description network model of the present invention;

[0047] Figure 2 This is a network structure diagram of the multi-scale local shared convolution module and the global information extraction module of the present invention;

[0048] Figure 3 The figure shows the experimental results of key point detection and description in this invention. Detailed Implementation

[0049] The technical solution of the present invention will now be fully described with reference to the accompanying drawings. The following description is merely a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0050] Example 1:

[0051] This application discloses a method for key point detection in multimodal remote sensing images based on a dual-branch network architecture, which includes the following steps:

[0052] Step 1: Collect RGB and SAR remote sensing images and register them;

[0053] The collected 3-band RGB images were acquired from Google Earth, with a pixel resolution of 0.5-2 meters. The SAR images were from the GF-3 satellite, with a pixel resolution of 1 meter. The collection includes 10 4468*2844 pixel RGB images and SAR images of the same area, covering fields and towns in multiple cities around the world. The reference image for registration was an RGB image, and registration was performed using professional registration software.

[0054] Step 2: Preprocess the dataset by cropping the images into 512×512 pixel image patches, and divide the dataset to form a training database with a training set:validation set ratio of 8:2.

[0055] Step 3: Construct a neural network model for key point detection and description of multimodal remote sensing images.

[0056] like Figure 1 As shown, a multimodal remote sensing image keypoint detection and description model based on a dual-branch network architecture is presented:

[0057] This model uses RGB and SAR images for keypoint detection and description in remote sensing images. The 3-channel RGB and SAR images are input into two identical branch networks. Each branch contains four parts: an encoder network, a multi-scale locally shared convolutional module, a global information extraction module, and a lightweight detection head. The encoder network quickly extracts basic features of different modalities, obtaining dense feature descriptors at minimal cost. Then, parameter-shared serial multi-scale dilated convolutions are used to obtain local features of different receptive field regions of the image. Next, a multi-head attention architecture from the Transformer is employed to interactively combine global contextual semantic information with local features, obtaining a feature descriptor for each pixel. Finally, average pooling and max operations are used to obtain the relative saliency of each point relative to local region points. Finally, softmax is used to obtain the keypoint detection score map.

[0058] A deep neural network model for keypoint detection and description of multimodal remote sensing images is constructed. The network consists of two branches with identical structures. Each branch comprises an encoder module, a multi-scale locally shared convolutional module, a global information extraction module, and a lightweight detection head module. The encoder module consists of six standard convolutional layers, taking an RGB image (SAR image) as input and outputting a feature map with invariant width and height channels upsized to 128 dimensions. The multi-scale locally shared convolutional module takes the output of the encoder module as input and consists of multiple dilated convolutions with different dilation rates (parameters are shared between the two branches). Its output shape is consistent with that of the encoder module.

[0059] The global information extraction module consists of Transformers, whose input is the output of the multi-scale locally shared convolutional module, and whose output is a feature descriptor for each pixel (with a shape consistent with the encoder module). The lightweight detection head module takes the output of the global information extraction module as input and outputs a keypoint score map (a probability map of each pixel being a keypoint).

[0060] (1) Encoder network:

[0061] The encoder network uses six standard convolutional layers (3 × 3 Convolution + BatchNorm + ReLU) stacked together to increase the channel dimension to 128 dimensions, thereby obtaining dense feature descriptors quickly at the lowest cost.

[0062] (2) Multi-scale locally shared convolution module:

[0063] Multi-scale local features of images are extracted using multiple dilated convolutions, with the same structure and shared parameters applied to two images of different modalities. Several dilated convolutions with different dilation rates are organized sequentially to learn local features of multimodal images. Figure 2 As shown in yellow, the output of the image depth features obtained from the encoder network after dilated convolution is represented by the following formula:

[0064] ;

[0065] In the formula, Represents the convolution kernel. This indicates the position on the output feature map. This indicates the position on the corresponding convolution kernel. This represents the dilation rate, and the receptive field size is adjusted by scaling the spacing between convolutional kernel elements. This degenerates into ordinary convolution. Dilated convolution can expand the receptive field without changing the image resolution; that is, the feature maps before and after dilated convolution are transformed. The size remains unchanged, where These represent the height and width of the feature map, respectively, with a depth of 128 dimensions. Three consecutive dilated convolutions are used in this modality to obtain multi-scale features, as shown in the following formula:

[0066] ;

[0067] In the formula, It is a 3×3 dilated convolution. This is a batch normalization layer. To preserve detailed information and provide more comprehensive local information, skip connections are used to process the original input features. Output features after dilated convolution To make the connection, use the following formula:

[0068] ;

[0069] This represents a 1×1 pointwise convolution operation, followed by a BN layer and a ReLU activation function. After skip connections, it goes through another 1×1 pointwise convolution to obtain the final output.

[0070] (3) Global Information Extraction Module:

[0071] Obtain the output of the multi-scale locally shared convolutional module As input to the global information extraction module, it is first stretched into a flattened 2D image block. Then, relative positional encoding is superimposed to focus on the positional relationships between elements. To better reflect the characteristics of the image, [the following is done]... The positions in each of the two dimensions are encoded into vectors of length 128. Finally, the two vectors are concatenated to obtain the final encoded vector. Specifically, when encoding each dimension, even-numbered positions are encoded using the sine function, and odd-numbered positions using the cosine function. For example... Figure 2 As shown in blue, the overall calculation code is illustrated in the following formula:

[0072] ;

[0073] ;

[0074] ;

[0075] ;

[0076] Will Each position in both directions is encoded into a 128-dimensional vector, which is then concatenated to obtain a vector of length 256. This is done in order to integrate with the feature map. The shape remains consistent, and its length is reduced to 128 by a 1×1 convolution.

[0077] Then, it passes through three linear mapping layers, with the weight matrix being... , , This represents the feature dimension of each pixel in different mapping layers; Mapped to queries with the same dimensions Keywords Sum : .

[0078] To reduce computational overhead, a self-attention function based on cross-covariance is used to calculate the attention coefficients along the channel dimension, as shown in the following formula:

[0079] .

[0080] To further enhance the representational power and nonlinearity of the features, a residual structure and a Linear layer are used to remap the features to obtain the final output, as shown in the following formula:

[0081] .

[0082] (4) Lightweight detection head:

[0083] To reduce the network load, a lightweight detection head was designed, capable of detecting keypoints based on both local and global information. To determine the relative prominence of a point compared to surrounding pixels, a saliency score and a maximum channel score were obtained through training. The saliency score is calculated using the following formula:

[0084] .

[0085] In the formula, This indicates that average pooling is performed within a 3×3 range. The purpose is to measure the relative significance of the current point and the points in the local region by calculating the difference between the current point's features and the average features of the local region.

[0086] The optimal channel is selected for each point by calculating the maximum score of the channel, as shown in the following formula:

[0087] ;

[0088] Finally, by multiplying the two scores and then applying a softmax operation, the final keypoint detection score map is obtained, as shown in the following formula:

[0089] .

[0090] Step 4: Train the deep neural network model from Step 3 using the training database processed in Step 2. Training was conducted on an Ubuntu 21.0 platform using the AdamW optimizer, with Python 3.7 and PyTorch 1.1.0 used to build the model. An NVIDIA RTX 3090 with 24 GB of video memory was used for training. The batch size was set to 16, and 100 iterations were performed to determine the optimal model parameters. The initial learning rate was 0.001, and an annealing strategy was introduced during training to adjust the learning rate.

[0091] Step 5: Input the image to be detected into the deep neural network trained in Step 3. The intermediate output of the deep neural network obtains the feature descriptor of each pixel. The output value of the last layer is used to calculate the key point score of each pixel through the softmax function, thereby realizing key point detection and description of multimodal images.

[0092] Key point detection and description test results are as follows Figure 3 As shown. This method utilizes deep learning to automatically learn the required features from the training dataset and perform keypoint detection and description. During training, it continuously optimizes the features and parameters used for discrimination. This method uses 998 pairs of multimodal remote sensing images for training, and the test set contains 248 pairs of multimodal images. After training, the deep neural network model described in this invention can detect an average of 30 keypoints per image. The multimodal remote sensing image keypoint detection and description algorithm of this patent exhibits excellent performance. Compared to single-modal algorithms, it can better perceive different modal features. Furthermore, compared to other multimodal algorithms, it also considers the combination of local and global features, resulting in superior performance.

[0093] The above embodiments are for illustrative purposes only and are not intended to limit the invention. Any modifications or variations made by those skilled in the art without departing from the spirit and scope of the invention are within the scope of the invention.

[0094] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope. The scope of protection of the present invention is defined by the appended claims, the specification, and their equivalents.

Claims

1. A method for detecting key points in multimodal remote sensing images based on a dual-branch network architecture, characterized in that, The method includes the following steps: Step 1: Obtain a multimodal remote sensing image dataset, which includes RGB images and SAR images; Step 2: Divide the multimodal remote sensing image dataset to form a training database; Step 3: Construct a deep neural network model for keypoint detection and description of multimodal remote sensing images. This model comprises two branches, each consisting of an encoder network, a multi-scale locally shared convolutional module, a global information extraction module, and a lightweight detection head. The inputs to the two branches are an RGB image and a SAR image, respectively. After registration, the global information extraction module outputs feature descriptors for each image, and the lightweight detection head outputs keypoint scores for each modality. Specifically, the encoder network rapidly extracts basic features of different modalities to quickly obtain dense feature descriptors at minimal cost. Then, parameter-shared serial multi-scale dilated convolutions are used to obtain local features of different receptive field regions of the image. Next, a multi-head attention architecture from the Transformer is used to interactively combine global contextual semantic information with local features to obtain the feature descriptor for each pixel. Finally, average pooling and maximum value operations are used to obtain the relative saliency of each point relative to local region points. Finally, softmax is used to obtain the keypoint detection score map. Step 4: Train the deep neural network model from Step 3 using the training database to obtain the trained deep neural network model for multimodal remote sensing image key point detection and description. Step 5: Input the multimodal image to be detected into the deep neural network model trained in Step 4. The global information extraction module outputs the feature descriptor of each pixel. The output value of the last layer of the lightweight detection head is used to calculate the key point score of each pixel through the softmax function, thereby realizing the key point detection and description of multimodal images.

2. The method for detecting key points in multimodal remote sensing images based on a dual-branch network architecture as described in claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Input the RGB image and SAR image into two encoder networks respectively. The encoder networks increase the channel dimension through six standard convolutional layers with non-shared parameters to obtain the first feature map f. Step 3.2, establish a multi-scale locally shared convolutional module: For the first feature map f obtained from the encoder network, the output after dilated convolution is represented by the following formula: DConv r (f*w) m,n =∑ i,j f i,j ·w i+m·r,j+n·r ; In the formula, w represents the convolution kernel, m and n represent the positions on the first feature map f, i and j represent the positions on the corresponding convolution kernel, and r represents the dilation rate. The size of the feature map remains unchanged before and after dilated convolution. Dconv r (·) represents a 3×3 dilated convolution; In each modality, three consecutive dilated convolutions are used to obtain multi-scale features, resulting in a second feature map. The following formula represents: In the formula, BN(·) is the batch normalization layer; Step 3.3, establishing a global information extraction module: This includes using the output of the obtained multi-scale local shared convolution module as input to the global information extraction module, stretching it to a flattened 2D image patch, and then focusing on the positional relationships between elements by superimposing relative positional encodings. The final output of the global information extraction module is... Step 3.4, Establish a lightweight detection head: Obtain the saliency score u and the maximum channel score z through training. The calculation formula for the saliency score is as follows: In the formula, AvgPool 3×3 (·) indicates that average pooling is performed in a 3×3 range; The optimal channel is selected for each point by calculating the maximum score of the channel, as shown in the following formula: Finally, by multiplying the saliency score by the maximum score of each channel and then applying a softmax operation, the keypoint detection score map is obtained, as shown in the following formula: sores = softmax(max(u·z)).

3. The method for detecting key points in multimodal remote sensing images based on a dual-branch network architecture as described in claim 2, characterized in that, Also includes: The first feature map f is connected to the output feature map after dilated convolution using a skip connection. By performing a connection, a third feature map is obtained. As shown in the formula below: Conv(·) represents a 1×1 pointwise convolution operation, followed by a BN layer and a ReLU activation function.

4. The method for detecting key points in multimodal remote sensing images based on a dual-branch network architecture as described in claim 3, characterized in that, The third feature map is encoded into a vector of length 128 at each position in the x and y dimensions. Finally, the modal vectors are concatted to obtain the final encoded vector. Specifically, When encoding each dimension, the even-numbered positions are encoded using the sine function, and the odd-numbered positions are encoded using the cosine function. The encoding is calculated using the following formula: Then concatenate them to obtain a vector of length 256, in order to match the third feature map. The shape remains consistent, and its length is reduced to 128 by a 1×1 convolution; Then, it passes through three linear mapping layers, with the weight matrix being... d q =d k =d v This represents the feature dimension of each pixel in different mapping layers; The third feature map Mapped to queries with the same dimensions Keywords Sum A self-attention function based on cross-covariance is used to calculate the attention coefficients along the channel dimension, as shown in the following formula: Attention(Q,K,V)=V·Softmax(Q T ·K); The features are remapped using a residual structure and a linear layer to obtain the final output, as shown in the following formula:

Images