Multi-modal remote sensing image key point detection method based on double-branch network architecture

Abstract

The invention discloses a multi-modal remote sensing image key point detection method based on a double-branch network architecture, and the method comprises the following steps: respectively inputting an RGB image and an SAR image into two branch networks with the same structure, each branch comprising an encoder, a multi-scale local sharing convolution module, a global information extraction module and a lightweight detection head; in each branch, the channel dimension of the image is raised through six standard convolutional layers with unshared parameters, and dense feature descriptors are quickly obtained at the minimum cost; acquiring local features of different receptive field regions of the image through parameter-shared serial multi-scale hole convolution; performing interactive combination on the global information and the local features by adopting a multi-head attention architecture in a Transform to obtain a feature descriptor of each pixel; then, average pooling and maximum value operation are adopted to obtain the relative significance of each point and the local area point, and finally, a key point detection score graph is obtained through softmax.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multimodal remote sensing image key point detection and description method based on a dual-branch network. Background Art

[0002] With the continuous advancement of sensor technology and the widespread application of multi-source data, multimodal remote sensing data plays an increasingly important role in geographic information extraction and analysis. In recent years, the rapid development of deep learning has provided strong technical support for feature extraction and information fusion of multimodal remote sensing images, especially in key point detection and description. In the field of remote sensing, key point detection refers to the identification of significant and stable pixels in an image, and key point description further extracts local feature information of these points for matching, recognition and analysis. This task is of great significance to the geometric correction, target recognition and change detection of remote sensing images.

[0003] Compared with single-modal images, multimodal remote sensing data (such as optical and SAR data) have significant advantages. Multimodal data can not only provide more comprehensive target characteristic information, but also supplement the missing information in single-modal data due to lighting conditions, weather changes or complexity of objects. For example, optical data is good at providing rich texture and color information, while SAR data is robust to weather and lighting conditions. The effective fusion of multimodal data can more comprehensively and accurately characterize the key features of ground objects, greatly improving the reliability and accuracy of key point detection and description.

[0004] To this end, the deep dual-branch network proposed in recent years has become an effective method for solving the key point detection and description of multimodal remote sensing images. By extracting the features of each modality data through shared and independent branches and making full use of the complementarity of different modalities in the feature fusion stage, this method can avoid information confusion and feature loss, and enhance the robustness and generalization ability of the model.

[0005] Therefore, how to utilize the complementarity and spatial information of multimodal remote sensing images, organically combine the characteristics of different modalities, and construct a high-precision and robust key point detection and description model is an important issue that needs to be urgently solved in the current field of remote sensing image analysis. Summary of the invention

[0006] In order to solve the problems in the above-mentioned background technology, the present invention aims to provide a multimodal remote sensing image key point detection and description method with a dual-branch network architecture, which can make full use of the multimodal remote sensing image training model of the training data set, so that the model can automatically detect the location of the key points and their feature descriptors.

[0007] In order to solve the above technical problems, the present invention provides a multimodal remote sensing image key point detection method based on a dual-branch network architecture, the method comprising the following steps: Step 1: Acquire a multimodal remote sensing image dataset, where the multimodal remote sensing image dataset includes RGB images and SAR images; Step 2: Divide the multimodal remote sensing image data set to form a training database; Step 3: Construct a deep neural network model for key point detection and description of multimodal remote sensing images. The deep neural network model contains two branches, each of which is composed of four parts: an encoder network, a multi-scale local shared convolution module, a global information extraction module, and a lightweight detection head. The input of the model is two registered RGB images and a SAR image, respectively. The output of the global information extraction module is the feature descriptor of each of the two images, and the output of the lightweight detection head is the key point score of each of the two modal images. Step 4: using the training database to train the deep neural network model of step 3, to obtain a trained deep neural network model for key point detection and description of multimodal remote sensing images; Step 5: Input the multimodal image to be detected into the deep neural network model trained in step 4. The global information extraction module outputs the feature descriptor of each pixel. The output value of the last layer of the lightweight detection head is calculated by the softmax function to obtain the key point score of each pixel, thereby realizing the key point detection and description of the multimodal image.

[0008] Furthermore, the step 3 is specifically as follows: Step 3.1, input the RGB image and the SAR image into the encoder network, and use a lightweight encoder network to increase the channel dimension to 128 dimensions through 6 standard convolutional layers with unshared parameters to quickly obtain the first feature map f; Step 3.2, establish a multi-scale local shared convolution module: For the first feature map f obtained from the encoder network, the output after the hole convolution is expressed by the following formula: ; In the formula, represents the convolution kernel, represents the position on the first feature map f, Indicates the position on the corresponding convolution kernel, Represents the dilation rate. The size of the feature map remains unchanged before and after the dilation convolution. In each mode, three consecutive dilated convolutions are used to obtain multi-scale features and obtain the second feature map , as shown in the following formula: ; In the formula, is a 3×3 dilated convolution, is the batch normalization layer; Step 3.3, establish a global information extraction module: use the output of the multi-scale local shared convolution module as the input of the global information extraction module, stretch it to a flattened 2D image block, and then focus on the positional relationship between elements by superimposing relative position encoding; Step 3.4, establish a lightweight detection head: obtain the significance score and the maximum channel score through training. The significance score is calculated as shown in the following formula: ; In the formula, Indicates that average pooling operation is performed in a 3×3 range; The optimal channel is selected for each point by calculating the maximum channel score, as shown in the following formula: ; Finally, the key point detection score map is obtained by multiplying the significance score by the maximum score of the channel and then performing a softmax operation, as shown in the following formula: .

[0009] Furthermore, the first feature map is connected using a skip connection. And the output features after the dilated convolution Connect and get the third feature map , as shown in the following formula: ; Represents a 1×1 point-wise convolution operation, followed by a BN layer and a ReLu activation function.

[0010] Further, the third feature map is Each position in the two dimensions is encoded as a vector of length 128, and finally the modal vectors are concat-operated to obtain the final encoding vector. Specifically, When encoding each dimension, the sin function is used for even positions and the cos function is used for odd positions. The encoding is calculated as shown in the following formula: ; ; ; ; Then concatenate to get a vector of length 256, in order to match the third feature map Keep the shape of , and reduce its length to 128 through a 1×1 convolution; After three linear mapping layers, the weight matrix is , , , Represents the feature dimension of each pixel in different mapping layers; The third feature map Mapping to queries with the same dimensions , Keywords Sum : ; The self-attention function based on mutual covariance is used to calculate the attention coefficient along the channel dimension, as shown in the following formula: ; The residual structure and the Linear layer are used to remap the features to obtain the final output, as shown in the following formula: .

[0011] The beneficial effects of the present invention are as follows: the present invention can use images with a smaller amount of data to train the model, and after the multimodal remote sensing image data is aligned, deep learning is used to automatically learn the required features of different modalities from the training database and perform key point detection and description. During the training process, the data features used for judgment are continuously corrected and the hyperparameters in the deep neural network are adjusted, thereby realizing key point detection and description of multimodal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a structural diagram of a multimodal remote sensing image key point detection and description network model of the present invention; Figure 2 This is a network structure diagram of the multi-scale local shared convolution module and the global information extraction module of the present invention; Figure 3 This is a diagram of the key point detection and description experimental results of the present invention. DETAILED DESCRIPTION

[0013] The technical solution of the present invention is now fully described in conjunction with the accompanying drawings. The following description is only a part of the implementation cases of the present invention, not all. Based on the implementation cases in the present invention, all other implementation cases obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0014] Embodiment 1:

[0015] The present application is a multimodal remote sensing image key point detection method based on a dual-branch network architecture, which includes the following steps: Step 1: Collect RGB and SAR remote sensing images and align them; The collected 3-band RGB images were obtained from the Google Earth platform with a pixel resolution of 0.5-2 meters, and the SAR images were from the GF-3 satellite with a pixel resolution of 1 meter. It contains 10 4468*2844 pixel RGB images and SAR images of the same area, covering fields and towns in many cities around the world. The reference image for registration is the RGB image, which is registered using professional registration software.

[0016] Step 2: Preprocess the dataset, cut the image into 512×512 pixel image patch pairs by center cropping, and divide the dataset into a training database, with a training set: validation set ratio of 8:2. Step 3: Construct a neural network model for key point detection and description of multimodal remote sensing images.

[0017] like Figure 1 As shown in the figure, a multimodal remote sensing image key point detection and description model based on a dual-branch network architecture: The model uses RGB and SAR images for remote sensing image key point detection and description. The 3-channel RGB image and SAR image are respectively input into two branch networks with the same structure. Each branch contains four parts: encoder network, multi-scale local shared convolution module, global information extraction module and lightweight detection head; the encoder network quickly extracts the basic features of different modalities and quickly obtains dense feature descriptors at the minimum cost; then the serial multi-scale hole convolution with parameter sharing is used to obtain the local features of different receptive field areas of the image; the multi-head attention architecture in Transformer is then used to interactively combine the global contextual semantic information with the local features to obtain the feature descriptor of each pixel; then the average pooling and maximum value operations are used to obtain the relative significance of each point and the local area points, and finally the key point detection score map is obtained through softmax.

[0018] A deep neural network model for key point detection and description of multimodal remote sensing images is constructed. The network contains two branches, and the two branches maintain the same structure. Each branch consists of four parts: encoder module, multi-scale local shared convolution module, global information extraction module and lightweight detection head module. The encoder module consists of 6 standard convolution layers, with RGB image (SAR image) as input and 128-dimensional feature map with width and height unchanged channels as output. The input of the multi-scale local shared convolution module is the output of the encoder module, which consists of multiple dilated convolution sequences with different dilation rates (parameters are shared in the two branches), and its output shape is consistent with the output shape of the encoder module.

[0019] The global information extraction module is composed of Transformer, whose input is the output of the multi-scale local shared convolution module, and its output is the feature descriptor of each pixel (the shape is consistent with the encoder module). The input of the lightweight detection head module is the output of the global information extraction module, and the output is the key point score map (the probability map of each pixel becoming a key point).

[0020] (1) Encoder network: The encoder network uses 6 standard convolutional layers (3 × 3 Convolution + BatchNorm + ReLu) with unshared parameters to stack and use once, increasing the channel dimension to 128 dimensions, and quickly obtaining dense feature descriptors at minimal cost; (2) Multi-scale local shared convolution module: Multiple dilated convolutions are used to extract multi-scale local features of images. The same structure and shared parameters are adopted for two different modal images. Several dilated convolutions with different dilation rates are organized in a serial manner to achieve the learning of local features of multi-modal images. Figure 2 As shown in the yellow part, for the image depth features obtained from the encoder network, the output after the dilated convolution is expressed as follows: ; In the formula, represents the convolution kernel, represents the position on the output feature map, Indicates the position on the corresponding convolution kernel, Represents the void rate, and adjusts the receptive field size by scaling the spacing between convolution kernel elements. It degenerates into a normal convolution. The dilated convolution can expand the receptive field without changing the image resolution. That is, the feature map before and after the dilated convolution The size remains unchanged, Represent the height and width of the feature map respectively, and the depth is 128 dimensions. In this mode, three consecutive dilated convolutions are used to obtain multi-scale features, as shown in the following formula: ; In the formula, is a 3×3 dilated convolution, is a batch normalization layer. In order to retain detailed information and provide more comprehensive local information, a skip connection is used to convert the original input features And the output features after the dilated convolution Make the connection as shown in the following formula: ; Represents a 1×1 point-by-point convolution operation, followed by a BN layer and a ReLu activation function. After a skip connection, a 1×1 point-by-point convolution is performed to obtain the final output result.

[0021] (3) Global information extraction module: Get the output of the multi-scale local shared convolution module As the input of the global information extraction module, it is first stretched into a flat 2D image block , and then focus on the positional relationship between elements by superimposing relative position codes. In order to better match the characteristics of the image, The positions in the two dimensions are respectively encoded into vectors of length 128, and finally the two vectors are concat-operated to obtain the final encoded vector. Specifically, when encoding each dimension, the sin function is used for even positions and the cos function is used for odd positions. Figure 2 As shown in the blue part, the overall calculation code is as follows: ; ; ; ; Will Each position in the two directions is encoded into a 128-dimensional vector and then concatenated to obtain a vector of length 256. The shape of is kept consistent, and its length is reduced to 128 by a 1×1 convolution.

[0022] After three linear mapping layers, the weight matrix is , , Represents the feature dimension of each pixel in different mapping layers; Mapping to queries with the same dimensions Keywords Sum : .

[0023] In order to reduce computational overhead, a self-attention function based on mutual covariance is used to calculate the attention coefficient along the channel dimension, as shown in the following formula: .

[0024] Then, in order to improve the representation ability and nonlinearity of the features, the residual structure and the Linear layer are used to re-map the features to obtain the final output, as shown in the following formula: .

[0025] (4) Lightweight detection head: In order to reduce the burden on the network, a lightweight detection head is designed, which can detect key points based on local and global information. In order to judge the relative prominence of the point compared with other surrounding pixels, the saliency score and the maximum channel score are obtained through training. The saliency score is calculated as shown in the following formula: .

[0026] In the formula, Indicates that an average pooling operation is performed in a 3×3 range. The purpose is to measure the relative significance of the point and the points in the local area by calculating the difference between the current point feature and the average feature of the local area.

[0027] The optimal channel is selected for each point by calculating the maximum channel score, as shown in the following formula: ; Finally, the final key point detection score map is obtained by multiplying the two scores and then performing a softmax operation, as shown in the following formula: .

[0028] Step 4: Use the training database processed in step 2 to train the deep neural network model in step 3. AdamW optimizer was used on Ubuntu 21.0 platform. The model was built using Python 3.7 and PyTorch 1.1.0. NVIDIA RTX 3090 with 24 GB of video memory was used to train the model. The batch size was set to 16 and 100 iterations were performed to determine the best model parameter selection. The initial learning rate was 0.001, and an annealing strategy was introduced during training to adjust the learning rate. Step 5: Input the image to be detected into the deep neural network trained in step 3. The intermediate output of the deep neural network obtains the feature descriptor of each pixel. The output value of the last layer is calculated by the softmax function to obtain the key point score of each pixel, thereby realizing the key point detection and description of multimodal images.

[0029] Key point detection and description test results are as follows Figure 3As shown. This method uses deep learning to automatically learn the required features from the training data set and perform key point detection and description. The features and parameters used for discrimination are continuously optimized during the training process. This method selected and used 998 pairs of multimodal remote sensing images for training, and the test set contained 248 pairs of multimodal images. After training, the deep neural network model described in the present invention can detect 30 key points per image on average. The multimodal remote sensing image key point detection and description algorithm of this patent has relatively excellent performance. Compared with single-modal algorithms, it can better perceive different modal features. At the same time, compared with other multimodal algorithms, it also takes into account the combination of local features and global features, and performs better.

[0030] The above embodiments are only used to illustrate the present invention, rather than to limit the present invention. Various changes or modifications made by technicians in the relevant technical field without departing from the spirit and scope of the present invention all belong to the scope of the present invention.

[0031] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention, and the scope of protection claimed in the present invention is defined by the attached claims, description and their equivalents.

Claims

1. A multimodal remote sensing image key point detection method based on a dual-branch network architecture, characterized in that: The method comprises the following steps: Step 1: Acquire a multimodal remote sensing image dataset, where the multimodal remote sensing image dataset includes RGB images and SAR images; Step 2: Divide the multimodal remote sensing image data set to form a training database; Step 3: Construct a deep neural network model for key point detection and description of multimodal remote sensing images. The deep neural network model contains two branches, each of which is composed of four parts: an encoder network, a multi-scale local shared convolution module, a global information extraction module, and a lightweight detection head. The input of the model is two registered RGB images and a SAR image, respectively. The output of the global information extraction module is the feature descriptor of each of the two images, and the output of the lightweight detection head is the key point score of each of the two modal images. Step 4: using the training database to train the deep neural network model of step 3, to obtain a trained deep neural network model for key point detection and description of multimodal remote sensing images; Step 5: Input the multimodal image to be detected into the deep neural network model trained in step 4. The global information extraction module outputs the feature descriptor of each pixel. The output value of the last layer of the lightweight detection head is calculated by the softmax function to obtain the key point score of each pixel, thereby realizing the key point detection and description of the multimodal image.

2. The multimodal remote sensing image key point detection method based on a dual-branch network architecture as claimed in claim 1, characterized in that: The step 3 is specifically as follows: Step 3.1, input the RGB image and the SAR image into the encoder network, and use a lightweight encoder network to increase the channel dimension through 6 standard convolutional layers with unshared parameters to obtain the first feature map f; Step 3.2, establish a multi-scale local shared convolution module: For the first feature map f obtained from the encoder network, the output after the hole convolution is expressed by the following formula: ； In the formula, represents the convolution kernel, represents the position on the first feature map f, Indicates the position on the corresponding convolution kernel, Represents the dilation rate. The size of the feature map remains unchanged before and after the dilation convolution. In each mode, three consecutive dilated convolutions are used to obtain multi-scale features and obtain the second feature map , as shown in the following formula: ； In the formula, is a 3×3 dilated convolution, is the batch normalization layer; Step 3.3, establish a global information extraction module: use the output of the multi-scale local shared convolution module as the input of the global information extraction module, stretch it to a flattened 2D image block, and then focus on the positional relationship between elements by superimposing relative position encoding; Step 3.4, establish a lightweight detection head: obtain the significance score and the maximum channel score through training. The significance score is calculated as shown in the following formula: ； In the formula, Indicates average pooling operation in a 3×3 range; The optimal channel is selected for each point by calculating the maximum channel score, as shown in the following formula: ； Finally, the key point detection score map is obtained by multiplying the significance score by the maximum score of the channel and then performing a softmax operation, as shown in the following formula: 。 3. The multimodal remote sensing image key point detection method based on a dual-branch network architecture as claimed in claim 1, characterized in that: Use skip connection to transform the first feature map And the output features after the dilated convolution Connect and get the third feature map , as shown in the following formula: ； Represents a 1×1 point-wise convolution operation, followed by a BN layer and a ReLu activation function.

4. The multimodal remote sensing image key point detection method based on a dual-branch network architecture as claimed in claim 3, characterized in that: The third characteristic map is Each position in the two dimensions is encoded as a vector of length 128, and finally the modal vectors are concat-operated to obtain the final encoding vector. Specifically, When encoding each dimension, the sin function is used for even positions and the cos function is used for odd positions. The encoding is calculated as shown in the following formula: ； ； ； ； Then concatenate to get a vector of length 256, in order to match the third feature map Keep the shape of , and reduce its length to 128 through a 1×1 convolution; After three linear mapping layers, the weight matrix is , , , Represents the feature dimension of each pixel in different mapping layers; The third feature map Mapping to queries with the same dimensions , keywords Sum : ; The self-attention function based on mutual covariance is used to calculate the attention coefficient along the channel dimension, as shown in the following formula: ； The residual structure and the Linear layer are used to remap the features to obtain the final output, as shown in the following formula: 。

Images