A SAR-visible light remote sensing image matching method

Abstract

The present invention provides a SAR-visible light remote sensing image matching method, which belongs to the field of remote sensing image processing. The present invention uses geographic coding to perform coarse registration of SAR and visible light images; extracts multi-scale fusion structure directional maps of SAR and visible light remote sensing images respectively; inputs the structure directional maps of SAR and visible light into a pseudo-twin network, optimizes the image structure directional map using an attention mechanism; obtains the correlation between the two structure directional maps and determines the best matching position. The present invention can be used for matching visible light remote sensing images and synthetic aperture radar images, and realizes high-precision and efficient SAR-visible light remote sensing image matching.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a SAR-visible light remote sensing image matching method, which can be used for matching visible light remote sensing images and synthetic aperture radar (SAR) images. Background Art

[0002] Multimodal image data can reflect the different attributes and characteristics of the ground objects, make up for the shortcomings of single-modal images, and increase the amount of information in the image. SAR-visible light remote sensing images are typical multimodal images. SAR-visible light image matching refers to the process of detecting control points between images. This is a key preprocessing step for the comprehensive use of SAR and optical images for remote sensing applications and analysis, such as heterogeneous image registration, image fusion, and change detection. Due to the different imaging mechanisms of SAR images and visible light images, there are also large differences in the image information performance of the two. Visible light images are affected by clouds and light; SAR images are coherent images formed by side-looking radar through slant range projection, which are prone to noise and geometric distortion during the imaging process; therefore, to achieve high-quality SAR-visible light remote sensing image matching, it is necessary to enable the matching algorithm to adapt to the nonlinear grayscale distortion and SAR image noise between heterogeneous images.

[0003] At present, the methods of SAR-visible light remote sensing image matching are mainly divided into three categories. The first category is the matching method based on image grayscale information, which realizes heterogeneous image matching through the grayscale distribution or grayscale mapping relationship between SAR images and visible light images. The representative algorithms of this type of method are Normalized Cross-Correlation (NCC), MTM and Mutual Information (MI). NCC is suitable for linear grayscale mapping relationship and cannot meet the nonlinear grayscale distortion between SAR-visible light remote sensing images; MTM can adapt to the nonlinear mapping of grayscale. Its working premise is that the grayscale mapping between the two images satisfies the functional relationship, while the grayscale values ​​between SAR-visible light remote sensing images usually do not meet the functional relationship. MI can adapt to the nonlinear distortion between heterogeneous images, but its computational efficiency is low and it does not utilize the structural information of the image, so it is difficult to adapt to the nonlinear radiation distortion between heterogeneous images. The second category is the matching method based on image structural features, which realizes heterogeneous image matching by extracting common features between SAR-visible light remote sensing images. Representative algorithms of this type of method include Projecting and Quantizing Histograms of Oriented Gradients (PQHOG), Histogram of Oriented Phase Congruency (HOPC), and Channel Features of Orientated Gradients (CFOG). These methods are more stable than grayscale information-based matching methods, but image structure information requires manual design, making it difficult to further improve matching performance. In addition, when the image structure signs are not obvious, these methods are prone to failure. The third type is a matching method based on a neural convolutional network, which achieves heterogeneous image matching by extracting the common features of SAR and visible light in nonlinear radiation distortion scenarios. This type of method directly inputs SAR and visible light remote sensing images into the neural network, making it difficult to effectively capture the common features of SAR and visible light remote sensing images. In addition, the deep neural network has too many parameters and is easily affected by useless information. Summary of the invention

[0004] In view of this, the present invention provides a SAR-visible light remote sensing image matching method. The method is based on the attention mechanism and the image structure direction, which can solve the problems of nonlinear grayscale distortion and image noise in the SAR-visible light remote sensing images in the prior art, and can enhance useful features while reducing the impact of useless features on training.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A SAR-visible light remote sensing image matching method comprises the following steps:

[0007] Step 1: Use geocoding to roughly match the SAR remote sensing image and the visible light remote sensing image, and perform subsequent steps for the SAR remote sensing image and the visible light remote sensing image with the same area;

[0008] Step 2: Extract multi-scale fusion structure direction maps of SAR remote sensing images and visible light remote sensing images respectively;

[0009] Step 3: Input the multi-scale fusion structure direction map of the SAR remote sensing image into the trained first pseudo-twin neural network to obtain the first structure direction feature; input the multi-scale fusion structure direction map of the visible light remote sensing image into the trained second pseudo-twin neural network to obtain the second structure direction feature; the structures of the first pseudo-twin neural network and the second pseudo-twin neural network are exactly the same;

[0010] Step 4: Use the cosine similarity measurement based on FFT acceleration as the loss function of the pseudo-twin neural network to calculate the similarity heat map of the first structural directional feature and the second structural directional feature. The image position corresponding to the pixel with the highest thermal value is the best matching position of the SAR remote sensing image and the visible light image block.

[0011] Furthermore, the specific method of step 2 is:

[0012] Step 2.1: Calculate the basic gradient of the SAR / visible light remote sensing image. The basic gradient uses the Sobel gradient extraction operator to extract the X direction and Y direction respectively;

[0013] Step 2.2: Using the basic gradient information obtained in step 2.1, calculate the gradient square and gradient cosquare of SAR / visible light remote sensing images of four different domain scales;

[0014] Step 2.3: According to the calculation formula of image consistency, calculate the fusion weight coefficients of the four neighborhood scales of the SAR / visible light remote sensing image;

[0015] Step 2.4: Fuse the gradient squares and co-squares of the four domain scales of the SAR / visible light remote sensing image according to the fusion weight coefficient;

[0016] Step 2.5: Use the fused gradient square sum cosquare to calculate the multi-scale fusion structure direction map of the SAR / visible light remote sensing image.

[0017] Furthermore, the pseudo twin neural network includes a convolutional layer, a feature fusion module, a channel attention mechanism module, a spatial attention mechanism module and a loss function; wherein the channel attention module is used to implement the stacked channel attention mechanism, and the spatial attention mechanism module is used to implement the spatial attention mechanism of the matching process;

[0018] The convolution layer is used to extract the structural directional features in the structural directional map. It uses the BR neural network. Each convolution layer in the network is followed by a BN layer and a ReLU activation function.

[0019] The feature fusion module is used to fuse multi-scale structural directional features. The original structural directional map is passed through the convolution layer to obtain the structural directional features. The downsampled structural directional map is passed through the convolution layer to obtain the downsampled structural directional features. The two structural directional features are fused. The fusion method adopts a parallel strategy connection or directly connects multiple features.

[0020] The channel attention mechanism module is used to balance the speed and accuracy of image feature extraction. It is plug-and-play. After the convolution layer uses the convolution kernel for convolution, the sigmoid function is used to learn the channel information.

[0021] The spatial attention mechanism module focuses on the information of the position features of the target image in each channel and is divided into a context module and a channel conversion module.

[0022] Furthermore, the loss function is:

[0023]

[0024] Among them, N p and N h They represent the number of positive samples and negative samples respectively, i represents the i-th sample, and f logi (S) = log(1 + exp(-S)), S = F -1 [F(O Opt )·F * (F(O Sar ))], F represents the forward Fourier transform, F -1 represents the inverse Fourier transform, F * represents the complex conjugate matrix of F, O Opt is the directional characteristic of the visible light structure, O Sar It is the directional characteristic of SAR structure;

[0025] According to the above loss function, the similarity heat map of the first structural directional feature and the second structural directional feature is obtained, and the image position corresponding to the point with the largest thermal value is found, which is the best matching position of the final visible light image and the SAR image.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. In view of the defects of the prior art, the present invention uses a multi-scale fusion method to extract the structural direction of the image. The multi-scale fused image structural direction is more resistant to noise than the single-scale image structural direction, and can adapt to the speckle noise in the SAR remote sensing image. The multi-scale fused image structural direction of the SAR and visible light remote sensing images is used as the input of the pseudo-twin neural network. Compared with the gray value of the image, the structural direction information of the image can more fully represent the common characteristics between the SAR and visible light remote sensing images. The pseudo-twin neural network is used to optimize the image structural direction characteristics and improve the matching performance.

[0028] 2. The present invention adopts a shallow pseudo-twin neural network based on the attention mechanism to train SAR and visible light remote sensing images, forming a stacked channel attention mechanism that can fuse features, strengthening the extraction of useful channel information while reducing the image of useless channel information, thereby improving the generalization ability of feature extraction. Compared with image grayscale information, shallow neural networks are easier to extract features in structural diagrams.

[0029] 3. After the neural network extracts the image structure directional features, the cosine measurement criterion accelerated by FFT is used to calculate the correlation of the extracted image structure direction. The cosine similarity measurement criterion can adapt to the gradient flipping caused by nonlinear grayscale distortion between heterogeneous images, and finally obtain the best matching position of SAR-visible light remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a pair of SAR-visible light remote sensing images used in the embodiment of the present invention, where the left image is a visible light remote sensing image and the right image is a SAR remote sensing image.

[0031] Figure 2 It is a flow chart of a SAR-visible light remote sensing image matching method based on an attention mechanism and image structure direction provided in an embodiment of the present invention.

[0032] Figure 3 It is a flow chart of the multi-scale fusion image structure direction extraction method provided by an embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of a pseudo-twin neural network provided by an embodiment of the present invention.

[0034] Figure 5 It is the network output SAR-visible light matching map. DETAILED DESCRIPTION

[0035] The following will be combined with the embodiments to clearly and completely describe the concept, technical solution advantages and technical effects of the present invention, so as to fully understand the purpose, features and effects of the present invention. It should be noted that the specific embodiments described herein are only used to explain the present invention and do not limit the present invention.

[0036] A SAR-visible light remote sensing image matching method, see Figure 2 , this method first uses prior information (geocoding) to SAR and visible light images (see Figure 1 ) for coarse registration; then, the multi-scale fusion image structure direction maps of the SAR remote sensing image and the visible light remote sensing image are extracted respectively, and different scales are fused according to the consistency size; secondly, the extracted multi-scale fusion structure direction map is input into the pseudo-twin neural network. The training process of the SAR image structure direction map and the visible light image structure direction map does not share parameters. The attention mechanism is embedded in the neural network, and the structure direction features are extracted through the structure direction feature extraction network; finally, the cosine similarity measurement based on FFT acceleration is used to calculate the similarity of the two images to obtain the best matching position between the SAR and visible light remote sensing images.

[0037] The specific process is as follows:

[0038] S1: Use geographic coding to roughly align SAR and visible light remote sensing images. The imaging range of remote sensing images is large, and image information is acquired with prior geographic information. Use geographic coding to correct the two images and perform rough alignment as a preprocessing for matching.

[0039] S2: See Figure 3 , extract the multi-scale fusion image structure direction of SAR and visible light remote sensing images:

[0040] First, assuming that the original image is I(x, y), use the Sobel operator to extract the basic gradient vector G of the image x ,G y ] T :G x =f x *I(x,y),G y =f y *I(x,y), where f x and f y It is a filter in two directions of the Sobel operator, and then calculates the gradient square and gradient cross product of different scales: Weighting coefficients w of different scales i Calculated by gradient consistency: The structural direction of the final image

[0041] S3: Using pseudo-twin neural network to optimize image structure direction information:

[0042] See also Figure 4 The present invention adopts a two-branch neural network that does not share weights, namely a pseudo-twin neural network. The pseudo-twin neural network includes a convolutional layer, a feature fusion module, a channel attention mechanism module, a spatial attention mechanism module, and a loss function. Among them, the channel attention module is used to implement the stacked channel attention mechanism, and the spatial attention mechanism module is used to implement the spatial attention mechanism of the matching process.

[0043] The convolution layer is used to extract the structural directional features in the structural directional map. It uses the BR network. Each convolution layer in the convolution network has a BM layer and a ReLU activation function after each convolution.

[0044] The feature fusion module is a module for fusing multi-scale structural directional features. The original structural directional map is passed through the convolution layer to obtain the structural directional features, and the downsampled structural directional map is passed through the convolution layer to obtain the downsampled structural directional features. The two structural directional features are fused, and the fusion method is not limited to the concat (directly connecting multiple features) and add (parallel strategy connection) methods.

[0045] The channel attention mechanism module can balance the speed and accuracy of image feature extraction, has the characteristics of plug-and-play, and can be efficiently embedded in the feature extraction network. After the convolution layer uses the convolution kernel for convolution, the sigmoid function is then used to learn the channel information.

[0046] The spatial attention mechanism module is used to focus on the information of the position features of the target image in each channel, which is divided into a context module and a channel conversion module.

[0047] The pseudo twin neural networks involved in this method include a CNN convolutional neural network N1 connected to the structural direction of the SAR remote sensing image, and a CNN convolutional neural network N2 connected to the structural direction feature image of the visible light remote sensing image. The outputs of N1 and N2 are optimized image structural direction features, which are marked as D SAR and D OPT .

[0048] S4: Obtain the matching results of SAR and visible light remote sensing images:

[0049] Get the optimized SAR image structure directional feature D SAR and visible light image structural directional features D OPT After that, calculate the similarity matrix between feature maps, see Figure 3 .

[0050] This method adopts the attention mechanism of stacked channels, integrates global average pooling and global maximum pooling, and develops the calculation method of the cosine absolute value similarity measurement criterion of direction difference based on cosine similarity. In order to improve the calculation speed of the similarity measurement criterion, the similarity measurement criterion accelerated by FFT is used: S = F -1 [F(D SAR )·F*(F(D OPT ))]. The loss function is:

[0051]

[0052] Among them, N p and N h They represent the number of positive samples and negative samples respectively, i represents the i-th sample, and f logi (S) = log(1 + exp(-S)), F represents the forward Fourier transform, F -1 represents the inverse Fourier transform, F * represents the complex conjugate matrix of F, O Opt is the directional characteristic of the visible light structure, O Sar It is the directional characteristic of SAR structure;

[0053] According to the above loss function, we obtain the similarity heat map of the first structural directional feature and the second structural directional feature, and find the image position corresponding to the point with the largest heat value, which is the best matching position of the final visible light image and SAR image. Figure 5 .

Claims

1. A SAR-visible light remote sensing image matching method, characterized in that: The steps include: Step 1: Use geocoding to roughly match the SAR remote sensing image and the visible light remote sensing image, and perform subsequent steps for the SAR remote sensing image and the visible light remote sensing image with the same area; Step 2: Extract multi-scale fusion structure direction maps of SAR remote sensing images and visible light remote sensing images respectively; Step 3: Input the multi-scale fusion structure direction map of the SAR remote sensing image into the trained first pseudo-twin neural network to obtain the first structure direction feature; The multi-scale fusion structure direction map of the visible light remote sensing image is input into the trained second pseudo-twin neural network to obtain a second structure direction feature; the structure of the first pseudo-twin neural network and the second pseudo-twin neural network is exactly the same, and the pseudo-twin neural network includes a convolutional layer, a feature fusion module, a channel attention mechanism module, a spatial attention mechanism module and a loss function; wherein the channel attention module is used to implement the stacked channel attention mechanism, and the spatial attention mechanism module is used to implement the spatial attention mechanism of the matching process; The convolution layer is used to extract the structural directional features in the structural directional map. It uses the BR neural network. Each convolution layer in the network is followed by a BN layer and a ReLU activation function. The feature fusion module is used to fuse multi-scale structural directional features. The original structural directional map is passed through the convolution layer to obtain the structural directional features. The downsampled structural directional map is passed through the convolution layer to obtain the downsampled structural directional features. The two structural directional features are fused. The fusion method adopts a parallel strategy connection or directly connects multiple features. The channel attention mechanism module is used to balance the speed and accuracy of image feature extraction. It is plug-and-play. After the convolution layer uses the convolution kernel for convolution, the sigmoid function is used to learn the channel information. The spatial attention mechanism module focuses on the information of the position features of the target image in each channel, which is divided into a context module and a channel conversion module; Step 4: Use the cosine similarity measurement based on FFT acceleration as the loss function of the pseudo-twin neural network to calculate the similarity heat map of the first structural directional feature and the second structural directional feature. The image position corresponding to the pixel with the highest thermal value is the best matching position of the SAR remote sensing image and the visible light image block.

2. A SAR-visible light remote sensing image matching method as claimed in claim 1, characterized in that: The specific method of step 2 is: Step 2.1: Calculate the basic gradient of the SAR / visible light remote sensing image. The basic gradient uses the Sobel gradient extraction operator to extract the X direction and Y direction respectively; Step 2.2: Using the basic gradient information obtained in step 2.1, calculate the gradient square and gradient cosquare of SAR / visible light remote sensing images of four different domain scales; Step 2.3: According to the calculation formula of image consistency, calculate the fusion weight coefficients of the four neighborhood scales of the SAR / visible light remote sensing image; Step 2.4: Fuse the gradient squares and co-squares of the four domain scales of the SAR / visible light remote sensing image according to the fusion weight coefficient; Step 2.5: Use the fused gradient square sum cosquare to calculate the multi-scale fusion structure direction map of the SAR / visible light remote sensing image.

3. A SAR-visible light remote sensing image matching method as claimed in claim 1, characterized in that: The loss function is: Among them, N p and N h They represent the number of positive samples and negative samples respectively, i represents the i-th sample, and f logi (S) = log(1 + exp(-S)), S = F -1 [F(O Opt )·F * (F(O Sar ))], F represents the forward Fourier transform, F -1 represents the inverse Fourier transform, F * represents the complex conjugate matrix of F, O Opt is the directional characteristic of the visible light structure, O Sar It is the directional characteristic of SAR structure; According to the above loss function, the similarity heat map of the first structural directional feature and the second structural directional feature is obtained, and the image position corresponding to the point with the largest thermal value is found, which is the best matching position of the final visible light image and the SAR image.

Images