A method for fusion of synthetic aperture radar image and multispectral image
By fusing synthetic aperture radar images with multispectral images through convolutional neural networks and attention mechanisms, the problems of spectral distortion and spatial distortion in existing technologies are solved, high-quality image fusion is achieved, and image interpretation capabilities are improved.
Patent Information
- Application Number
- CN202310656501.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2043-06-05
AI Technical Summary
Existing technologies make it difficult to effectively fuse synthetic aperture radar images with multispectral images, resulting in spectral distortion or spatial distortion in the fused images, and deep learning methods fail to fully utilize the connections between image feature channels.
A method based on convolutional neural networks and attention mechanism is adopted to extract image information through spatial feature extraction network and spectral preservation network, and the attention mechanism is used for feature fusion, combined with unsupervised joint loss function to optimize the training model.
The fused image has rich spatial and spectral information, which improves the image interpretation capability and the results of tasks such as urban land cover and land feature classification.
Smart Images
Figure CN116863283B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image fusion method, in particular to a method for fusing synthetic aperture radar images and multispectral images. Background Art
[0002] With the rapid development of remote sensing technology, imaging sensor types are becoming increasingly diverse. Remote sensing systems often cannot simultaneously achieve high spectral and spatial resolution. Synthetic aperture radar, as an active microwave sensor, offers all-day, all-weather coverage, wide swath coverage, and long range. Its imaging does not rely on sunlight, so SAR images do not contain spectral information. Multispectral sensors are passive and rely primarily on the reflectivity of objects to directly reflect their color and texture. However, their resolution is relatively low. Therefore, fusing multispectral and SAR images produces high-quality fused images with richer object feature information and spectral structure. This significantly enhances the structural information of the original images and can reveal hidden information within SAR images, improving image interpretation capabilities. This facilitates subsequent tasks such as urban land cover, object classification, and road detection.
[0003] Currently, image fusion algorithms can be categorized into four main categories: principal component replacement, multiscale analysis, hybrid methods, and deep learning-based approaches. Of these four methods, principal component replacement offers lower computational complexity, but these methods are highly dependent on image correlation. Due to the significant differences between SAR and MS images, the fused image can suffer from severe spectral distortion. Multiscale analysis can achieve better performance by integrating images in different frequency spaces. However, multiscale analysis is extremely time-consuming and heavily relies on decomposition methods and fusion rules. Selecting appropriate decomposition methods and fusion rules for different images remains challenging. Deep learning methods can eliminate the need for significant human intervention and enable autonomous network learning. However, most networks simply add feature maps without considering the connections between feature map channels. Designing an appropriate network architecture for specific source image fusion tasks is also challenging. Summary of the Invention
[0004] The present invention provides a method for fusing Synthetic Aperture Radar (SAR) images with multispectral (MS) images based on a convolutional neural network and an attention mechanism. The convolutional neural network is used to extract image features, and the attention module is used to achieve image feature fusion. This method can effectively enhance the spatial and spectral information of the image, thereby solving the problems existing in the prior art.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is:
[0006] A method for fusing synthetic aperture radar images and multispectral images comprises the following steps:
[0007] Step 1: Obtain SAR images and MS images of the same scene, preprocess the SAR images and MS images respectively, and construct a training set with the preprocessed SAR images and MS images;
[0008] Step 2: obtaining the high-frequency components of the SAR images in the training set, inputting the high-frequency components of the SAR images into a spatial feature extraction network, and extracting the spatial information of the SAR images through the spatial feature extraction network;
[0009] Upsampling the MS image in the training set by a factor of 3 to obtain an upsampled MS image, inputting the upsampled MS image into a spectrum preserving network, and extracting spectral information of the MS image through the spectrum preserving network;
[0010] Step 3: A feature fusion network based on the attention mechanism is used to fuse the spatial information of the SAR image and the spectral information of the MS image obtained in step 2.
[0011] Step 4: Using an adaptive moment estimation optimization algorithm, the spatial feature extraction network after extracting the spatial information of the SAR image in step 2, the spectrum preservation network after extracting the spectral information of the MS image in step 2, and the feature fusion network based on the attention mechanism after fusion in step 3 are combined into a fusion network model, and the training set constructed in step 1 is used to train the fusion network model. During training, an unsupervised joint loss function is used to constrain the training of the fusion network model until the fusion network model converges;
[0012] Step 5: Obtain the SAR image and MS image to be fused, obtain the high-frequency component of the SAR image to be fused, and upsample the MS image to be fused by a factor of 3 to obtain an upsampled MS image to be fused; input the high-frequency component of the SAR image to be fused and the upsampled MS image to be fused into the fusion network model trained to convergence obtained in step 4, and obtain a fused image through the fusion network model.
[0013] Furthermore, the spatial feature extraction network described in step 2 includes a dual-channel feature extraction network and a residual network; the high-frequency components of the SAR image in the training set are input into the dual-channel feature extraction network, and the features of the SAR image are learned by the dual-channel feature extraction network; the learned features of the SAR image are then input into the residual network for deep learning, and finally the spatial information of the SAR image is obtained through the residual network.
[0014] Furthermore, the dual-channel feature extraction network includes a multi-scale feature extraction module and a basic module, wherein: the multi-scale feature extraction module includes a convolution layer composed of convolution kernels of different sizes, and extracts different scale features of the SAR image through multiple convolution layers with convolution kernels of different sizes, and the features extracted by different convolution layers are linked by channels while performing jump links; the basic module includes three convolution layers composed of convolution kernels, and extracts shallow features of the SAR image through the basic module.
[0015] Furthermore, the multi-scale feature module is composed of two multi-scale channel networks in cascade, each scale channel contains two convolutional layers, wherein the convolution kernel size of the first convolution layer of the first scale channel is 1×3, and the convolution kernel size of the second convolution layer is 3×1; the convolution kernel size of the first convolution layer of the second scale channel is 1×5, and the convolution kernel size of the second convolution layer is 5×1; the convolution kernel size of the first convolution layer of the third scale channel is 1×7, and the convolution kernel size of the second convolution layer is 7×1;
[0016] Furthermore, the basic module includes three convolutional layers, wherein the convolution kernel size of the first convolutional layer is 9×9, the convolution kernel size of the second convolutional layer is 1×1, and the convolution kernel size of the third convolutional layer is 5×5.
[0017] Furthermore, the residual network includes four residual blocks and one convolutional layer, wherein the residual block consists of two convolutional layers and residual edges, which can be expressed as shown in the following formula:
[0018] y=W2δ(W1x)+x,
[0019] Where: x, y represent the input and output of the residual network structure respectively; W1, W2 represent the weights of the two convolutional layers in the residual block respectively; δ represents the ReLU activation function.
[0020] Furthermore, in step 3, the feature fusion network based on the attention mechanism can be expressed as the following formula:
[0021]
[0022] Among them: X1, X2 represent two input feature maps respectively; OUT represents the output of the feature fusion network; M i (X i ) represents the adaptive weight obtained by the channel attention module, i = 1 or 2; represents element-wise multiplication; Represents element-wise summation.
[0023] Furthermore, in step 4, the unsupervised joint loss function includes the spectral loss L spectral and detail loss L spatial, the calculation formula is as follows:
[0024] Loss = L spectral +λL spatial ,
[0025] Where λ represents the weight coefficient;
[0026] Spectral loss function L spectral It is expressed as the L1 norm between the fused image and the multispectral image, which is used to constrain the spectral distortion of the fused image. The spectral loss function L spectral The formula is as follows:
[0027] L spectral =|F-MS↑|1,
[0028] Where: F represents the fused image; MS↑ represents the 3x upsampled MS input image; |·|1 is the L1 norm;
[0029] Detail loss function L spatial Indicates the structural similarity between the fused image and the input SAR image, so that the fused image carries more SAR image spatial detail information. The detail loss function L spatial The formula is as follows:
[0030] L spatial =1-SSIM(F,SAR HP )
[0031] SAR HP represents the SAR image after high-pass filtering, and SSIM represents the computational fusion of the image and SAR HP The structural similarities between them.
[0032] Furthermore, the calculation formula of the structural similarity SSIM index is as follows:
[0033]
[0034] Where: μ R ,μ F represents the mean of the reference image and the fused image, represents the variance of the reference image and the fused image, σ RF represents the covariance between the reference image and the fused image; C1 = (k1L) 2 ,C2=(k2L) 2 , C3=C2 / 2, k1=0.01, k2=0.03, L is the dynamic range of image pixel values.
[0035] Compared with the prior art, the beneficial effects of the present invention are embodied in:
[0036] (1) This paper proposes a SAR image and MS image fusion method based on convolutional neural network and attention mechanism. It fully utilizes the spatial information of SAR image and the spectral information of MS image, so that the fusion result has rich spatial and spectral information.
[0037] (2) The present invention designs a dual-channel spatial feature extraction network. The spatial feature extraction network consists of a multi-scale feature extraction module, a three-layer convolutional basic module, and a residual module. The multi-scale feature extraction module is used to extract fine texture features of SAR images, and the basic module is used to extract the coarse structure of the image.
[0038] (3) This paper designs a feature fusion network based on the attention mechanism. Unlike the conventional direct linear addition strategy, this feature fusion network fully considers the connections between the feature map channels to be fused, adaptively weighting the feature map channels before fusion. This effectively improves the network's spectral preservation ability.
[0039] (4) This paper proposes an unsupervised joint loss function to constrain the training of the fusion network model. This guides the fusion network model to train towards better spatial and spectral preservation capabilities. The fusion result can simultaneously contain the spatial information of the SAR image and the spectral information of the MS image. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is a diagram of the overall architecture of an embodiment of the present invention.
[0041] Figure 2 It is a multi-scale feature extraction network structure diagram, where: Figure 2 a is the original multi-scale feature extraction network structure diagram; Figure 2 b is the optimized multi-scale feature extraction network structure diagram.
[0042] Figure 3 It is the residual block structure diagram.
[0043] Figure 4 This is the structure diagram of the residual module.
[0044] Figure 5 This is the structural diagram of the feature fusion network based on the attention mechanism.
[0045] Figure 6 is the first set of test dataset images, where: (a) is the upsampled MS image; (b) is the SAR image.
[0046] Figure 7 is the second set of test dataset images, where: (a) is the upsampled MS image; (b) is the SAR image.
[0047] Figure 8Figure 1 is the experimental result diagram of parameter λ, where: (a) is the correlation result between PSNR value and λ value; (b) is the correlation result between PSNR value and λ value.
[0048] Figure 9 The fusion results of each method for the first group of test images are as follows: (a) is the upsampled MS image; (b) is the SAR image; (c) is the fusion result of the HIS method; (d) is the fusion result of the NSCT method; (e) is the fusion result of the Wavelet method; (f) is the fusion result of the NSCT_FL method; (g) is the fusion result of the NSCT_PCNN method; (h) is the fusion result of the MSDCNN method; and (i) is the fusion result of the present invention.
[0049] Figure 10 for Figure 9 The magnified results of the white box area in the middle, where: (a) is the upsampled MS image; (b) is the SAR image; (c) is the fusion result of the HIS method; (d) is the fusion result of the NSCT method; (e) is the fusion result of the Wavelet method; (f) is the fusion result of the NSCT_FL method; (g) is the fusion result of the NSCT_PCNN method; (h) is the fusion result of the MSDCNN method; and (i) is the fusion result of the present invention.
[0050] Figure 11 for Figure 9 The magnification results of the black box area, where: (a) is the upsampled MS image; (b) is the SAR image; (c) is the fusion result of the HIS method; (d) is the fusion result of the NSCT method; (e) is the fusion result of the Wavelet method; (f) is the fusion result of the NSCT_FL method; (g) is the fusion result of the NSCT_PCNN method; (h) is the fusion result of the MSDCNN method; and (i) is the fusion result of the present invention.
[0051] Figure 12 The fusion results of each method in the second group of test images, among which: (a) is the upsampled MS image; (b) is the SAR image; (c) is the fusion result of the HIS method; (d) is the fusion result of the NSCT method; (e) is the fusion result of the Wavelet method; (f) is the fusion result of the NSCT_FL method; (g) is the fusion result of the NSCT_PCNN method; (h) is the fusion result of the MSDCNN method; and (i) is the fusion result of the present invention.
[0052] Figure 13 for Figure 12The enlarged results of the boxed area, where: (a) is the upsampled MS image; (b) is the SAR image; (c) is the fusion result of the HIS method; (d) is the fusion result of the NSCT method; (e) is the fusion result of the Wavelet method; (f) is the fusion result of the NSCT_FL method; (g) is the fusion result of the NSCT_PCNN method; (h) is the fusion result of the MSDCNN method; and (i) is the fusion result of the present invention.
[0053] Figure 14 To verify the result diagram of the attention feature fusion module, (a) is the upsampled MS image; (b) is the result of the additive fusion strategy method; (c) is the result of the attention feature fusion module. DETAILED DESCRIPTION
[0054] The present invention will be further described below with reference to the accompanying drawings and examples.
[0055] This embodiment proposes a method for fusing Synthetic Aperture Radar (SAR) images and multispectral (MS) images based on convolutional neural networks and attention mechanisms to achieve the fusion of SAR images and MS images. Figure 1 This is a complete network structure principle diagram of the method proposed in this embodiment. This embodiment includes the following steps:
[0056] Step 1: Obtain SAR images and MS images of the same scene, preprocess the SAR images and MS images respectively, and construct a training set with the preprocessed SAR images and MS images.
[0057] In this embodiment, the preprocessing process is:
[0058] 1a) The preprocessing process of SAR images is as follows: thermal noise removal → orbit correction → radiometric calibration → stitching → speckle filtering → terrain correction.
[0059] 1b) The preprocessing process of MS images is: radiometric calibration → atmospheric correction.
[0060] 1c) The preprocessed MS image and SAR image are sequentially registered and cropped to obtain a training set.
[0061] Step 2: Perform high-pass filtering on the SAR images in the training data set to obtain the high-frequency components of the SAR images, and perform 3x upsampling on the MS images in the training set to obtain upsampled MS images.
[0062] In this embodiment, the high frequency components of the obtained SAR image are input into the spatial feature extraction network, and the spatial information SAR of the SAR image is extracted through the spatial feature extraction network learning. FThe spatial feature extraction network consists of a dual-channel feature extraction network and a residual network, where:
[0063] The high frequency components of the SAR image are input into the dual-channel feature extraction network to learn the features of the SAR image. The dual-channel feature extraction network consists of a Figure 2 The multi-scale feature extraction module shown in FIG1 is composed of a basic module.
[0064] The multi-scale feature extraction module uses three convolutional layers with different sizes of convolution kernels, such as Figure 2 As shown in Figure a, the convolution kernel size of the first scale channel convolution layer of the multi-scale feature extraction module is 3×3, the convolution kernel size of the second scale channel convolution layer of the multi-scale feature extraction module is 5×5, and the convolution kernel size of the third scale channel convolution layer of the multi-scale feature extraction module is 7×7. The multi-scale feature extraction module extracts different scale features of the SAR image, and the feature maps extracted by different convolution layers are linked by channel. At the same time, the multi-scale feature extraction module introduces skip links, which can make fuller use of the spatial information in the SAR image and avoid information loss due to network deepening.
[0065] In order to speed up network training without affecting the network feature extraction capability, the multi-scale feature extraction module can also use six convolutional layers instead of three convolutional layers, such as Figure 2 As shown in Figure 2b, the convolution kernel size of the first convolution layer of the first scale channel in the multi-scale feature extraction module with six convolution layers is 1×3, the convolution kernel size of the second convolution layer of the first scale channel in the multi-scale feature extraction module with six convolution layers is 3×1, the convolution kernel size of the first convolution layer of the second scale channel in the multi-scale feature extraction module with six convolution layers is 1×5, the convolution kernel size of the second convolution layer of the second scale channel in the multi-scale feature extraction module with six convolution layers is 5×1, the convolution kernel size of the first convolution layer of the third scale channel in the multi-scale feature extraction module with six convolution layers is 1×7, and the convolution kernel size of the second convolution layer of the third scale channel in the multi-scale feature extraction module with six convolution layers is 7×1. High-level features of the SAR image are extracted through the multi-scale feature extraction module with six convolution layers.
[0066] The basic module consists of three convolutional layers: the first convolutional layer has a 9×9 kernel size, the second has a 1×1 kernel size, and the third has a 5×5 kernel size. The basic module is used to extract shallow features from SAR images.
[0067] like Figure 3 As shown in the figure, the features learned by the dual-channel feature extraction network are input into the residual network for further deep learning, and finally the spatial feature information of the SAR image is obtained.F The residual network consists of four residual blocks and a convolutional layer. The residual network can avoid the problem of network gradient disappearance caused by network deepening while deepening the network. The residual block structure is shown in the figure below. Figure 4 As shown in Figure 2, the residual block consists of two convolutional layers and a residual edge, and the formula can be expressed as:
[0068] y=W2δ(W1x)+x,
[0069] Where: x, y represent the input and output of the residual network structure respectively; W1, W2 represent the weights of the two convolutional layers in the residual block respectively; δ represents the ReLU activation function.
[0070] In this embodiment, the upsampled MS image is input into the spectrum preservation network, and the network extracts the spectrum information MS of the MS image. F The spectrum preserving network consists of a residual block with the following structure: Figure 4 shown.
[0071] Step 3: Use the feature fusion network based on the attention mechanism to transform the spatial information of the SAR image obtained in step 2 into F and spectral information of MS images F to integrate.
[0072] The feature fusion network structure based on the attention mechanism is as follows Figure 5 As shown, unlike the common method of directly linearly adding two images, the feature fusion network based on the attention mechanism in this embodiment fully learns the correlation between feature map channels and obtains channel adaptive weights. The feature map channels are then weighted and summed according to the adaptive weights. The feature fusion network based on the attention mechanism in this embodiment can be expressed as:
[0073]
[0074] Among them: X1, X2 represent two input feature maps respectively; OUT represents the output of the feature fusion network; M i (X i ) represents the adaptive weight obtained by the channel attention module, i = 1 or 2; represents element-wise multiplication; Represents element-wise summation.
[0075] Step 4: Adaptive moment estimation optimization algorithm (Adam) is used to combine the spatial feature extraction network after extracting the spatial information of the SAR image in step 2, the spectrum preservation network after extracting the spectral information of the MS image in step 2, and the feature fusion network based on the attention mechanism after fusion in step 3 into a complete fusion network model. The fusion network model is trained using the training set constructed in step 1. During training, an unsupervised joint loss function is used to constrain the training of the fusion network model until the fusion network model converges.
[0076] In this embodiment, the unsupervised joint loss function fully considers the spatial detail information of the SAR image and the spectral information of the MS image. The unsupervised joint loss function includes the spectral loss L spectral and detail loss L spatial , the calculation formula is as follows:
[0077] Loss = L spectral +λL spatial ,
[0078] Where λ represents the weight coefficient.
[0079] Spectral loss function L spectral It is expressed as the L1 norm between the fused image and the multispectral image, which is used to constrain the spectral distortion of the fused image. The spectral loss function L spectral The formula is as follows:
[0080] L spectral =|F-MS↑|1,
[0081] Where: F represents the fused image; MS↑ represents the 3x upsampled MS input image; |·|1 is the L1 norm.
[0082] Detail loss function L spatial Indicates the structural similarity between the fused image and the input SAR image, so that the fused image carries more SAR image spatial detail information. The detail loss function L spatial The formula is as follows:
[0083] L spatial =1-SSIM(F,SAR HP ),
[0084] SAR HP Represents the SAR image after high-pass filtering. The calculation formula of the structural similarity SSIM index is as follows:
[0085]
[0086] Among them, μ R ,μ F represents the mean of the reference image and the fused image, represents the variance of the reference image and the fused image, σ RF represents the covariance between the reference image and the fused image; C1 = (k1L) 2 ,C2=(k2L) 2 , C3=C2 / 2, k1=0.01, k2=0.03, L is the dynamic range of image pixel values.
[0087] Step 5: Obtain the SAR image and MS image to be fused, obtain the high-frequency component of the SAR image to be fused, and upsample the MS image to be fused by a factor of 3 to obtain an upsampled MS image to be fused; input the high-frequency component of the SAR image to be fused and the upsampled MS image to be fused into the fusion network model trained to convergence obtained in step 4, and obtain a fused image through the fusion network model.
[0088] The effectiveness of this embodiment is further illustrated by simulation experiments below.
[0089] Simulation experiment:
[0090] (1) Dataset
[0091] The training data for the simulation experiments comes from Sentinel-1 and LandSat 8. Sentinel-1 is an Earth observation satellite in the European Space Agency's Copernicus Mission (GMES). It consists of two satellites and carries a C-band synthetic aperture radar, providing continuous imagery (day, night, and all weather conditions). Landsat is a series of Earth observation satellites used by the United States to monitor Earth's resources and environment, primarily for resource exploration, environmental monitoring, and natural disaster prevention. The simulation experiments used Sentinel-1 Ground Range Detected (GRD) SAR data with a resolution of 10 meters and a VH polarization mode. Multispectral images were selected from Landsat 8's true color images, consisting of bands 4, 3, and 2, with a resolution of 30 meters. The images include features such as ocean areas, cities, vegetation, and mountains, verifying the applicability of the proposed method.
[0092] In order to better train the network, the source images are cropped into small patches for training. The MS images are cropped to 32×32 pixels, and the SAR images are cropped to 96×96 pixels. A total of 25,600 pairs of images are cropped for training the network. In addition, 656 pairs of SAR and MS images with sizes of 768×768 and 256×256 pixels are selected for testing the network. Figure 6 and Figure 7 Shown are two sets of test images.
[0093] (2) Experimental setup
[0094] The simulation experiment used a dataset split in a 9:1 ratio as the training and validation sets for network training. Training was completed within 200 epochs with a batch size of 8. Adaptive data estimator (Adam) was used to constrain the loss function. The initial learning rate was 0.0002, and the learning decay rate was 0.5. If the model performance did not improve after three epochs, the learning rate was reduced. If the model performance did not improve after 10 epochs, training was terminated to avoid overfitting.
[0095] In order to further verify the effectiveness of the proposed joint loss function, the simulation experiment sets different lambda values to train the network. The figure below shows the PSNR (the larger the better) and SAM (the smaller the better) of the network trained with different lambda values on the test set. Figure 8 As can be seen from the figure, PSNR decreases as λ increases, while SAM increases. The best performance is achieved when λ is set to 0.1. Therefore, the simulation experiment sets the loss function parameter λ to 0.1. The network was implemented in Python 3.8 and TensorFlow 2.7. Experiments were conducted on an NVIDIA GeForce RTX 3060 laptop GPU.
[0096] (3) Evaluation indicators and comparative fusion methods
[0097] In order to quantitatively evaluate the advantages and disadvantages of the proposed algorithm compared with traditional fusion algorithms and other deep learning algorithms, seven evaluation indicators are selected to evaluate the quality of the fusion results, namely: correlation coefficient (CC), peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), structural similarity (SSIM), no reference image index (QNR), spatial distortion index D s , spectral distortion index D λ .
[0098] Comparing the fusion methods, we selected traditional fusion methods and deep learning fusion methods: HIS, NSCT, Wavelet, NSCT_FL, NSCR_PCNN, and MSDCNN.
[0099] (4) Result analysis
[0100] In the experiment, two sets of images were selected from a test set of 656 images containing various types of ground features to analyze the visual effects and objective evaluation indicators of the proposed method and the comparative method. In the objective evaluation indicator table, the optimal value is bolded, the suboptimal value is underlined, and the third-best value is marked with a downward wavy line.
[0101] The first group selected areas with rivers, and the fusion results of each method were as follows: Figure 9In order to better observe the texture details of the fused image, the white and black box areas in the figure are enlarged, as shown in Figure 10 , Figure 11 .from Figure 9 As can be seen from the figure, HIS, NSCT, and Wavelet have serious spectral distortion and also exhibit severe spatial distortion. NSCT_FL and NSCT_PCNN have relatively good performance in spectral preservation, but exhibit severe spatial distortion in spatial details. In comparison, CNN-based methods perform better in spectral preservation than traditional methods, but MSDCNN is inferior to DAFCNN in feature extraction capabilities. Figure 10 In the white box magnified area, such as Figure 11 ,It is obvious that MSDCNN has a certain spectral distortion.,From the objective evaluation results as shown in Table 1, the performance of the CNN-based fusion ,method is far ahead of other methods, and DAFCNN is in a leading position compared with ,other algorithms.
[0102] Table 1 Figure 9 The first group of quantitative evaluation indicators of fusion results
[0103]
[0104] The second group selected urban and mountainous areas, and the fusion results were as follows: Figure 12 As shown, the box area in the figure is enlarged to Figure 13 From the fusion results, we can see that the traditional methods HIS, NSCT, Wavelet spectrum and space have serious distortion, NSCT_FL and NSCT_PCNN have better performance in spectrum preservation, but Figure 13 It can be seen that these two methods have serious spatial distortion. The fusion method based on CNN is still better than the traditional method. Figure 13 It can be seen that MSDCNN introduces too much speckle noise, which causes the structure of objects in the image to change. DAFCNN can also achieve better spectral fidelity while adding more details. Figure 12 Corresponding objective evaluation indicators (as shown in Table 2), DAFCNN shows the best fusion value in most indicators.
[0105] Table 2 Figure 12 The second group of quantitative evaluation indicators of fusion results
[0106]
[0107] (5) Verify the performance of the attention feature fusion network
[0108] In order to verify the effectiveness of the AFF module, the simulation experiment will Figure 5The AFF module in the network structure DAFCNN_AFF shown in the figure is replaced by the fusion strategy (DAFCNN_ADD) of adding feature maps by elements. Figure 14 , we can see that the AFF module has superior spectral preservation. Table 3 shows that DAFCNN_AFF outperforms DAFCNN_ADD in almost all quantitative metrics. This also demonstrates that the proposed AFF module significantly improves spectral fidelity in the SAR and MS image fusion task, as shown in Table 3.
[0109] Table 3 Comparison results between AFF module and ADD strategy
[0110]
[0111] In summary, the dual-channel feature extraction and attention fusion convolutional neural network proposed in this embodiment shows excellent performance in the SAR image and MS image fusion task.
[0112] The preferred embodiments of the present invention are described in detail above with reference to the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. Such combinations should also be regarded as the contents disclosed in this disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.
[0113] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the scope of the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by those skilled in the art should fall within the scope of protection of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.
Claims
1. A method for fusing synthetic aperture radar images and multispectral images, characterized in that: The following steps are involved: Step 1: Obtain SAR images and MS images of the same scene, preprocess the SAR images and MS images respectively, and construct a training set with the preprocessed SAR images and MS images; Step 2: obtaining the high-frequency components of the SAR images in the training set, inputting the high-frequency components of the SAR images into a spatial feature extraction network, and extracting the spatial information of the SAR images through the spatial feature extraction network; Upsampling the MS image in the training set by a factor of 3 to obtain an upsampled MS image, inputting the upsampled MS image into a spectrum preserving network, and extracting spectral information of the MS image through the spectrum preserving network; Step 3: A feature fusion network based on the attention mechanism is used to fuse the spatial information of the SAR image and the spectral information of the MS image obtained in step 2. Step 4: Using an adaptive moment estimation optimization algorithm, the spatial feature extraction network after extracting the spatial information of the SAR image in step 2, the spectrum preservation network after extracting the spectral information of the MS image in step 2, and the feature fusion network based on the attention mechanism after fusion in step 3 are combined into a fusion network model, and the training set constructed in step 1 is used to train the fusion network model. During training, an unsupervised joint loss function is used to constrain the training of the fusion network model until the fusion network model converges; Step 5: Obtain the SAR image and MS image to be fused, obtain the high-frequency component of the SAR image to be fused, and perform 3-fold upsampling on the MS image to be fused to obtain an upsampled MS image to be fused; input the high-frequency component of the SAR image to be fused and the upsampled MS image to be fused into the fusion network model trained to convergence obtained in step 4, and obtain a fused image through the fusion network model; The spatial feature extraction network described in step 2 includes a dual-channel feature extraction network and a residual network; the high-frequency components of the SAR image in the training set are input into the dual-channel feature extraction network, and the features of the SAR image are obtained by learning through the dual-channel feature extraction network; The learned features of the SAR image are then input into the residual network for deep learning, and finally the spatial information of the SAR image is obtained through the residual network; The dual-channel feature extraction network includes a multi-scale feature extraction module and a basic module, wherein: the multi-scale feature extraction module includes a convolution layer composed of convolution kernels of different sizes, and extracts different scale features of the SAR image through multiple convolution layers with different convolution kernels of different sizes, and the features extracted by different convolution layers are linked by channels while performing jump links; the basic module includes three convolution layers composed of convolution kernels, and extracts shallow features of the SAR image through the basic module; The multi-scale feature module consists of two multi-scale channel networks in cascade, each scale channel contains two convolutional layers, where the convolution kernel size of the first convolution layer of the first scale channel is 1×3, and the convolution kernel size of the second convolution layer is 3×1; the convolution kernel size of the first convolution layer of the second scale channel is 1×5, and the convolution kernel size of the second convolution layer is 5×1; the convolution kernel size of the first convolution layer of the third scale channel is 1×7, and the convolution kernel size of the second convolution layer is 7×1; The basic module includes three convolutional layers, wherein the convolution kernel size of the first convolutional layer is 9×9, the convolution kernel size of the second convolutional layer is 1×1, and the convolution kernel size of the third convolutional layer is 5×5.
2. The method for fusing synthetic aperture radar images and multispectral images according to claim 1, characterized in that: The residual network includes four residual blocks and one convolutional layer, where the residual block consists of two convolutional layers and residual edges, which can be expressed as shown in the following formula: y=W2δ(W1x)+x, Where: x, y represent the input and output of the residual network structure respectively; W1, W2 represent the weights of the two convolutional layers in the residual block respectively; δ represents the ReLU activation function.
3. The method for fusing synthetic aperture radar images and multispectral images according to claim 1, characterized in that: In step 3, the feature fusion network based on the attention mechanism can be expressed as the following formula: Among them: X1, X2 represent two input feature maps respectively; OUT represents the output of the feature fusion network; M i (X i ) represents the adaptive weight obtained by the channel attention module, i = 1 or 2; represents element-wise multiplication; Represents element-wise summation.
4. The method for fusing synthetic aperture radar images and multispectral images according to claim 1, characterized in that: In step 4, the unsupervised joint loss function includes the spectral loss L spectral and detail loss L spatial , the calculation formula is as follows: Loss=L spectral +λL spatial , Where λ represents the weight coefficient; Spectral loss function L spectral It is expressed as the L1 norm between the fused image and the multispectral image, which is used to constrain the spectral distortion of the fused image. The spectral loss function L spectral The formula is as follows: L spectral =|F-MS↑|1, Where: F represents the fused image; MS↑ represents the 3x upsampled MS input image; |·|1 is the L1 norm; Detail loss function L spatial Indicates the structural similarity between the fused image and the input SAR image, so that the fused image carries more SAR image spatial detail information. The detail loss function L spatial The formula is as follows: L spatial =1-SSIM(F,SAR HP ) SAR HP represents the SAR image after high-pass filtering, and SSIM represents the computational fusion of the image and SAR HP The structural similarities between them.
5. The method for fusing synthetic aperture radar images and multispectral images according to claim 4, characterized in that: The calculation formula of the structural similarity SSIM index is as follows: Where: μ R ,μ F represents the mean of the reference image and the fused image, represents the variance of the reference image and the fused image, σ RF represents the covariance between the reference image and the fused image; C1 = (k1L) 2 ,C2=(k2L) 2 , C3=C2 / 2, k1=0.01, k2=0.03, L is the dynamic range of image pixel values.