Method, system, storage medium and computer program product for SAR and optical image fusion and intelligent classification

Abstract

The present invention discloses a method, system, storage medium and computer program product for SAR and optical image fusion and intelligent classification, belonging to the field of artificial intelligence and synthetic aperture radar target recognition technology, and solving the problems that the existing technology is prone to sensor characteristic mismatch when fusing multi-source images, which leads to loss of fusion information or makes the model more and more complex, which leads to high model operation cost and performance degradation. The present invention preprocesses the source data set; constructs a dual-coding fusion classification network including two independent encoders, and trains the source data set obtained by preprocessing to obtain a trained dual-coding fusion classification network; uses the trained dual-coding fusion classification network to fuse SAR images and optical images, and performs end-to-end intelligent classification of ground objects. The present invention is used for end-to-end image fusion and intelligent classification of ground objects.

Description

Technical Field

[0001] A method, system, storage medium and computer program product for SAR and optical image fusion and intelligent classification, which are used for end-to-end image fusion and intelligent classification of soil objects, and belong to the technical field of artificial intelligence and synthetic aperture radar target recognition. Background Art

[0002] Synthetic aperture radar (SAR) and optical sensors are the most widely used earth observation sensors at present, and they have different imaging capabilities. SAR images present grayscale images. When the object environment is complex, it is far from enough to distinguish the object category based on contour information alone. If SAR images can be given color and texture detail information, the application scope of SAR images will be greatly increased. Optical images happen to have good color and texture detail information. By fusing the contour information of SAR images with the color and texture detail information of optical images, a high-quality fused image with complementary information can be formed. On the contrary, optical remote sensing images often lose object information due to cloud cover or cloud shadow, which seriously limits the application of optical images in military and scientific research. If it is possible to obtain object information through the clouds, it will effectively make up for the shortcomings of optical images. SAR images happen to be able to work all day and all weather and have strong penetration ability. In a cloud-covered environment, fusing SAR images with optical images can effectively make up for the lack of information.

[0003] Based on the existing methods, although there are many methods of multi-source image fusion and intelligent classification based on deep learning, there are some problems at this stage:

[0004] First: existing multi-source images have different physical properties and imaging principles. When fusing, they often encounter the problem of sensor characteristic mismatch, which leads to the loss of fusion information. To address this problem, some single encoder structures such as DeFusion, DIDFuse, MCANet, etc., separate high-order and low-order features for cross-fusion. Although the problem of sensor mismatch is effectively controlled, the model is becoming more and more complex. Research on dual encoding, such as the invention patent application with publication number CN117095265A, discloses a dual-branch fusion method, which can better retain image information by extracting deep features of high-frequency information and then fusing them, but it is difficult to solve the problem of sensor mismatch.

[0005] Second, optical images are seriously affected by weather, and SAR images have serious speckle noise. When fusion is performed, how to effectively extract this part of the obscured information and effectively reduce the impact of noise becomes a difficulty. Some popular methods include cloud removal and denoising of the two images. However, these methods are all focused on before fusion, and it is difficult to adjust the problem during the fusion process. Another idea is to build an attention mechanism to reconstruct the detailed information of the optical image. For example, Fu et al. built a dual attention module to adaptively integrate local-global relationships. Huang et al. proposed a cross-attention module for obtaining contextual information of all pixels. The invention patent application with publication number CN113420838A discloses a multi-scale attention feature fusion and proposes a spatial / spectral attention module to capture spatial and spectral information. However, the effect is not ideal when the optical image is severely obscured, and it is difficult to consider the denoising problem of SAR images.

[0006] Third, the non-uniform distribution of object categories often leads to sample imbalance, which in turn has a negative impact on model training and performance. In previous studies, the loss function was first introduced and then adjusted according to actual needs. Chen et al. used a gradient normalized weighted loss function to automatically balance the model training tasks and improve accuracy. Yeung et al. constructed a unified focal loss using a new hierarchical framework to deal with sample imbalance. These examples verify that the construction of the loss function is very effective in adjusting the network structure and alleviating sample imbalance. However, indifferent weights often cannot balance the loss and sample imbalance problems in the fusion and intelligent classification process. An adaptive combination and weighted loss function is crucial.

[0007] In summary, the prior art has the following technical problems:

[0008] 1. When fusing multi-source images, it is easy to cause mismatch of sensor characteristics, which leads to loss of fusion information or makes the model more and more complex, which leads to high model operation cost and performance degradation.

[0009] 2. It is impossible to effectively suppress SAR image noise interference and extract information blocked by optical images during the fusion process;

[0010] 3. Due to the non-uniform distribution of object categories, the use of indifferent weights cannot balance the loss and sample imbalance problems in the fusion and intelligent classification process. Summary of the invention

[0011] The purpose of the present invention is to provide a method, system, storage medium and computer program product for SAR and optical image fusion and intelligent classification, so as to solve the problems in the prior art that, when fusing multi-source images, sensor characteristics are easily mismatched, resulting in loss of fusion information or the model becomes increasingly complex, resulting in high model operation cost and performance degradation.

[0012] In order to achieve the above object, the technical solution adopted by the present invention is:

[0013] A method for SAR and optical image fusion and intelligent classification comprises the following steps:

[0014] S1, preprocess the source data set;

[0015] S2, constructing a dual-encoding fusion classification network including two independent encoders, and training it through the source data set obtained by preprocessing to obtain a trained dual-encoding fusion classification network;

[0016] S3. Use the trained dual-coding fusion classification network to fuse SAR images and optical images, and perform end-to-end intelligent classification of ground objects.

[0017] Further, the specific steps of step S1 are:

[0018] S1.1. Classify the label files in the source dataset into six categories: forest, field, water, building, road and background. The source dataset includes at least 100 SAR images, optical images and real label images of 5556×3704 size;

[0019] S1.2. Use the linear interpolation method to crop the SAR images, optical images, and true label images in the classified source dataset into at least 18,700 images of 320*320 size;

[0020] S1.3. Divide the trimmed data set into training set, validation set, and test set in a random ratio of 6:2:2. The training set should contain at least 11,220 images each, and the validation set and test set should contain at least 3,740 images each.

[0021] Furthermore, the dual-encoding fusion classification network includes a SAR encoder for extracting SAR features from SAR images, an optical encoder for extracting optical features from optical images, and a decoder for fusing and intelligently classifying SAR features and optical features;

[0022] The SAR encoder includes an input layer S1 for inputting a SAR image, a normalization layer S2 for performing channel normalization processing on the SAR image input by the input layer S1, a downsampling layer for downsampling the normalization layer S2, a shallow feature extraction module for suppressing noise interference of the SAR image while compensating for the occluded information of the optical image and performing SAR shallow feature extraction on the shallow features obtained by the downsampling layer, and an upsampling layer for upsampling the SAR shallow features to obtain SAR features;

[0023] The optical encoder includes a semantic feature extraction module for extracting optical features from an optical image.

[0024] Furthermore, the specific implementation steps of the shallow feature extraction module are:

[0025] For shallow features f SAR ∈R B×C×H×W Synchronous convolution extracts low-resolution feature maps D1 and D2 of shallow features;

[0026] The low-resolution feature map D1 is reconstructed and transposed, and then point-multiplied with the low-resolution feature map D2 to obtain the combined feature map of the SAR image. The low-resolution feature map D1 is reconstructed into Then transpose it into R B×Q×C' , the low-resolution feature map D2 is reconstructed into i represents the i-th SAR image, n represents the n-th SAR image, and C' represents the number of new channels after convolution, which is reduced by 2 based on the original channels. 3 , B represents the batch size, C represents the number of channels, and R represents a set of real numbers;

[0027] Then the combined feature map is processed by exponential operation, summation and normalization in sequence to obtain the feature weight of the SAR image;

[0028] The feature weight formula is:

[0029]

[0030] in, represents the feature weight of the i-th SAR image in the SAR image, represents the low-resolution feature map D1 of the i-th SAR image, represents the low-resolution feature map D2 of the i-th SAR image, The low-resolution feature map D1 represents the j-th pixel of the i-th SAR image. represents the low-resolution feature map D2 of the j-th pixel of the i-th SAR image, T represents the transposition operation, H and W represent the height and width respectively, Q = H × W, exp represents the exponential function with base e;

[0031] Then flatten the shallow feature space into R B×C×Q The SAR shallow features are obtained by weighted multiplication with the result of transposing the feature weights of each pixel in the SAR image;

[0032] The formula for SAR shallow features is:

[0033]

[0034] in, represents the SAR shallow features of the i-th image in the SAR image, and γ is a self-learning parameter that can learn and adjust itself. represents the feature weight of the jth pixel of the i-th SAR image, Represents the shallow features of the jth pixel in the i-th SAR image.

[0035] Further, the downsampling layer includes a convolutional layer S3, a convolutional layer S4 and a convolutional layer S5 which are connected in sequence;

[0036] The upsampling layer includes a convolutional layer S6 and a convolutional layer S7 connected in sequence;

[0037] The semantic feature extraction module includes a 3-channel input layer O1, which is convolved to obtain a 32-channel normalized layer O2, and the output of the normalized layer O2 is convolved into a 16-channel feature block A, and the feature block A is sequentially convolved, normalized and mixed activation function operations are performed to obtain a convolution layer O3 of feature block 1, feature block 1 and feature block A are added to obtain feature block B, and feature block B is sequentially convolved, normalized and mixed activation function operations are performed to obtain a convolution layer O4 of feature block 2, feature block 2, feature block 1 and feature block A are added to obtain feature block C, and feature block C is sequentially convolved, normalized and mixed activation function operations are performed to obtain a convolution layer O5 of feature block 3, and feature block 3, feature block 2, feature block 1 and feature block A are added to obtain a convolution layer O6 of optical features;

[0038] The decoder includes a fusion feature module F1 for fusing SAR features and optical features, a fusion image module F2 for performing convolution, ReLU activation function and normalization operations on the fused features to obtain a fused image, and a classification module F3 for performing deconvolution processing and decoding on the fused image to obtain an intelligent classification result of the ground object.

[0039] Furthermore, a new loss function consisting of information loss, pixel loss, and noise loss is constructed to train and evaluate the dual encoding fusion classification network. Specifically:

[0040] L total =L I +aL P +βL N

[0041] Among them, L I It represents the information loss between the fused image and the true label image, L P It refers to the pixel loss of the SAR features output by the SAR encoder and the optical features and fusion features output by the optical encoder. N It refers to the noise loss of the input and output characteristics of the SAR encoder and the optical encoder. α and β represent two tuning parameters, which are 1 or 5 respectively after adaptive adjustment;

[0042] Information loss refers to the information entropy between the fused image and the true label image. The amount of information is inversely proportional to the probability of the information occurring. If the probability of an event occurring is A(x), the amount of information I(x) is expressed as:

[0043] I(x)=-log(A(x))

[0044] The information entropy H(X) of all information is expressed as:

[0045]

[0046] If there are two independent probability distributions P(x) and q(x) for the same random variable, p(x i ) represents the true distribution of the i-th SAR image or optical image, q(x i ) represents the predicted distribution of the dual encoding fusion classification network, x i represents the i-th SAR image or optical image, p(x i ) and q(x i ) The difference between the two distributions is expressed as the relative entropy D KL To express:

[0047]

[0048] Breaking down the relative entropy formula:

[0049]

[0050] The former H(p(x)) is the information entropy, the latter That is the cross entropy, which is equal to the sum of relative entropy and information entropy. It can measure the cross entropy of the difference between two different probability distributions in the same random variable. In order to retain the amount of information of the fused image, the cross entropy is finally used as the information loss, which is expressed as:

[0051]

[0052] Pixel loss First determine the maximum pixel value of the SAR feature output by the SAR encoder and the optical feature output by the optical encoder, and then calculate the L1 norm loss of the pixel value of the fused feature after fusion and the maximum pixel value of the SAR feature and optical feature before fusion:

[0053]

[0054] Where H and W represent the height and width of the SAR image and the optical image, respectively. f represents the fusion feature, I SAR ,I OPT Represent the SAR characteristics and optical characteristics output by the SAR encoder and the optical encoder;

[0055] Noise loss:

[0056]

[0057] in, represents the Sobel gradient operator.

[0058] A system for SAR and optical image fusion and intelligent classification comprises a memory and a processor. The memory stores a computer program, the computer program comprises the method for SAR and optical image fusion and intelligent classification, and the processor executes the computer program to implement the steps of the method for SAR and optical image fusion and intelligent classification.

[0059] A computer-readable storage medium stores a computer program, the computer program includes the method for fusion and intelligent classification of SAR and optical images, and the steps of the method for fusion and intelligent classification of SAR and optical images are implemented when the computer program is executed by a processor.

[0060] A computer program product comprises a computer program, wherein the computer program comprises the method for fusion and intelligent classification of SAR and optical images, and the steps of the method for fusion and intelligent classification of SAR and optical images are implemented when the computer program is executed by a processor.

[0061] Compared with the prior art, the advantages of the present invention are:

[0062] 1. The present invention utilizes two independent encoders in a dual-coding fusion classification network to extract features from SAR images and optical images respectively, which effectively solves the mismatch problem between SAR images and optical images. That is, it solves the problem of sensor characteristic mismatch while effectively reducing the model complexity and avoiding the problems of high model operation cost and performance degradation.

[0063] Second, the SAR encoder of the present invention constructs a shallow feature extraction module SEM that suppresses SAR image noise interference and extracts SAR shallow features at the same time. The shallow feature extraction module uses the difference between samples within a class caused by SAR noise to synchronously extract two low-resolution feature maps to control noise interference, and then processes the two low-resolution feature maps separately to obtain a combined feature map by point multiplication, thereby increasing shallow information and taking into account both global and local details. The shallow feature extraction module SEM that is resistant to noise interference is constructed to capture SAR image information in the SAR encoder, suppress noise interference, and compensate for the problem of optical image occlusion;

[0064] 3. The loss function in the present invention is a new loss function that is a weighted combination of information loss, pixel loss, and noise loss, and introduces category weights to the information loss, so that the model pays more attention to minority categories during training and better balances the importance of categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 It is a schematic diagram of the framework of the present invention;

[0067] Figure 2 It is a structural diagram of the dual-coding fusion classification network in the present invention;

[0068] Figure 3 It is a processing flow chart of the shallow feature extraction module in the present invention;

[0069] Figure 4 are some images of the original data set in the present invention, where (a) represents a SAR image, (b) represents an optical image, and (c) represents a true labeled image;

[0070] Figure 5 are partial images of the source data set after preprocessing in the present invention, wherein (a) represents the preprocessed SAR image, (b) represents the preprocessed optical image, and (c) represents the preprocessed true label image;

[0071] Figure 6: is a diagram of the classification results of the dual-encoding fusion classification network in the present invention and other popular networks, wherein (a) represents a SAR image, (b) represents an optical image, (c) represents a true label image, (d) represents a Densefuse classification result, (e) represents a DeFusion classification result, (f) represents a DIDFuse classification result, (g) represents a PSFN classification result, (h) represents an MCANet classification result, and (i) represents a classification result of the dual-encoding fusion classification network of the present invention;

[0072] Figure 7 : It is a comparative classification result diagram showing that the shallow feature extraction module in the present invention can effectively extract semantic information and suppress noise interference, wherein (a) represents a SAR image, (b) represents an optical image, (c) represents a true label image, (d) represents a No-SEM classification result, (e) represents an OPT+SEM, SAR+SEM classification result, (f) represents an OPT+SEM classification result, and (g) represents a SAR+SEM classification result adopted by the present invention;

[0073] Figure 8 : is a comparative classification result diagram of the loss function in the present invention and the popular loss function, where (a) represents the SAR image, (b) represents the optical image, (c) represents the real label image, and (d) represents the information loss L I Classification result, (e) represents the reconstruction loss L R and the eigendecomposition loss L FD Classification result, (f) represents texture loss L T and structural loss L S Classification results, (g) indicates that the present invention adopts information loss L I , pixel loss L P , Noise loss L N Classification results; DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1 and Figure 2 As shown, this embodiment discloses a method for fusing SAR and optical images, comprising the following steps:

[0076] S1. Preprocess the source data set, which includes the matched and calibrated SAR images, optical images and real label images;

[0077] The specific steps are:

[0078] S1.1. Classify the label files in the source data set into six categories: forest, field, water, building, road and background. The source data set includes at least 100 SAR images, optical images and real label images of 5556×3704 size. The source data set used in the experiment in this embodiment is the open source data set WHU-OPT-SAR for fusion of SAR images and optical images. The source data set contains 100 5556×3704 optical and SAR images of the same area, covering an area of ​​about 50,000 square kilometers in Hubei Province, China (30°N-33°N). This area has a suitable climate, with an altitude difference ranging from 50 meters to 3000 meters, covering many different terrains, and can carry out fusion research well. The original image is 5556×3704 in size and is divided into 8 categories, such as Figure 4 shown.

[0079] S1.2. Use the linear interpolation method to crop the SAR images, optical images, and true label images in the classified source dataset into at least 18,700 images of 320*320 size;

[0080] S1.3. Divide the trimmed data set into training set, validation set, and test set in a random ratio of 6:2:2. The training set should contain at least 11,220 images each, and the validation set and test set should contain at least 3,740 images each.

[0081] S2. Based on the different imaging mechanisms of SAR and optical images, there is often a problem of sensor mismatch. Two independent encoders are constructed to perform targeted processing on the imaging mechanisms of SAR and optical images respectively. That is, a dual-encoding fusion classification network including two independent encoders is constructed, and the source data set obtained by preprocessing is used for training to obtain a trained dual-encoding fusion classification network.

[0082] The dual-encoding fusion classification network includes a SAR encoder for extracting SAR features from SAR images, an optical encoder for extracting optical features from optical images, and a decoder for fusing and intelligently classifying SAR features and optical features;

[0083] The feature images output by the two encoders are fused and input into the decoder to obtain a 320*320*128 fusion feature. After convolution, ReLU activation function and normalization operation, the fused image is obtained. Finally, after deconvolution processing and decoding, the fused intelligent classification result is obtained to realize an end-to-end ground object fusion classification.

[0084] Taking into account the respective problems of SAR images and optical images: SAR images have serious speckle noise, and optical images are seriously affected by weather and easily blocked. By utilizing the intra-class differences between samples caused by SAR noise, a shallow feature extraction module SEM is added to the SAR encoder to suppress noise interference and extract SAR shallow features. Therefore, the SAR encoder includes an input layer S1 for inputting SAR images, a normalization layer S2 for performing channel normalization processing on the SAR image input by the input layer S1, a downsampling layer for downsampling the normalization layer S2, a shallow feature extraction module for suppressing SAR image noise interference and compensating for the blocked information of the optical image, and extracting SAR shallow features from the shallow features obtained by the downsampling layer, and an upsampling layer for upsampling the SAR shallow features to obtain SAR features;

[0085] The optical encoder includes a semantic feature extraction module for extracting optical features from an optical image.

[0086] The specific implementation steps of the shallow feature extraction module are:

[0087] For shallow features f SAR ∈R B×C×H×W Synchronous convolution extracts low-resolution feature maps D1 and D2 of shallow features;

[0088] The low-resolution feature map D1 is reconstructed and transposed, and then point-multiplied with the low-resolution feature map D2 to obtain the combined feature map of the SAR image. The low-resolution feature map D1 is reconstructed into Then transpose it into R B×Q×C' , the low-resolution feature map D2 is reconstructed into i represents the i-th SAR image, n represents the n-th SAR image, and C' represents the number of new channels after convolution, which is reduced by 2 based on the original channels. 3 , B represents the batch size, C represents the number of channels, and R represents a feature layer;

[0089] Then the combined feature map is processed by exponential operation, summation and normalization in sequence to obtain the feature weight of the SAR image;

[0090] The feature weight formula is:

[0091]

[0092] in, represents the feature weight of the i-th SAR image in the SAR image, represents the low-resolution feature map D1 of the i-th SAR image, represents the low-resolution feature map D2 of the i-th SAR image, The low-resolution feature map D1 represents the j-th pixel of the i-th SAR image. represents the low-resolution feature map D2 of the j-th pixel of the i-th SAR image, T represents the transposition operation, H and W represent the height and width respectively, Q = H × W, exp represents the exponential function with base e;

[0093] Then flatten the shallow feature space into R B×C×Q The SAR shallow features are obtained by weighted multiplication with the result of transposing the feature weights of each pixel in the SAR image;

[0094] The formula for SAR shallow features is:

[0095]

[0096] in, represents the SAR shallow features of the i-th image in the SAR image, and γ is a self-learning parameter that can learn and adjust itself. represents the feature weight of the jth pixel of the i-th SAR image, Represents the shallow features of the jth pixel in the i-th SAR image.

[0097] Experiments were conducted on the WHU-OPT-SAR dataset, comparing the results of not adding a shallow feature extraction module (No SEM), adding a shallow feature extraction module to the SAR encoder (SAR+SEM) and the optical encoder (OPT+SEM), and adding a shallow feature extraction module to both encoders (OPT+SEM, SAR+SEM). The specific classification results are shown in the figure. Figure 6 As shown in Tables 1 to 4.

[0098] Table 1 Test results of adding shallow feature extraction module

[0099] SEM OA IOU KAPPA No SEM 0.759 0.336 0.535 OPT+SEM, SAR+SEM 0.575 0.286 0.319 OPT+SEM 0.627 0.296 0.398 SAR+SEM 0.777 0.300 0.509

[0100] Table 2 Results of different evaluation indicators after adding shallow feature extraction module

[0101] SEM Information Entropy Standard Deviation Spatial frequency Peak signal-to-noise ratio Mean square error No SEM 11.21 32.62 21.68 5.98 1.48 OPT+SEM, SAR+SEM 11.40 24.78 11.67 5.70 4.10 OPT+SEM 11.21 26.69 13.67 7.13 2.99 SAR+SEM 11.43 34.02 21.94 6.03 1.43

[0102] Table 3 Detection accuracy of different categories with shallow feature extraction module added

[0103] OA forest field Water architecture the way background No SEM 0.1375 0.8351 0.6278 0.6630 0.5966 0.0069 OPT+SEM, SAR+SEM 0.3756 0.6373 0.3364 0.5769 0.6443 0.0225 OPT+SEM 0.1990 0.6614 0.5631 0.6079 0.6270 0.1827 SAR+SEM 0.0147 0.9070 0.6631 0.4126 0.1984 0.0549

[0104] Table 4 KAPPA values ​​of different categories with shallow feature extraction module added

[0105]

[0106]

[0107] The downsampling layer includes a convolutional layer S3, a convolutional layer S4 and a convolutional layer S5 which are connected in sequence;

[0108] The upsampling layer includes a convolutional layer S6 and a convolutional layer S7 connected in sequence;

[0109] SAR images have high resolution, strong penetration, and clear contours. The SAR encoder inputs the SAR image through the input layer S1, performs feature scaling, and extracts the contour information of the SAR image, that is, obtains the SAR feature. The image size input to the SAR encoder is 320*320*32. First, it is normalized through the normalization layer S2, and then downsampled through the downsampling layer. The normalized image size is convolved to 160, 80, and 40, and the number of channels is convolved to 32, 64, and 128. Then, it is processed by the shallow feature module to obtain 32-channel SAR shallow features. Finally, it is upsampled through the upsampling layer to obtain 320*320*64 or 320*320*40 SAR features, and outputs the SAR encoder. The SAR encoder changes the image size through continuous feature scaling, but the contour information remains unchanged, and finally realizes the extraction of contour features to obtain SAR features.

[0110] The SAR encoder includes an input layer S1 for inputting a SAR image, a normalization layer S2 for performing channel normalization processing on the SAR image input by the input layer S1, a downsampling layer for downsampling the normalization layer S2, a shallow feature extraction module for extracting SAR shallow features from shallow features obtained from the downsampling layer, suppressing SAR image noise interference while compensating for occluded information of the optical image, and an upsampling layer for upsampling the SAR shallow features to obtain SAR features.

[0111] The semantic feature extraction module includes a 3-channel input layer O1, which is convolved to obtain a 32-channel normalized layer O2, the output of the normalized layer O2 is convolved into a 16-channel feature block A, and the feature block A is sequentially convolved, normalized and mixed activation function operations are performed to obtain a convolution layer O3 of feature block 1, feature block 1 and feature block A are added to obtain feature block B, and feature block B is sequentially convolved, normalized and mixed activation function operations are performed to obtain a convolution layer O4 of feature block 2, feature block 2, feature block 1 and feature block A are added to obtain feature block C, and feature block C is sequentially convolved, normalized and mixed activation function operations are performed to obtain a convolution layer O4 of feature block 2. The sh activation function operation obtains the convolution layer O5 of feature block 3, and the feature block 3, feature block 2, feature block 1 and feature block A are added to obtain the convolution layer O6 of the optical feature; among them, when feature block 1 is a feature block with 16 channels, feature block B is a feature block with 32 channels, feature block 2 is a feature block with 16 channels, feature block C is a feature block with 48 channels, feature block 3 is a feature block with 16 channels, and the optical feature is a feature block with 64 channels; as shown in Table 5, the optical image has good texture and detail information, and semantic features such as color spectrum texture are continuously extracted through convolution and dense link operations. The image size of the input optical encoder is 320*320*32, and the size of the optical image is always 320*320. Through the accumulation of the number of channels, the first accumulation is 16, the second accumulation is 32, and the third accumulation is 48. After each accumulation, the number of channels is adjusted to 16 through convolution, and finally the fourth accumulation channel number is 64, that is, the optical encoder outputs an optical feature of size 320*320*64, which contains optically rich texture details and spectral features.

[0112] When feature block 1 is an 8-channel feature block, feature block B is a 24-channel feature block, feature block 2 is an 8-channel feature block, feature block C is a 32-channel feature block, feature block 3 is an 8-channel feature block, and the optical feature is a 40-channel feature block.

[0113] Table 5 Network parameters of each layer in the dual-coding fusion classification network

[0114]

[0115]

[0116] The decoder includes a fusion feature module F1 for fusing SAR features and optical features, a fusion image module F2 for sequentially performing convolution, ReLU activation function, and normalization operations on the fused features to obtain a fused image, and a classification module F3 for performing deconvolution processing and decoding on the fused image to obtain the result of intelligent classification of ground objects. In this embodiment, the respective feature images output by the SAR encoder and the optical encoder are fused, that is, the SAR features and the optical features are input into the decoder, and the fusion features of 320*320*128 or 320*320*80 are obtained by fusion. After convolution, ReLU activation function, and normalization operations, the fusion image is obtained, and finally the result of intelligent classification after fusion is obtained by deconvolution processing and decoding, so as to realize an end-to-end fusion classification of ground objects.

[0117] To supplement the fusion classification problem, a new loss function is used to regulate the information loss, pixel loss, and noise impact in each fusion process, and to adjust the training and intelligent classification. That is, a new loss function consisting of information loss, pixel loss, and noise loss is constructed to train and evaluate the dual-encoding fusion classification network. Specifically:

[0118] L total =L I +aL P +βL N

[0119] Among them, L I It represents the information loss between the fused image and the true label image, L P It refers to the pixel loss of the SAR features output by the SAR encoder and the optical features and fusion features output by the optical encoder. N It refers to the noise loss of the input and output characteristics of the SAR encoder and the optical encoder. α and β represent two tuning parameters, which are 1 or 5 respectively after adaptive adjustment;

[0120] Information loss refers to the information entropy between the fused image and the true label image. The amount of information is inversely proportional to the probability of the information occurring. If the probability of an event occurring is A(x), the amount of information I(x) is expressed as:

[0121] I(x)=-log(A(x))

[0122] The information entropy H(X) of all information is expressed as:

[0123]

[0124] If there are two independent probability distributions P(x) and q(x) for the same random variable, p(x i ) represents the true distribution of the i-th SAR image or optical image, q(xi ) represents the predicted distribution of the dual encoding fusion classification network, x i represents the i-th SAR image or optical image, p(x i ) and q(x i ) The difference between the two distributions is expressed as the relative entropy D KL To express:

[0125]

[0126] Breaking down the relative entropy formula:

[0127]

[0128] The former H(p(x)) is the information entropy, the latter That is the cross entropy, which is equal to the sum of relative entropy and information entropy. It can measure the cross entropy of the difference between two different probability distributions in the same random variable. In order to retain the amount of information of the fused image, the cross entropy is finally used as the information loss, which is expressed as:

[0129]

[0130] Cross entropy can measure the difference between two different probability distributions in the same random variable, which is expressed as the difference between the true probability distribution and the predicted probability distribution in machine learning. In deep learning networks, information entropy is a constant. The smaller the relative entropy, the better the prediction effect. However, cross entropy is relatively simpler to calculate, so cross entropy is often used as the loss function for classification problems. In the proposed method, the information cross entropy between the weighted classification result of the fused image and the true value is calculated. The cross entropy loss is used to continuously improve and optimize the information of the fused image, effectively retaining the information content of the fused image.

[0131] Pixel loss First determine the maximum pixel value of the SAR feature output by the SAR encoder and the optical feature output by the optical encoder, and then calculate the L1 norm loss of the pixel value of the fused feature after fusion and the maximum pixel value of the SAR feature and optical feature before fusion:

[0132]

[0133] Where H and W represent the height and width of the SAR image and the optical image, respectively. f represents the fusion feature, I SAR ,I OPT Represent the SAR characteristics and optical characteristics output by the SAR encoder and the optical encoder;

[0134] Noise loss:

[0135]

[0136] in, It represents the Sobel gradient operator, which has a better processing effect on images with more grayscale gradient noise and is more accurate in edge positioning. The input and output eigenvalues ​​of the SAR encoder and the optical encoder are convolved with the Sobel operator and then the L1 norm loss is calculated.

[0137] In addition, to address the sample imbalance problem, the sample size is counted, and each category is weighted using the proportion of each sample. The category with a large number of samples has a small weight, and the category with a small number of samples has a large weight. The statistically calculated weight value is weighted into the information loss, which effectively alleviates the sample imbalance problem. Table 6 shows the specific pixel statistics of each category and the corresponding penalty weight coefficient.

[0138] Table 6 Number of pixels and penalty weights for each category

[0139] Serial number category Pixel Value Weight 0 forest 431486685 0.0252 1 field 396141451 0.0275 2 Water 163945591 0.0664 3 architecture 120265056 0.0904 4 the way 10878261 1.0000 5 background 26210956 0.4150

[0140] The experiment was still completed on the WHU-OPT-SAR dataset. The pixel loss was aimed at the standard deviation problem, the noise loss was aimed at the peak signal-to-noise ratio problem, and the information loss was added to improve the standard deviation and peak signal-to-noise ratio while retaining the amount of fusion information. At the same time, several existing image fusion loss functions that were widely used were studied and added to the proposed dual-encoding fusion classification network. The three latest classic loss functions in the new image fusion were added to the proposed dual-encoding fusion classification network and compared with the proposed new loss function. The classification results are shown in Figure 2. Figure 7 As shown in Tables 7 to 10, the experimental results prove that the proposed new loss function has better effect.

[0141] Table 7 Test results of various loss functions

[0142]

[0143]

[0144] Table 8 Results of different evaluation indicators for various loss functions

[0145] Loss Function Information Entropy Standard Deviation Spatial frequency Peak signal-to-noise ratio Mean square error <![CDATA[L I ]]> 11.43 34.02 21.94 6.03 1.43 <![CDATA[L R L FD ]]> 11.39 28.83 14.52 5.59 1.39 <![CDATA[L T L S ]]> 11.38 29.64 18.25 6.11 2.36 <![CDATA[L I L P L N ]]> 11.51 29.47 16.99 6.21 1.37

[0146] Table 9 Test accuracy of different categories

[0147] OA forest field Water architecture the way background <![CDATA[L I ]]> 0.0147 0.9070 0.6631 0.4126 0.1984 0.0549 <![CDATA[L R L FD ]]> 0.0783 0.8788 0.5054 0.5804 0.1526 0.0082 <![CDATA[L T L S ]]> 0.1188 0.7081 0.5749 0.3652 0.2435 0.2993 <![CDATA[L I L P L N ]]> 0.1988 0.9266 0.6052 0.3972 0.4033 0.0674

[0148] Table 10 KAPPA values ​​of different categories

[0149]

[0150]

[0151] After training with the training set, use the validation set for verification and the test set for testing, specifically:

[0152] Qualitative analysis compares the images output by the end-to-end classification results with the real classification results generated by the real label files, and visual discriminant analysis is performed at the same time.

[0153] Quantitative analysis uses three groups of indicators to judge the fusion effect.

[0154] The first group is the evaluation of model fusion effect, including overall accuracy OA: the most important evaluation indicator refers to the proportion of correctly classified samples in the total number of samples; intersection over union IoU: the ratio of the intersection area to the predicted and true combined area, which is the main indicator for evaluating model accuracy. The larger the value, the higher the classification accuracy; KAPPA coefficient: a consistency test indicator used to measure whether the classification result is consistent with the actual classification result; floating point operations per second F lops: used to measure the computational complexity or amount of computation of the neural network model to evaluate the computational efficiency and inference speed of the model; model parameter Params: refers to the adjustable variables in the model that need to be learned through training, which determines the fitting ability and performance of the model.

[0155] The second group is the evaluation of image fusion effect, including information entropy EN: a measure of the amount of information, the larger the value, the more information the image contains; standard deviation SD: the degree of aggregation of pixels around the mean, the larger the value, the higher the image quality and the clearer the image; spatial frequency SF: reflects the rate of change of image grayscale, the larger the value, the higher the image quality; peak signal-to-noise ratio PSNR: used to measure the ratio between effective information and noise in the image, the larger the value, the smaller the noise of the fused image and the higher the fusion quality; mean square error MSE: used to calculate the average of the sum of squares of the true value and the predicted value error, which is very sensitive to fusion abnormal values. The smaller the value, the better the fusion effect.

[0156] The third group is the intelligent classification effect after fusion, including the average accuracy OA and KAPPA coefficient of each of the six categories.

[0157] A comparative experiment was conducted on the WHU-OPT-SAR dataset using several existing popular image fusion networks. Figure 8 Tables 11 to 14 are the classification results.

[0158] Table 11 Test results of six network structures

[0159]

[0160]

[0161] Table 12 Results of different evaluation indicators for six network structures

[0162] Model Information Entropy Standard Deviation Spatial frequency Peak signal-to-noise ratio Mean square error Densefuse 11.22 32.77 21.93 5.94 1.44 DeFusion 11.18 32.23 16.01 6.97 1.56 DIDFuse 11.27 31.52 27.90 7.29 2.89 PSFN 11.16 30.70 19.09 5.79 2.61 MCANet 11.33 28.44 14.68 6.02 1.58 Ours 11.43 34.02 21.94 6.03 1.43

[0163] Table 13 Detection accuracy of different categories of six network structures

[0164] OA forest field Water architecture the way background Densefuse 0.1301 0.8431 0.6346 0.6480 0.5627 0.0017 DeFusion 0.0288 0.7890 0.7286 0.6952 0.9048 0.0211 DIDFuse 0.1341 0.0660 0.7265 0.6801 0.4684 0.1931 PSFN 0.1970 0.7130 0.6428 0.7654 0.6663 0.0380 MCANet 0.0775 0.8481 0.5433 0.5875 0.7941 0.1110 Ours 0.2147 0.9070 0.6631 0.4126 0.1984 0.0549

[0165] Table 14 KAPPA values ​​of different categories of six network structures

[0166]

[0167]

[0168] It can be seen from the above data that the present invention is overall superior to several popular image fusion networks.

[0169] S3. Use the trained dual-coding fusion classification network to fuse SAR images and optical images, and perform end-to-end intelligent classification of ground objects.

Claims

1. A method for SAR and optical image fusion and intelligent classification, characterized in that: The steps include: S1, preprocess the source data set; S2, constructing a dual-encoding fusion classification network including two independent encoders, and training it through the source data set obtained by preprocessing to obtain a trained dual-encoding fusion classification network; S3, using the trained dual-encoding fusion classification network to fuse SAR images and optical images, and perform end-to-end intelligent classification of ground objects; The specific steps of step S1 are: S1.

1. Classify the label files in the source dataset into six categories: forest, field, water, building, road and background. The source dataset includes at least 100 SAR images, optical images and real label images of 5556×3704 size; S1.

2. Use the linear interpolation method to crop the SAR images, optical images, and true label images in the classified source dataset into at least 18,700 images of 320*320 size; S1.

3. Divide the trimmed data set into training set, validation set, and test set in a random distribution of 6:2:2, with a training set of at least 11,220 images, a validation set of at least 3,740 images, and a test set of at least 3,740 images. The dual-encoding fusion classification network includes a SAR encoder for extracting SAR features from SAR images, an optical encoder for extracting optical features from optical images, and a decoder for fusing and intelligently classifying SAR features and optical features; The SAR encoder includes an input layer S1 for inputting a SAR image, a normalization layer S2 for performing channel normalization processing on the SAR image input by the input layer S1, a downsampling layer for downsampling the normalization layer S2, a shallow feature extraction module for performing SAR shallow feature extraction on shallow features obtained by the downsampling layer, and an upsampling layer for upsampling the SAR shallow features to obtain SAR features; The optical encoder includes a semantic feature extraction module for extracting optical features from the optical image; The specific implementation steps of the shallow feature extraction module are: For shallow features f SAR ∈R B×C×H×W Synchronous convolution extracts low-resolution feature maps D1 and D2 of shallow features; The low-resolution feature map D1 is reconstructed and transposed, and then point-multiplied with the low-resolution feature map D2 to obtain the combined feature map of the SAR image. The low-resolution feature map D1 is reconstructed into Then transpose it into R B×Q×C ', the low-resolution feature map D2 is reconstructed into i represents the i-th SAR image, n represents the n-th SAR image, and C' represents the number of new channels after convolution, which is reduced by 2 based on the original channels. 3 , B represents the batch size, C represents the number of channels, and R represents a set of real numbers; Then the combined feature map is processed by exponential operation, summation and normalization in sequence to obtain the feature weight of the SAR image; The feature weight formula is: in, represents the feature weight of the i-th SAR image in the SAR image, represents the low-resolution feature map D1 of the i-th SAR image, represents the low-resolution feature map D2 of the i-th SAR image, The low-resolution feature map D1 represents the j-th pixel of the i-th SAR image. represents the low-resolution feature map D2 of the j-th pixel of the i-th SAR image, T represents the transposition operation, H and W represent the height and width respectively, Q = H × W, exp represents the exponential function with base e; Then flatten the shallow feature space into R B×C×Q The SAR shallow features are obtained by weighted multiplication with the result of transposing the feature weights of each pixel in the SAR image; The formula for SAR shallow features is: in, represents the SAR shallow features of the i-th image in the SAR image, and γ is a self-learning parameter that can learn and adjust itself. represents the feature weight of the jth pixel of the i-th SAR image, Represents the shallow features of the jth pixel in the i-th SAR image.

2. The method of SAR and optical image fusion and intelligent classification according to claim 1, characterized in that: The downsampling layer includes a convolutional layer S3, a convolutional layer S4 and a convolutional layer S5 which are connected in sequence; The upsampling layer includes a convolutional layer S6 and a convolutional layer S7 connected in sequence; The semantic feature extraction module includes a 3-channel input layer O1, which is convolved to obtain a 32-channel normalized layer O2, and the output of the normalized layer O2 is convolved into a 16-channel feature block A, and the feature block A is sequentially convolved, normalized and mish activation function operated to obtain a convolution layer O3 of feature block 1, feature block 1 and feature block A are added to obtain feature block B, and feature block B is sequentially convolved, normalized and mish activation function operated to obtain a convolution layer O4 of feature block 2, feature block 2, feature block 1 and feature block A are added to obtain feature block C, and feature block C is sequentially convolved, normalized and mish activation function operated to obtain a convolution layer O5 of feature block 3, and feature block 3, feature block 2, feature block 1 and feature block A are added to obtain a convolution layer O6 of optical features; The decoder includes a fusion feature module F1 for fusing SAR features and optical features, a fusion image module F2 for performing convolution, ReLU activation function and normalization operations on the fused features to obtain a fused image, and a classification module F3 for performing deconvolution processing and decoding on the fused image to obtain an intelligent classification result of the ground object.

3. A method for SAR and optical image fusion and intelligent classification according to any one of claims 1 to 2, characterized in that: Construct a new loss function consisting of information loss, pixel loss, and noise loss to train and evaluate the dual encoding fusion classification network. Specifically: L total =L I +aL P +βL N Among them, L I It represents the information loss between the fused image and the true label image, L P It refers to the pixel loss of the SAR features output by the SAR encoder and the optical features and fusion features output by the optical encoder. N It refers to the noise loss of the input and output characteristics of the SAR encoder and the optical encoder. α and β represent two tuning parameters, which are 1 or 5 respectively after adaptive adjustment; Information loss refers to the information entropy between the fused image and the true label image. The amount of information is inversely proportional to the probability of the information occurring. If the probability of an event occurring is A(x), the amount of information I(x) is expressed as: I(x)=-log(A(x)) The information entropy H(X) of all information is expressed as: If there are two independent probability distributions P(x) and q(x) for the same random variable, p(x i ) represents the true distribution of the i-th SAR image or optical image, q(x i ) represents the predicted distribution of the dual encoding fusion classification network, x i represents the i-th SAR image or optical image, p(x i ) and q(x i ) The difference between the two distributions is expressed as the relative entropy D KL To express: Breaking down the relative entropy formula: The former H(p(x)) is the information entropy, the latter That is the cross entropy, which is equal to the sum of relative entropy and information entropy. It can measure the cross entropy of the difference between two different probability distributions in the same random variable. In order to retain the amount of information of the fused image, the cross entropy is finally used as the information loss, which is expressed as: Pixel loss First determine the maximum pixel value of the SAR feature output by the SAR encoder and the optical feature output by the optical encoder, and then calculate the L1 norm loss of the pixel value of the fused feature after fusion and the maximum pixel value of the SAR feature and optical feature before fusion: Where H and W represent the height and width of the SAR image and the optical image, respectively. f represents the fusion feature, I SAR ,I OPT Represent the SAR characteristics and optical characteristics output by the SAR encoder and the optical encoder; Noise loss: in, Represents the Sobel gradient operator.

4. A system for SAR and optical image fusion and intelligent classification, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: The computer program includes the method according to any one of claims 1 to 3, and the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program comprises the method according to any one of claims 1 to 3, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer program product, comprising a computer program, characterized in that: The computer program comprises the method according to any one of claims 1 to 3, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images