A multimodal remote sensing image feature fusion method based on frequency strategy

Abstract

The present invention discloses a multi-modal remote sensing image feature fusion method based on frequency strategy, comprising the following steps: data preprocessing: using bilinear interpolation upsampling for low-resolution SAR images; network construction: using corresponding encoders according to different modalities; low-frequency feature sharing and high-frequency feature image reconstruction to obtain a reconstructed image, and unsupervised training is performed based on the reconstructed image, step S4, generating a fusion image of high-resolution RGB images, SAR remote sensing images, and near-infrared remote sensing images. The method of the present invention uses high-resolution image data from different satellites, and extracts features from SAR, RGB, and near-infrared images through a private encoder. The feature information of different frequencies is extracted by utilizing the characteristics of each image. The multi-modal high-frequency features are added element by element, and the local detail information of each image is fully retained, which is conducive to improving the quality of the fused image.

Description

Technical Field

[0001] The invention belongs to the field of satellite remote sensing technology, and specifically relates to a multi-modal remote sensing image feature fusion method based on frequency strategy. Background Art

[0002] Multimodal remote sensing images refer to different types of remote sensing images obtained using multiple sensors, which are integrated to obtain more comprehensive and richer information about objects. The main types of data included in multimodal remote sensing images include optical images, radar images, infrared images, etc. Optical images mainly use visible light bands to obtain surface information, radar images use microwave bands to obtain clear images even under complex weather conditions, and infrared images can capture the heat distribution of objects.

[0003] Multimodal data has many advantages over single-modal data. Multimodal data fusion can make up for the limitations of single sensor data to maintain the advantages of different modalities. However, there are some difficulties in realizing multimodal remote sensing data fusion. Different types of remote sensing data have different characteristics and resolutions, and data obtained by different sensors have different noise characteristics. In addition, multimodal data fusion also needs to consider the complementarity and consistency between different data sources.

[0004] The existing technology includes a deep and shallow modal feature fusion method and a globally interactive cross-modal fusion method. Among them, the deep and shallow modal feature fusion method fuses both the shallow modal features and the deep modal features through a model to obtain a corresponding joint expression, thereby improving the classification accuracy of remote sensing images. Patent CN105512661A involves a multimodal feature fusion method based on the RBM model. The specific method is as follows: 1. Extract the features of the two modalities, 2. Input the features of each modality into the RBM model for fusion to obtain a joint expression of the features of each modality, and 3. Finally, estimate the type of each superpixel area based on the joint expression, thereby realizing the classification of remote sensing images. However, in patent CN105512661A, the shallow shared encoder cannot distinguish modality-specific features, and the internal working mechanism is difficult to control and explain, resulting in insufficient extraction of cross-modal features.

[0005] The global interactive cross-modal fusion method is based on the rich spectral information of hyperspectral images and the rich spatial information of multispectral images. It uses a multimodal fusion network to reconstruct high-resolution hyperspectral fusion images to achieve the fusion of hyperspectral data and multispectral data. Patent CN202410321562 involves a spectrally fidelity global interactive hyperspectral multispectral cross-modal fusion method. The specific method is as follows: 1. Synthesize hyperspectral and multispectral data sets of different resolutions, 2. Construct a spectrally fidelity global interactive hyperspectral multispectral cross-modal fusion model, 3. Train the cross-modal fusion model, 4. Obtain the real hyperspectral and multispectral remote sensing images to be fused, and 5. Obtain the fusion results of hyperspectral and multispectral remote sensing images. The present invention patent CN202410321562 fuses spectral data to solve the distortion and spatial distortion of reconstructed spectrum caused by ignoring the complementary content of relatively distant field bands. However, the fused data source is limited to multispectral remote sensing data and hyperspectral remote sensing data. Remote sensing data such as synthetic aperture radar and near-infrared imagery cannot participate in the fusion, and no attention is paid to the unequal contributions of different frequency domain features of remote sensing images during the fusion period. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal remote sensing image feature fusion method based on frequency strategy to solve the problem that the existing multimodal data fusion technology proposed in the background technology is insufficient in fusion utilization of different frequency features. In addition, the same encoder is usually used for feature extraction for data of different modalities, which has the problem of inconvenient extraction and poor feature extraction effect.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A multimodal remote sensing image feature fusion method based on frequency strategy includes the following steps:

[0009] Step S1, data preprocessing: use bilinear interpolation upsampling for low-resolution SAR images to perform resolution alignment;

[0010] Step S2, network construction: According to different usage modes, build corresponding encoders for feature extraction and obtain low-frequency feature sharing And the high-frequency features after adding multi-modal high-frequency features ;

[0011] Step S3: low-frequency feature sharing And high frequency features After the concat operation, the complete features are obtained ; Different modal and after fusion The images are sent to the decoder for deconvolution and resolution restoration respectively, and the reconstructed image and the fused image are obtained as the basis for loss calculation. Unsupervised training is performed based on the reconstructed images of different modalities and the fused images after fusion features. The loss function Loss is driven by correlation. During the training process, the network will learn the correlation between different frequency features. ;

[0012] Step S4: The trained fusion model is used to generate a fusion image of high-resolution RGB images, SAR remote sensing images, and near-infrared remote sensing images.

[0013] According to the above technical solution, in step S1, bilinear interpolation upsampling is used for the low-resolution SAR image to perform resolution alignment as follows:

[0014] First, set the resolution of the low-resolution SAR image to M*M and the resolution of the high-resolution image to N*N; calculate the upsampling factor of the resolution:

[0015]

[0016] Where factor is the upsampling factor of the resolution; N is the resolution of the high-resolution image; M is the resolution of the low-resolution SAR image;

[0017] The upsampling factor is used to determine the source pixel corresponding to the pixel position after scaling, and then the weighted average of the four neighboring pixel values ​​above, below, left and right of the position is performed to obtain the final target pixel value to achieve resolution alignment.

[0018] According to the above technical solution, in step S2, the modalities include SAR images, high-resolution images and near-infrared images.

[0019] According to the above technical scheme, for SAR images, a dual-branch high-low frequency encoder is used to process low-frequency global features and high-frequency local information, and share features between adjacent positions; for high-resolution images and near-infrared images, a high-frequency encoder is used for local detail feature extraction, and low-resolution SAR low-frequency global features are shared.

[0020] According to the above technical solution, the specific processing method is as follows: the input of high-resolution image and near-infrared image are represented as V and N respectively, and the input of pre-processed SAR image is represented as S; wherein, the global feature extraction is:

[0021]

[0022] In the formula, Represents a high-frequency encoder for detailed feature extraction; Represents a low frequency encoder; They represent the features extracted by the encoder respectively.

[0023] According to the above technical solution, the extracted multimodal high-frequency features are added element by element to obtain Represents the high-frequency features after adding multi-modal high-frequency features, specifically:

[0024]

[0025] In the formula, Represents the high-frequency features after adding multi-modal high-frequency features, They represent the features extracted by the encoder respectively.

[0026] According to the above technical solution, the low-frequency feature sharing extracted from the SAR image is expressed as , specifically:

[0027]

[0028] In the formula, represents low-frequency feature sharing, Represents the features extracted by the encoder in the SAR image;

[0029] According to the above technical solution, Perform concat operation, send to decoder, and calculate complete features Calculated by the following formula:

[0030]

[0031] In the formula, Represents the complete features, Represents the high-frequency features after adding multi-modal high-frequency features, Indicates low-frequency feature sharing, and concat is an extension method used to connect two or more features.

[0032] According to the above technical solution, in step S3, the correlation-driven loss function Loss is used, and during the training process, the network will learn the correlation between different frequency features. Specifically:

[0033]

[0034] The result is a similarity between [-1, 1]; the denominator is the similarity of low-frequency features , similarity = 1; the numerator is the high-frequency feature similarity of S (SAR), N (near infrared) and V (optical) ; Minimize the loss function, the numerator becomes smaller, so that the high-frequency feature encoder can extract different high-frequency local features.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention uses high-resolution image data from different satellites, and extracts features from SAR, RGB and near-infrared images through encoders. SAR images are not affected by clouds and have stable quality because of their low resolution and special imaging methods, so a dual-branch high-low frequency encoder is used. High-frequency encoders are used for high-resolution images and near-infrared images to extract important local detail information.

[0037] The present invention makes full use of the characteristics of each image to extract feature information of different frequencies. Then, the multimodal high-frequency features are added element by element to fully retain the local detail information of each image, which is conducive to improving the quality of the fused image. Then, the low-frequency features extracted from the SAR image are concat-operated to obtain complete features, thereby protecting the overall structure and semantic information of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a feature fusion flow chart of the present invention;

[0039] Figure 2 This is a schematic diagram of feature fusion of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Embodiment 1

[0042] like Figure 1 As shown, a multimodal remote sensing image feature fusion method based on frequency strategy includes the following steps:

[0043] Step S1, data preprocessing: using bilinear interpolation upsampling for low-resolution SAR images (SAR images with a resolution of 10 meters or more are low-resolution SAR images) to perform resolution alignment;

[0044] Step S2, network construction: According to different usage modes, build corresponding encoders for feature extraction and obtain low-frequency feature sharing And the high-frequency features after adding multi-modal high-frequency features ;

[0045] Step S3: low-frequency feature sharing And high frequency features After the concat operation, the complete features are obtained ; Different modal and after fusion The images are sent to the decoder for deconvolution and resolution restoration respectively, and the reconstructed image and the fused image are obtained as the basis for loss calculation. Unsupervised training is performed based on the reconstructed images of different modalities and the fused images after fusion features. The loss function Loss is driven by correlation. During the training process, the network will learn the correlation between different frequency features. ; Step S4, the trained fusion model is used to generate fusion images of high-resolution RGB images, SAR remote sensing images, and near-infrared remote sensing images (images with a resolution of 2 meters or less are high-resolution images).

[0046] The method of the present invention uses high-resolution image data from different satellites, and extracts features from SAR, RGB and near-infrared images through a private encoder. Because SAR images have lower resolution and special imaging methods, they are not affected by clouds and have stable quality. A dual-branch high- and low-frequency encoder is used, in which a residual network composed of hole convolution is used in the low-frequency encoder. Hollow convolution can be used to increase the receptive field of the network, thereby better capturing global information. High-frequency encoders are used for high-resolution images and near-infrared images to extract important local detail information. The characteristics of each image are fully utilized to extract feature information of different frequencies. Then the multi-modal high-frequency features are added element by element to fully retain the local detail information of each image, which is beneficial to improve the quality of the fused image. Then the low-frequency features extracted from the SAR image are concat-operated to obtain complete features, protecting the overall structure and semantic information of the image.

[0047] Through the method of the present invention, the proposal can better learn the unique frequency features from different modalities and the correlation between frequency features, thereby improving the quality of the fused image.

[0048] Embodiment 2

[0049] This embodiment is a further refinement of the first embodiment.

[0050] In step S1, bilinear interpolation upsampling is used for the SAR image with lower resolution, and the resolution alignment is performed as follows:

[0051] First, set the resolution of the lower-resolution SAR image to M*M and the resolution of the high-resolution image to N*N; calculate the resolution upsampling factor factor:

[0052]

[0053] The upsampling factor is used to determine the source pixel corresponding to the pixel position after scaling, and then the weighted average of the four neighboring pixel values ​​above, below, left and right of the position is performed to obtain the final target pixel value to achieve resolution alignment.

[0054] In step S2, the modalities include SAR images, high-resolution images, and near-infrared images.

[0055] For SAR images, a dual-branch high- and low-frequency encoder is used to process the low-frequency global features that change slowly in space and the high-frequency local information that changes quickly in space, and share the features between adjacent positions. For high-resolution images and near-infrared images, a high-frequency encoder is used to extract local detail features and share low-resolution SAR low-frequency global features.

[0056] The specific processing method is as follows: the input of high-resolution image and near-infrared image are represented as V and N respectively, and the pre-processed SAR image input is represented as S; among them, the global feature extraction is:

[0057]

[0058] In the formula, Represents a high-frequency encoder for detailed feature extraction; Represents a low frequency encoder; They represent the features extracted by the encoder respectively.

[0059] The extracted multimodal high-frequency features are added element by element to obtain Represents the high-frequency features after adding multi-modal high-frequency features, specifically:

[0060]

[0061] In the formula, Represents the high-frequency features after adding multi-modal high-frequency features, They represent the features extracted by the encoder respectively.

[0062] The low-frequency features extracted from SAR images are shared and expressed as , specifically:

[0063]

[0064] In the formula, represents low-frequency feature sharing, Represents the features extracted by the encoder in the SAR image;

[0065] right Perform concat operation, send to decoder, and calculate complete features The fused image is obtained by the following calculation:

[0066]

[0067] In the formula, Represents the complete features, Represents the high-frequency features after adding multi-modal high-frequency features, Indicates low-frequency feature sharing.

[0068] In step S3, the correlation-driven loss function Loss is used. During the training process, the network will learn the correlation between different frequency features. Specifically:

[0069]

[0070] The result is a similarity between [-1, 1]; the denominator is the similarity of low-frequency features , similarity = 1; the numerator is the high-frequency feature similarity of S (SAR), N (near infrared) and V (optical) ; Minimize the loss function, the numerator becomes smaller, so that the high-frequency feature encoder can extract different high-frequency local features.

[0071] This embodiment provides a specific implementation method.

[0072] Step 1, data preprocessing: The data used in the proposed experiment comes from WHU-OPT-SAR, covering 51448.56 square kilometers, collected in Hubei Province, with a resolution of 5 meters, and contains 100 pairs of 5556x3704 (pixel) images. The RGB high-resolution images in the data set come from the GF-1 satellite (2-meter resolution), the near-infrared images come from the GF-1 satellite (2-meter resolution), and the SAR images come from the GF-3 satellite (5-meter resolution). The low-resolution SAR images are upsampled using bilinear interpolation for resolution alignment.

[0073] Step 2, network construction: The proposal designs a private encoder for feature extraction according to different modalities. For SAR images, the proposal uses a private dual-branch high- and low-frequency encoder to process low-frequency global features that change slowly in space and high-frequency local information that changes quickly in space, respectively, and share features between adjacent positions. For high-resolution images and near-infrared images, the proposal uses a private high-frequency encoder for local detail feature extraction and shares low-resolution SAR low-frequency global features.

[0074] Specifically, the input of RGB high-resolution image and near-infrared image is represented as V and N respectively, and the preprocessed SAR image input is represented as S. The structure of the high-frequency encoding module of the three modalities is the same, which consists of a dense residual network. Represents the high-frequency encoder for detail feature extraction. The low-frequency encoding module of the SAR modality is composed of a residual network composed of dilated convolutions with a dilation rate of 2. Represents the low-frequency encoder, which performs global feature extraction.

[0075]

[0076] They represent the features extracted by the encoder respectively.

[0077]

[0078] The extracted multimodal high-frequency features are added element by element to obtain , Represents the high-frequency features after adding multi-modal high-frequency features.

[0079]

[0080] The low-frequency features extracted from SAR images are shared and expressed as After the concat operation, the complete features are obtained , sent to the decoder to get the fused image.

[0081]

[0082] In the formula, concat is an extension method used to connect two or more features.

[0083] Step 3, unsupervised training: Features F and L are reconstructed using separate decoders to obtain the reconstructed image as the basis for loss calculation. The proposal performs unsupervised training based on the reconstructed image and uses the correlation-driven loss function. During the training process, the network will learn the correlation between features of different frequencies.

[0084] Step 4, result input: The trained fusion model can be used to generate fusion images of high-resolution RGB images, SAR remote sensing images, and near-infrared remote sensing images.

[0085] like Figure 2 As shown in the figure, the fused image retains the advantages of each multimodal image while effectively improving the expressiveness of the overall edge information. By sharing low-frequency features and fusing high-frequency features, the generated image is richer in detail and can significantly distinguish different types of objects, such as water bodies, vegetation, and buildings. It can also provide higher-quality input data in various downstream tasks (such as target detection, object classification, and change detection), improving the accuracy and stability of the model.

[0086] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0087] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal remote sensing image feature fusion method based on frequency strategy, characterized by: The following steps are involved: Step S1, data preprocessing: use bilinear interpolation upsampling for low-resolution SAR images to perform resolution alignment; Step S2, network construction: According to different usage modes, build corresponding encoders for feature extraction and obtain low-frequency feature sharing And the high-frequency features after adding multi-modal high-frequency features ; In step S2, the modalities include SAR images, high-resolution images, and near-infrared images; for SAR images, a dual-branch high-low frequency encoder is used to process low-frequency global features and high-frequency local information, and features between adjacent positions are shared; for high-resolution images and near-infrared images, a high-frequency encoder is used for local detail feature extraction, and low-resolution SAR low-frequency global features are shared; Step S3: low-frequency feature sharing And high frequency features After the concat operation, the complete features are obtained ; Different modal and after fusion The images are sent to the decoder for deconvolution and resolution restoration respectively, and the reconstructed image and the fused image are obtained as the basis for loss calculation. Unsupervised training is performed based on the reconstructed images of different modalities and the fused images after fusion features. The loss function Loss is driven by correlation. During the training process, the network will learn the correlation between different frequency features. ; Different modal Refers to SAR image , high-resolution images and near-infrared imaging Through high frequency encoder Extracted features; Step S4: The trained fusion model is used to generate a fusion image of high-resolution RGB images, SAR remote sensing images, and near-infrared remote sensing images.

2. The multimodal remote sensing image feature fusion method based on frequency strategy according to claim 1, characterized in that: In step S1, bilinear interpolation upsampling is used for the low-resolution SAR image to perform resolution alignment as follows: First, set the resolution of the low-resolution SAR image to M*M and the resolution of the high-resolution image to N*N; calculate the upsampling factor of the resolution: Where factor is the upsampling factor of the resolution; N is the resolution of the high-resolution image; M is the resolution of the low-resolution SAR image; The upsampling factor is used to determine the source pixel corresponding to the pixel position after scaling, and then the weighted average of the four neighboring pixel values ​​above, below, left and right of the position is performed to obtain the final target pixel value to achieve resolution alignment.

3. The multimodal remote sensing image feature fusion method based on frequency strategy according to claim 2 is characterized in that: The specific processing method is as follows: the input of high-resolution image and near-infrared image are represented as V and N respectively, and the pre-processed SAR image input is represented as S; among them, the global feature extraction is: In the formula, Represents a high-frequency encoder for detailed feature extraction; Represents a low frequency encoder; They represent the features extracted by the encoder respectively.

4. A multimodal remote sensing image feature fusion method based on frequency strategy according to claim 3, characterized in that: The extracted multimodal high-frequency features are added element by element to obtain Represents the high-frequency features after adding multi-modal high-frequency features, specifically: In the formula, Represents the exclusive OR operation; Represents the high-frequency features after adding multi-modal high-frequency features, They represent the features extracted by the encoder respectively.

5. A multimodal remote sensing image feature fusion method based on frequency strategy according to claim 4, characterized in that: The low-frequency features extracted from SAR images are shared and expressed as , specifically: In the formula, Show complete features, Represents the high-frequency features after adding multi-modal high-frequency features, Indicates low-frequency feature sharing, and concat is an extension method used to connect two or more features.

6. The multimodal remote sensing image feature fusion method based on frequency strategy according to claim 5, characterized in that: right Perform concat operation, send to decoder, and calculate complete features Calculated by the following formula: In the formula, Represents the complete features, Represents the high-frequency features after adding multi-modal high-frequency features, Indicates low-frequency feature sharing, and concat is an extension method used to connect two or more features.

7. The multimodal remote sensing image feature fusion method based on frequency strategy according to claim 6 is characterized in that: In step S3, the correlation-driven loss function Loss is used. During the training process, the network will learn the correlation between different frequency features. Specifically: The result is a similarity between [-1, 1]; the denominator is the similarity of low-frequency features , similarity = 1; the numerator is the high-frequency feature similarity of S (SAR), N (near infrared) and V (optical) ; Minimize the loss function, the numerator becomes smaller, so that the high-frequency feature encoder can extract different high-frequency local features.

Images