Heterogeneous remote sensing image processing method based on three-dimensional mask autoencoder and visual saliency feature fusion

Abstract

The application discloses a kind of based on three-dimensional mask auto-encoder and visual saliency feature fusion heterogeneous remote sensing image processing method, belong to remote sensing image processing field.For the problem that existing heterogeneous remote sensing fusion method is not sufficient to multi-modal complementary feature, weak noise suppression ability and poor visual perception, the application first splices optical RGB image and SAR single-channel image into three-dimensional input tensor after registration, and is pre-trained by three-dimensional mask auto-encoder, forces model to learn cross-modal complementary information;Then hierarchical extraction main structure feature and detail texture feature, respectively using total variation constraint model and Gabor wavelet-KL divergence dynamic selection strategy are fused, suppress SAR speckle noise and retain texture details;Finally, through IHS inverse transformation restores color fusion image.The application can effectively utilize the complementary characteristics of optical and SAR, is superior to existing method in subjective visual quality and objective evaluation index, and is applicable to environmental monitoring, disaster assessment and other fields.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing. Background Technology

[0002] With the rapid development of remote sensing technology, synthetic aperture radar (SAR) and optical remote sensing imaging technology have been widely used in environmental monitoring, disaster assessment, agricultural yield estimation, and urban planning. Optical images, based on visible light or infrared radiation, can clearly reflect the spectral characteristics and color information of ground objects, but their imaging is limited by lighting and weather conditions. SAR images, based on microwave active imaging, have all-weather, all-day observation capabilities and can penetrate clouds and smoke, but are more sensitive to the geometric characteristics of ground objects.

[0003] Existing heterogeneous remote sensing image fusion methods have three main drawbacks: first, they rely too much on simple feature stacking and fail to fully consider the modal differences between optical and SAR images; second, they are insufficient in suppressing the inherent speckle noise of SAR images; and third, they lack effective visual perception optimization mechanisms, resulting in insufficient interpretability of the fusion results in practical applications. Summary of the Invention

[0004] This invention aims to address the problems of insufficient utilization of multimodal complementary features, weak noise suppression, and poor visual perception in existing heterogeneous remote sensing image fusion methods. It provides a heterogeneous remote sensing image processing method based on the fusion of three-dimensional mask autoencoder and visual saliency features.

[0005] The heterogeneous remote sensing image processing method based on 3D mask autoencoder and visual saliency feature fusion described in this application includes:

[0006] Step 1: Obtain the registered optical image and SAR image of the same region, and stitch the RGB three channels of the registered optical image and the single channel of the SAR image along the channel dimension to form a three-dimensional input tensor.

[0007] Step 2: Construct a 3D mask autoencoder and perform self-supervised pre-training on the 3D mask autoencoder using the 3D input tensor to obtain the pre-trained encoder.

[0008] Step 3: Use the pre-trained encoder to extract features from the registered optical image and SAR image described in Step 1 to obtain the main structure feature (MSF) and detail texture feature (DTF).

[0009] The main structural feature MSF is fused using a total variational constraint model to obtain the structural fusion result;

[0010] The detailed texture features (DTF) are decomposed into multi-directional and multi-scale components using Gabor wavelet transform to obtain multi-directional and multi-scale texture response coefficients. The texture response coefficients are then used to obtain the texture fusion result.

[0011] Step 4: Perform weighted fusion of the structure fusion result and the texture fusion result, and recover the RGB color information through IHS inverse transform to obtain the final fused image.

[0012] Furthermore, in this invention, step two, the process of performing self-supervised pre-training on the three-dimensional mask autoencoder using the three-dimensional input tensor to obtain the pre-trained encoder, is as follows:

[0013] The three-dimensional input tensor is divided into N image blocks, and random masking without replacement is performed simultaneously in the spatial and channel dimensions to obtain visible image blocks and mask markers.

[0014] The encoder of the three-dimensional mask autoencoder performs feature encoding on the visible image blocks to generate a latent representation matrix, and inserts the mask markers into the latent representation matrix to obtain a complete sequence of the same length as the total number N of the original image blocks;

[0015] The lightweight decoder of the 3D mask autoencoder receives the complete sequence, reconstructs the masked image region, and completes self-supervised pre-training.

[0016] Furthermore, in this invention, the mask ratio for performing random no-replacement masking operations simultaneously in the spatial and channel dimensions is 75%.

[0017] Furthermore, in this invention, in step three, the main structural feature (MSF) originates from the feature map output by the deep network of the encoder, and the detail texture feature (DTF) originates from the feature map output by the shallow network of the encoder; the deep network is the last K layers of the encoder, and the shallow network is the first K layers of the encoder, where K=4.

[0018] Furthermore, in this invention, in step three, the total variational constraint model is:

[0019] ;

[0020] in, For optimal structure fusion results, For the desired MSF fusion result, For regularization parameters, For gradient operators, This represents the L1 norm, used to enhance sparsity and edge preservation. The pixel saliency map is calculated from the encoder's self-attention weight matrix and is used to locate important pixels. The structural saliency map is obtained by normalizing the attention gradient using the Frobenius norm and is used to capture edge structure information.

[0021] ;

[0022] in, A pixel saliency map representing an optical image. A pixel saliency map representing a SAR image; Represents the main structural features of a SAR image; Represents the main structural features of an optical image;

[0023] ;

[0024] in, This represents the self-attention weight matrix of the encoder. This represents the spatial gradient of the attention weight matrix. This represents the Frobenius norm of the attention gradient matrix.

[0025] Furthermore, in this invention, in step three, the texture fusion result is:

[0026] ;

[0027] ; ;

[0028] Where P and Q are the normalized probability distribution vectors of the multi-directional, multi-scale texture response coefficients at each pixel location in the optical image and the normalized probability distribution vectors of the texture response coefficients at the corresponding location in the SAR image, respectively. It is a symmetric similarity measure at pixel position (m,n); and ∇ represents the texture response coefficients at pixel location (m,n) in the optical image and SAR image, respectively; ∇ is the gradient operator; For the blended detailed texture features, This represents the two-dimensional spatial pixel feature matrix (its size is determined by the image's height H and width W) corresponding to the entire remote sensing image currently being input or to be processed, and the mean SMV of all pixel locations. and and represent the normalized probability values ​​of the texture response coefficients on feature channel x in the texture feature response space and the normalized probability values ​​of the texture response coefficients on feature channel x in the SAR image, respectively; x represents the feature channel index in the texture feature response space. The KL divergence is used to measure the difference between distribution Q and distribution P. KL divergence is represented by a measure of the difference between distributions P and Q.

[0029] Furthermore, in this invention, the final fused image is:

[0030] ;

[0031] in, The IHS transformation matrix is... Its inverse matrix, Indicates the image tone. Indicates image saturation; , and These represent the color component values ​​of the red, green, and blue channels of the final fused image, respectively, while R, G, and B represent the color component values ​​of the red, green, and blue channels of the original optical image, respectively. This represents the final fusion intensity component.

[0032] Furthermore, in this invention, the image intensity is the fused intensity component. The value;

[0033] ;

[0034] ;

[0035] in, For detail enhancement factor, This represents the structural fusion result (spatial distribution) obtained after fusing the principal structural features (MSF) of an optical image using a total variational constraint model. Represents the dynamic fusion weights of the SAR image; This represents the blended detail texture features (DTF). This represents the structural fusion result obtained by fusing the main structural features (MSF) of a SAR image using a total variational constrained model. This represents the reconstruction error of the 3D mask autoencoder on the optical image mask region. This represents the reconstruction error of the 3D mask autoencoder in the masked region of the SAR image; This represents the dynamic fusion weights of the optical images.

[0036] This invention constructs a three-dimensional mask autoencoder and simultaneously performs mask pre-training in both spatial and channel dimensions, forcing the model to effectively learn cross-modal complementary information between optical and SAR images. Based on this, it extracts master structure features and detailed texture features layer by layer, fusing them using a total variational constraint model and a Gabor wavelet-KL divergence dynamic selection strategy, significantly improving the suppression of SAR speckle noise and the preservation of texture details. Furthermore, it introduces pixel saliency maps and structural saliency maps to guide the fusion process and adaptively allocates fusion weights based on reconstruction errors, enhancing robustness under degradation conditions such as cloud cover and noise interference. Compared with existing methods, this invention can more fully utilize multimodal complementary information, achieving significant advantages in both subjective visual quality and objective evaluation metrics (structural similarity improvement of over 0.94, peak signal-to-noise ratio improvement of 2.8 dB, and change detection accuracy improvement of 5.2%), demonstrating good practical application value. Attached Figure Description

[0037] Figure 1 This is a flowchart of the method described in this invention;

[0038] Figure 2 This is a diagram of the pre-training architecture of the 3D mask autoencoder (3D-MAE) of this invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0040] Specific Implementation Method 1: Referring to the specific description of this implementation method, the heterogeneous remote sensing image processing method based on the fusion of three-dimensional mask autoencoder and visual saliency features described in this implementation method includes:

[0041] Step 1: Obtain the registered optical image and SAR image of the same region, and stitch the RGB three channels of the registered optical image and the single channel of the SAR image along the channel dimension to form a three-dimensional input tensor.

[0042] Step 2: Construct a 3D mask autoencoder and perform self-supervised pre-training on the 3D mask autoencoder using the 3D input tensor to obtain the pre-trained encoder.

[0043] Step 3: Use the pre-trained encoder to extract features from the registered optical image and SAR image described in Step 1 to obtain the main structure feature (MSF) and detail texture feature (DTF).

[0044] The main structural feature MSF is fused using a total variational constraint model to obtain the structural fusion result;

[0045] The detailed texture features (DTF) are decomposed into multi-directional and multi-scale components using Gabor wavelet transform to obtain multi-directional and multi-scale texture response coefficients. The texture response coefficients are then used to obtain the texture fusion result.

[0046] Step 4: Perform weighted fusion of the structure fusion result and the texture fusion result, and recover the RGB color information through IHS inverse transform to obtain the final fused image.

[0047] Furthermore, in this invention, step two, the process of performing self-supervised pre-training on the three-dimensional mask autoencoder using the three-dimensional input tensor to obtain the pre-trained encoder, is as follows:

[0048] The three-dimensional input tensor is divided into N image blocks, and random masking without replacement is performed simultaneously in the spatial and channel dimensions to obtain visible image blocks and mask markers.

[0049] The encoder of the three-dimensional mask autoencoder performs feature encoding on the visible image blocks to generate a latent representation matrix, and inserts the mask markers into the latent representation matrix to obtain a complete sequence of the same length as the total number N of the original image blocks;

[0050] The lightweight decoder of the 3D mask autoencoder receives the complete sequence, reconstructs the masked image region, and completes self-supervised pre-training.

[0051] Furthermore, in this invention, the mask ratio for performing random no-replacement masking operations simultaneously in the spatial and channel dimensions is 75%.

[0052] Furthermore, in this invention, in step three, the main structural feature (MSF) originates from the feature map output by the deep network of the encoder, and the detail texture feature (DTF) originates from the feature map output by the shallow network of the encoder; the deep network is the last K layers of the encoder, and the shallow network is the first K layers of the encoder, where K=4.

[0053] Furthermore, in this invention, in step three, the total variational constraint model is:

[0054] ;

[0055] in, For optimal structure fusion results, For the desired MSF fusion result, For regularization parameters, For gradient operators, This represents the L1 norm, used to enhance sparsity and edge preservation. The pixel saliency map is calculated from the encoder's self-attention weight matrix and is used to locate important pixels. The structural saliency map is obtained by normalizing the attention gradient using the Frobenius norm and is used to capture edge structure information.

[0056] ;

[0057] in, A pixel saliency map representing an optical image. A pixel saliency map representing a SAR image; Represents the main structural features of a SAR image; Represents the main structural features of an optical image;

[0058] ;

[0059] in, This represents the self-attention weight matrix of the encoder. This represents the spatial gradient of the attention weight matrix. This represents the Frobenius norm of the attention gradient matrix.

[0060] Furthermore, in this invention, in step three, the texture fusion result is:

[0061] ;

[0062] ; ;

[0063] Where P and Q are the normalized probability distribution vectors of the multi-directional, multi-scale texture response coefficients at each pixel location in the optical image and the normalized probability distribution vectors of the texture response coefficients at the corresponding location in the SAR image, respectively. It is a symmetric similarity measure at pixel position (m,n); and ∇ represents the texture response coefficients at pixel location (m,n) in the optical image and SAR image, respectively; ∇ is the gradient operator; For the blended detailed texture features, This represents the two-dimensional spatial pixel feature matrix (its size is determined by the image's height H and width W) corresponding to the entire remote sensing image currently being input or to be processed, and the mean SMV of all pixel locations. and and represent the normalized probability values ​​of the texture response coefficients on feature channel x in the texture feature response space and the normalized probability values ​​of the texture response coefficients on feature channel x in the SAR image, respectively; x represents the feature channel index in the texture feature response space. The KL divergence is used to measure the difference between distribution Q and distribution P. It represents the divergence, which measures the difference between distribution P and distribution Q.

[0064] Furthermore, in this invention, the final fused image is:

[0065] ;

[0066] in, The IHS transformation matrix is... Its inverse matrix, Indicates the image tone. Indicates image saturation; , and These represent the color component values ​​of the red, green, and blue channels of the final fused image, respectively, while R, G, and B represent the color component values ​​of the red, green, and blue channels of the original optical image, respectively. This represents the final fusion intensity component.

[0067] Furthermore, in this invention, the image intensity is the fused intensity component. The value;

[0068] ;

[0069] ;

[0070] in, For detail enhancement factor, This represents the structural fusion result (spatial distribution) obtained after fusing the principal structural features (MSF) of an optical image using a total variational constraint model. Represents the dynamic fusion weights of the SAR image; This represents the blended detail texture features (DTF). This represents the structural fusion result obtained by fusing the main structural features (MSF) of a SAR image using a total variational constrained model. This represents the reconstruction error of the 3D mask autoencoder on the optical image mask region. This represents the reconstruction error of the 3D mask autoencoder in the masked region of the SAR image; This represents the dynamic fusion weights of the optical images.

[0071] The heterogeneous remote sensing image processing method based on the fusion of three-dimensional mask autoencoder and visual saliency features described in this invention specifically includes the following steps.

[0072] I. Construction of 3D Input Tensors;

[0073] Acquire registered optical images (RGB three-channel) and SAR images (single-channel) of the same region, and stitch them together along the channel dimension to form a three-dimensional input tensor. ,in (RGB three-channel + SAR single-channel). Image height, This represents the image width.

[0074] 2. Pre-training of 3D masked autoencoders;

[0075] The overall architecture of the 3D mask autoencoder (3D-MAE) is as follows: Figure 1 As shown.

[0076] Input tensor Divided into The size is Image patches. In each training sample, a random sampling strategy without replacement is used to select 75% of the patches for masking. Unlike existing MAEs that only mask in the spatial dimension, this invention performs masking simultaneously in both the spatial and channel dimensions, that is, it independently masks the optical and SAR channels at the same spatial location, forcing the model to obtain complementary information from the other modality.

[0077] The encoder employs a Transformer architecture, operating only on the visible, unmasked patches (approximately 25%). Each visible patch is flattened into a one-dimensional vector and mapped through a linear projection layer. The encoder constructs a 3D feature space and adds positional encoding. It extracts features through a multi-layer multi-head self-attention mechanism and a feedforward neural network, outputting a latent representation. ,in This represents the number of visible small blocks.

[0078] Insert the mask token ([MASK] token) into the latent representation at the position corresponding to the masked block to form the complete sequence. .

[0079] The lightweight decoder receives the complete sequence. The decoder decodes and reconstructs the pixel values ​​of each small block layer by layer. The decoder is only used during the pre-training phase, and its output is the target reconstructed image.

[0080] The pre-trained loss function is computed only on small patches of the mask, using mean squared error (MSE) loss:

[0081] (1);

[0082] in For the set of indices of the mask blocks, The number of mask blocks, These are the original pixel values. To reconstruct pixel values.

[0083] After pre-training is complete, the decoder is discarded, and only the encoder is retained for downstream feature extraction.

[0084] III. Hierarchical Feature Decomposition;

[0085] Feature extraction is performed on complete optical and SAR images using a pre-trained encoder. For example... Figure 2 As shown, the encoder is composed of The feature maps are composed of stacked Transformer blocks, and feature maps at different depths have different semantic levels.

[0086] (1) Master Structural Feature Extraction (MSF): Deep Encoder (Post-Encoder) The deep encoder layer has a large receptive field, capturing low-frequency global structural information of the image, including the macroscopic geometric features and overall spatial layout of the target. The feature map output from the deep encoder is subjected to Global Average Pooling (GAP) and then filtered by a Gaussian filter. Smoothing process yields the main structural features:

[0087] (2);

[0088] in The deep encoder outputs a feature map. For Gaussian kernel, This is the filtering scale.

[0089] (2) Detail Texture Feature (DTF) Extraction: Shallow Encoder (Pre-) The layer retains high spatial resolution, capturing high-frequency details of the image, including local textures, edges, and corners. It is combined with a Gabor wavelet filter bank. ( For direction parameters, For targeted enhancement using scale parameters, the Gabor filter's expression in the time domain is:

[0090] (3);

[0091] in , , For the filter direction, The radial center frequency is denoted as .

[0092] Noise threshold estimation using equivalent number of views (ENL) of SAR images Hard thresholding suppression is applied to the Gabor filter response to obtain detailed texture features:

[0093] (4);

[0094] in For threshold function, below The response was set to zero.

[0095] IV. Main Structural Features (MSF) Fusion

[0096] A total variation (TV) constrained model is used to fuse the MSF. Pixel saliency map (PSM) and structural saliency map (SSM) are constructed as visual saliency guides.

[0097] Pixel saliency map From the encoder's self-attention weight matrix The important pixels are located by calculating the sum of attention weights.

[0098] (5);

[0099] in For the number of attention heads, This refers to the number of small blocks.

[0100] Structural salience map Capturing edge structure information through the normalized attention gradient Frobenius norm:

[0101] (6);

[0102] PSM of optical images PSM of SAR images Pixel combinations with high saliency values ​​are used to generate a saliency map with pixel constraints.

[0103] (7);

[0104] Structural salience map It is generated by weighting the relative local gradient values ​​and pixel intensity values ​​at multiple scales.

[0105] The total variational optimization model for MSF fusion is defined as follows:

[0106] (8);

[0107] in For the desired MSF fusion result, For regularization parameters, The gradient operator is used. This optimization problem is solved using the Iteratively Reweighted Norm (IRN) algorithm.

[0108] V. Detail Texture Feature (DTF) Fusion

[0109] Gabor wavelet transform was applied to the DTF to obtain multi-directional, multi-scale texture responses. The optical DTF after Gabor filtering was then processed. and SARDTF Normalize to form a probability distribution vector and Calculate the KL divergence between the two:

[0110] (9);

[0111] Since the KL divergence does not satisfy symmetry, a symmetric similarity measure (SMV) is defined:

[0112] (10);

[0113] Calculate the mean of the SMV across the entire graph. Based on this threshold, dynamic feature selection is performed, and the fused DTF is:

[0114] (11);

[0115] VI. Feature-weighted fusion and color reconstruction

[0116] A dynamic weighting strategy is used to optimize the MSF fusion results. Fusion results with DTF Weighted fusion is performed. The dynamic weights are adaptively determined by the encoder's reconstruction error in the masked region:

[0117] For optical image reconstruction error and SAR image reconstruction error The fusion weights are:

[0118] (12);

[0119] When optical images are obscured by clouds or fog Larger When the value approaches 1, the fusion result is mainly contributed by the SAR image; when the SAR image has severe noise, Larger When the value approaches 1, the fusion result is mainly contributed by the optical image.

[0120] The final intensity component fusion result is as follows:

[0121] (13);

[0122] in Enhancement factor for details.

[0123] Finally, the RGB color information is recovered using the fast inverse IHS transform. The intensity of the original optical image's RGB three channels is obtained through a forward IHS transform. ,tone saturation Components, using the fused intensity components Replace the original The components are then subjected to an inverse transform to obtain the final fused image.

[0124] (14);

[0125] in The IHS transformation matrix is... It is its inverse matrix. Specific Implementation

[0126] Step 1: Obtain the registered optical image and SAR image of the same region. The optical image is RGB three-channel, and the SAR image is single-channel. The two images are stitched together along the channel dimension to form... The three-dimensional input tensor. Specifically, the three channels of the optical image and the one channel of the SAR image can be stacked to form a four-channel input, enabling the model to process two modalities simultaneously.

[0127] Step 2: Construct a 3D mask autoencoder (3D-MAE). Divide the input image into... Non-overlapping patches are obtained. The spatial masking ratio is set to 0.75, and the channel masking ratio is also set to 0.5 (i.e., some channel information is randomly discarded). A random sampling strategy without replacement is used to obtain the visible patches (approximately 25%) after masking and the mask markers. Unlike standard MAE, the masking operation of this invention is performed simultaneously in both spatial and channel dimensions, with the aim of disrupting intermodal shortcut learning and forcing the model to utilize cross-modal complementary information.

[0128] Step 3: The encoder encodes features only from visible patches. The encoder consists of 12 stacked Transformer layers, each containing 12 self-attention heads and an MLP module. The visible patches are flattened and mapped to a 768-dimensional feature space via linear projection, and learnable positional encodings are added. The encoder outputs a latent representation. Then, mask markers are inserted into the latent representation at the positions corresponding to the masked blocks to form a complete sequence. .

[0129] Step 4: The lightweight decoder receives the complete sequence. The masked image region is reconstructed. The decoder consists of 8 Transformer layers, with an output dimension of... (Corresponding to 4 channels, number of pixels per small block). The pre-training loss function uses the MSE loss defined by formula (1), calculated only on the mask small blocks. Pre-training was performed on the YYX-OPT-SAR dataset for 400 epochs, with a batch size of 64 and a learning rate of 1e-3, accelerated in parallel using RTX3080×2 GPU dual cards. The final reconstruction loss reached 0.093, and the structural similarity index (SSIM) reached 0.939. After pre-training, the decoder was discarded, and only the encoder was retained.

[0130] Step 5: Extract features from the complete optical and SAR images using the pre-trained encoder. Extract principal structure features from the deep layers of the encoder (layers 9-12): Perform global average pooling on the deep feature maps to compress the spatial dimension to... Then through the Gaussian kernel ( ) Filtering and smoothing yields the MSF. Detail texture features are extracted from the shallow layers (layers 1-4) of the encoder: a multi-directional Gabor filter bank (8 directions, 4 scales, 32 filters in total) filters the shallow feature map, combined with the noise threshold estimated by the SAR image ENL. Hard threshold suppression is performed to obtain DTF.

[0131] Step 6: Fusion of MSF using a fully variational constrained model. Calculate PSM and SSM according to formulas (5) and (6) respectively. Solve the TV optimization problem defined by formula (8), where the regularization parameter... The IRN algorithm was used to iterate 20 times to obtain the structural fusion result. .

[0132] Step 7: Fuse the DTF. Calculate the KL divergence between the optical and SARDTFs using formula (9), and calculate the symmetric SMV using formula (10). Use the average SMV of the entire image. Using the threshold value, dynamic feature selection is performed according to formula (11) to obtain the texture fusion result. .

[0133] Step 8: Feature-weighted fusion and color reconstruction. Based on the encoder's reconstruction error in the mask region, calculate the dynamic weights according to formula (12). and Calculate the strength components according to formula (13). Among them, the detail enhancement coefficient Finally, perform the inverse IHS transformation according to formula (14) to restore the RGB three channels and obtain the final fused image.

[0134] Experiments were conducted on the QXS-SAROPT and WHU-OPT-SAR datasets to validate the method. Compared with pixel-level fusion, weighted fusion, high-pass filter fusion (HPF), Laplacian pyramid fusion (LP), and existing visual saliency fusion (VSF) methods, the method of this invention achieved the best results in detail preservation, speckle suppression, and color restoration. Quantitative evaluation metrics: SSIM reached above 0.94, peak signal-to-noise ratio (PSNR) improved by 2.8 dB, and change detection accuracy improved by 5.2%, fully validating the effectiveness of the invention.

[0135] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A heterogeneous remote sensing image processing method based on 3D mask autoencoder and visual saliency feature fusion, characterized in that, include: Step 1: Obtain the registered optical image and SAR image of the same region, and stitch the RGB three channels of the registered optical image and the single channel of the SAR image along the channel dimension to form a three-dimensional input tensor. Step 2: Construct a 3D mask autoencoder and perform self-supervised pre-training on the 3D mask autoencoder using the 3D input tensor to obtain the pre-trained encoder. Step 3: Use the pre-trained encoder to extract features from the registered optical image and SAR image described in Step 1 to obtain the main structure feature (MSF) and detail texture feature (DTF). The main structural feature MSF is fused using a total variational constraint model to obtain the structural fusion result; The detailed texture features (DTF) are decomposed into multi-directional and multi-scale components using Gabor wavelet transform to obtain multi-directional and multi-scale texture response coefficients. The texture response coefficients are then used to obtain the texture fusion result. Step 4: Perform weighted fusion of the structure fusion result and the texture fusion result, and recover the RGB color information through inverse IHS transform to obtain the final fused image.

2. The heterogeneous remote sensing image processing method based on three-dimensional mask autoencoder and visual saliency feature fusion according to claim 1, characterized in that, In step two, the process of performing self-supervised pre-training on the 3D mask autoencoder using the 3D input tensor to obtain the pre-trained encoder is as follows: The three-dimensional input tensor is divided into N image blocks, and random masking without replacement is performed simultaneously in the spatial and channel dimensions to obtain visible image blocks and mask markers. The encoder of the three-dimensional mask autoencoder performs feature encoding on the visible image blocks to generate a latent representation matrix, and inserts the mask markers into the latent representation matrix to obtain a complete sequence of the same length as the total number N of the original image blocks; The lightweight decoder of the 3D masked autoencoder receives the complete sequence, reconstructs the masked image region, completes self-supervised pre-training, and obtains the pre-trained encoder.

3. The heterogeneous remote sensing image processing method based on 3D mask autoencoder and visual saliency feature fusion according to claim 1, characterized in that, The mask ratio for performing random masking operations without replacement simultaneously in both the spatial and channel dimensions is 75%.

4. The heterogeneous remote sensing image processing method based on 3D mask autoencoder and visual saliency feature fusion according to claim 1, characterized in that, In step three, the main structural feature (MSF) originates from the feature map output by the deep network of the encoder, and the detail texture feature (DTF) originates from the feature map output by the shallow network of the encoder; the deep network is the last K layers of the encoder, and the shallow network is the first K layers of the encoder, where K=4.

5. The heterogeneous remote sensing image processing method based on 3D mask autoencoder and visual saliency feature fusion according to claim 1, characterized in that, In step three, the total variation constraint model is: ； in, For the optimal structure fusion result, For the desired MSF fusion result, For regularization parameters, For gradient operators, This represents the L1 norm, used to enhance sparsity and edge preservation. The pixel saliency map is calculated from the encoder's self-attention weight matrix. The structural saliency map is obtained by normalizing the attention gradient using the Frobenius norm, and is used to capture edge structure information. ； in, A pixel saliency map representing an optical image. A pixel saliency map representing a SAR image; Represents the main structural features of a SAR image; Represents the main structural features of an optical image; ； in, This represents the self-attention weight matrix of the encoder. This represents the spatial gradient of the attention weight matrix. This represents the Frobenius norm of the attention gradient matrix.

6. The heterogeneous remote sensing image processing method based on 3D mask autoencoder and visual saliency feature fusion according to claim 5, characterized in that, In step three, the texture blending result is as follows: ； ； ； Where P and Q are the normalized probability distribution vectors of the multi-directional, multi-scale texture response coefficients at each pixel location in the optical image and the normalized probability distribution vectors of the texture response coefficients at the corresponding location in the SAR image, respectively. It is a symmetric similarity measure at pixel position (m,n); and ∇ represents the texture response coefficients at pixel location (m,n) in the optical image and SAR image, respectively; ∇ is the gradient operator; For the blended detailed texture features, This represents the two-dimensional spatial pixel feature matrix (its size is determined by the image's height H and width W) corresponding to the entire remote sensing image currently being input or to be processed, and the mean SMV of all pixel locations. and and represent the normalized probability values ​​of the texture response coefficients on feature channel x in the texture feature response space and the normalized probability values ​​of the texture response coefficients on feature channel x in the SAR image, respectively; x represents the feature channel index in the texture feature response space. The KL divergence is used to measure the difference between distribution Q and distribution P. KL divergence is represented by a measure of the difference between distributions P and Q.

7. The heterogeneous remote sensing image processing method based on 3D mask autoencoder and visual saliency feature fusion according to claim 1, characterized in that, In step four, the final fused image is: ； in, The IHS transformation matrix is... Its inverse matrix, Indicates the image tone. Indicates image saturation; , and These represent the color component values ​​of the red, green, and blue channels in the final fused image, respectively. This represents the final fusion intensity component.

8. The heterogeneous remote sensing image processing method based on 3D mask autoencoder and visual saliency feature fusion according to claim 7, characterized in that, Image intensity is the fused intensity component. The value; ； ； in, For detail enhancement factor, This represents the structural fusion result obtained after fusing the principal structural features of an optical image using a total variational constraint model. Represents the dynamic fusion weights of the SAR image; This represents the detailed texture features after fusion; This represents the structural fusion result obtained after fusing the main structural features of a SAR image using a total variational constraint model. This represents the reconstruction error of the 3D mask autoencoder on the optical image mask region. This represents the reconstruction error of the 3D mask autoencoder in the masked region of the SAR image; This represents the dynamic fusion weights of the optical images.

Images