A method, medium and system for multispectral and SAR data fusion

Abstract

The application provides a multispectral and SAR data fusion method, medium and system, and belongs to the technical field of electric digital data processing.The application constructs a training set of low-resolution SAR images and degraded multispectral images, takes original multispectral images as a supervision label, and designs a deep network architecture containing feature extraction, feature fusion and image reconstruction modules. A tile sample training strategy is adopted to optimize network parameters, in the prediction stage, the spatially registered multispectral images and high-resolution SAR images are input into the trained network, and the fusion image with improved spatial resolution and complete spectral information is output. The method realizes adaptive fusion of heterogeneous data features by using an attention mechanism, maintains spectral integrity through a degradation-reconstruction supervision learning strategy, and effectively solves the technical problems of low spatial resolution of multispectral images and spectral information distortion in the SAR data fusion process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital data processing, and particularly relates to a multispectral and SAR data fusion method, medium and system. BACKGROUND

[0002] Remote sensing data fusion is a key technology in the field of earth observation, especially the fusion of multispectral images and synthetic aperture radar (SAR) data. Traditional fusion methods mainly include component replacement-based methods (such as IHS, PCA transformation), multi-resolution analysis-based methods (such as wavelet transform, pyramid decomposition), and statistical learning-based methods (such as sparse representation, dictionary learning). These methods are widely used in different application scenarios, such as land cover classification, change detection, and environmental monitoring.

[0003] However, traditional fusion methods have obvious limitations. Component replacement-based methods can improve spatial resolution, but often lead to severe spectral distortion; multi-resolution analysis-based methods have difficulty in effectively utilizing texture and structure information of SAR images when processing multi-source data; statistical learning-based methods are computationally complex, require a large amount of prior knowledge, and are difficult to adjust parameters. In addition, when there is a large resolution difference between multispectral and SAR images, traditional methods cannot effectively maintain spectral integrity.

[0004] Therefore, how to improve the spatial resolution of multispectral images while fully utilizing the high-resolution structure information of SAR images and maintaining the spectral integrity of multispectral data has become a core technical problem that needs to be solved in this field. Especially in complex scenes with rich ground object details, existing technologies are difficult to achieve a balance between spatial information enhancement and spectral fidelity. That is, there is a technical problem of low spatial resolution of multispectral images and spectral information distortion in the SAR data fusion process in the existing technology. SUMMARY

[0005] Therefore, the present application provides a multispectral and SAR data fusion method, medium and system, which can solve the technical problem of low spatial resolution of multispectral images and spectral information distortion in the SAR data fusion process in the existing technology.

[0006] The application is implemented in the following manner: a first aspect of the application provides a multispectral and SAR data fusion method, comprising: when constructing a deep learning training set, down-sampling a SAR image to the same spatial resolution of a multispectral image to obtain a low-resolution SAR image; performing a down-sampling operation and then an up-sampling operation on the multispectral image to obtain a degraded multispectral image; taking the original multispectral image as a model training label; cutting the low-resolution SAR image and the degraded multispectral image into fixed-size tile training samples; designing a deep fusion network comprising a feature extraction module, a feature fusion module and an image reconstruction module; training the deep fusion network using the tile training samples; when predicting, processing a to-be-fused multispectral image and a high-resolution SAR image into a consistent spatial range and inputting the trained deep fusion network; and outputting a fused multispectral image through the deep fusion network.

[0007] The down-sampling operation specifically reduces the spatial resolution of a high-resolution image through an interpolation algorithm; and the up-sampling operation specifically improves the spatial resolution of a low-resolution image through an interpolation algorithm.

[0008] The tile training sample is specifically a training sample formed by cutting a large-size remote sensing image into fixed-size pixel blocks after labeling the large-size remote sensing image, and the fixed size is 256x256 pixels.

[0009] The fused multispectral image is specifically a multispectral image with improved spatial resolution and complete spectral information.

[0010] The feature extraction module specifically extracts multi-scale feature representations from the multispectral image and the SAR image using a convolutional neural network; the feature fusion module specifically effectively combines multi-source features through an attention mechanism or other fusion algorithms to retain respective advantage information; and the image reconstruction module specifically restores the fused features to a high-resolution multispectral image output through deconvolution or up-sampling convolution.

[0011] In the step of processing the to-be-fused multispectral image and the high-resolution SAR image into a consistent spatial range, the multispectral image and the SAR image are specifically adjusted to the same geographic coordinate system and spatial range through geometric transformation.

[0012] In the step of designing the deep fusion network comprising the feature extraction module, the feature fusion module and the image reconstruction module, a fusion quality evaluation equation set optimization step is further included, specifically, the feature fusion module is dynamically optimized in the network training stage by constructing a multi-objective fusion quality evaluation equation set; the fusion quality evaluation equation set comprises a structural consistency equation, a spectral fidelity equation, a texture enhancement equation, a global correlation equation and a fusion judgment equation.

[0013] In the step of processing the multispectral image to be fused and the high-resolution SAR image into a consistent spatial range, a multi-level dynamic cutting optimization step is further included, specifically, the multispectral image to be fused and the high-resolution SAR image are processed by multi-level dynamic cutting, and the cutting tile size is dynamically adjusted according to the image content complexity.

[0014] The second aspect of the present application provides a computer readable storage medium, wherein program instructions are stored in the computer readable storage medium, and the program instructions are used to execute the above-mentioned multispectral and SAR data fusion method when running in a computer.

[0015] The third aspect of the present application provides a multispectral and SAR data fusion system, which includes the above-mentioned computer readable storage medium, and the system is any one of a computer, a server, and a single-chip microcomputer, the computer readable storage medium is arranged in the system, and the system is provided with a microprocessor for executing the program instructions stored in the computer readable storage medium.

[0016] The present application realizes efficient fusion of multi-source remote sensing data by designing a specific deep learning model architecture and training strategy. In the training phase, the mapping relationship from low-quality input to high-quality output is established through image degradation simulation and supervised learning of original multispectral images, effectively solving the information expression and integration problem in multi-source data fusion. The method of the present application has the following advantages compared with traditional technologies: first, through deep feature extraction and attention mechanism fusion module, the complementary features of multispectral and SAR data can be adaptively learned, avoiding the limitations of artificial feature design in traditional methods; second, the degradation-reconstruction supervised learning strategy is adopted, so that the network can effectively learn the mapping from degraded images to original high-quality images, so that in the prediction stage, SAR data can be used to assist in reconstructing high-resolution multispectral images; finally, the entire fusion process realizes spatial resolution enhancement while maintaining the integrity of spectral information. Therefore, the present application effectively solves the technical problems of low spatial resolution of multispectral images and spectral information distortion in the SAR data fusion process, realizes efficient fusion of multi-source remote sensing data with complementary advantages, and provides a high-quality data basis for various remote sensing applications. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flowchart of the method of the present application.

[0018] Figure 2 The loss function change graph of the deep fusion network training process in Example 2. DETAILED DESCRIPTION

[0019] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0020] As Figure 1 shown in a flow chart of a method for fusing multispectral and SAR data provided by the first aspect of the present application, the method comprises the following steps:

[0021] S01, when constructing a deep learning training set, down-sampling a SAR image to the same spatial resolution as a multispectral image to obtain a low-resolution SAR image;

[0022] S02, performing a down-sampling operation and then an up-sampling operation on the multispectral image to reduce the quality of the multispectral image to obtain a degraded multispectral image;

[0023] S03, using an original multispectral image as a model training label to supervise network learning of high-quality multispectral image reconstruction;

[0024] S04, cutting the low-resolution SAR image and the degraded multispectral image into fixed-size tile training samples according to the same geographical range;

[0025] S05, designing a deep fusion network architecture, including a feature extraction module, a feature fusion module and an image reconstruction module;

[0026] S06, training the deep fusion network using the tile training samples and optimizing network parameters until the verification accuracy reaches a preset threshold;

[0027] S07, in the prediction stage, processing the multispectral image to be fused and the high-resolution SAR image into a consistent spatial range;

[0028] S08, inputting the multispectral image to be fused and the high-resolution SAR image in the consistent spatial range into the trained deep fusion network at the same time;

[0029] S09, outputting a fused multispectral image with improved spatial resolution and complete spectral information through the deep fusion network.

[0030] The down-sampling operation specifically refers to reducing the spatial resolution of a high-resolution image through an interpolation algorithm, and common algorithms include nearest neighbor interpolation, bilinear interpolation or bicubic interpolation.

[0031] The up-sampling operation specifically refers to improving the spatial resolution of a low-resolution image through an interpolation algorithm, but without increasing the actual information quantity, resulting in a decrease in image quality.

[0032] The tile training sample is specifically a fixed size, such as 256*256 pixel block, which is cut from a large size remote sensing image for batch processing of a deep learning network.

[0033] The feature extraction module is specifically a multi-scale feature representation extracted from the multispectral image and the SAR image using a convolutional neural network.

[0034] The feature fusion module is specifically an effective combination of multi-source features through an attention mechanism or other fusion algorithms to retain the respective advantage information.

[0035] The image reconstruction module is specifically a high-resolution multispectral image output recovered from the fused features through deconvolution or up-sampling convolution.

[0036] The spatial registration is specifically a geometric transformation of the multispectral image and the SAR image to the same geographic coordinate system and spatial range.

[0037] In step S05, the fusion quality evaluation equation set optimization step is further included, which is specifically a dynamic optimization of the feature fusion module in the network training stage through the construction of a multi-objective fusion quality evaluation equation set, thereby improving the fusion effect and generalization ability.

[0038] The fusion quality evaluation equation set includes a structure consistency equation, a spectral fidelity equation, a texture enhancement equation, a global correlation equation, and a fusion determination equation, which respectively evaluate the fusion quality from different dimensions. In the network training process, the parameter update of the feature fusion module is no longer dependent on a single loss function, but is guided by the comprehensive guidance of this fusion quality evaluation equation set. The specific implementation method is: first, a plurality of feature representations are extracted from the fused multi-spectral image, including a fused multi-spectral image gradient graph, a fused multi-spectral image mean value of each band, a fused multi-spectral image local entropy value, and a fused multi-spectral image feature vector; then, the features are compared and calculated with the corresponding features of the original multi-spectral image and the high-resolution SAR image, respectively obtaining a structure consistency index, a spectral fidelity index, a texture enhancement index, and a global correlation index; then, the indexes are integrated into a fusion quality determination value through a fusion determination equation. In the back propagation phase, the system calculates the gradient and updates the feature fusion module parameters according to the fusion quality determination value, and the weight coefficient of the fusion determination equation can also be dynamically adjusted to adapt to different scene requirements. This multi-objective optimization mechanism balances the factors such as structure information preservation, spectral information integrity, texture detail enhancement, and global correlation, avoiding the problem of quality decline in other aspects caused by single index optimization. For example, focusing only on structure consistency may lead to spectral distortion; focusing only on spectral fidelity may lose spatial details. The entire optimization process is carried out in an iterative manner, and as the training progresses, the system will gradually find the optimal fusion parameters that balance all indexes, thereby obtaining a fused multi-spectral image that performs well in multiple quality dimensions. This method greatly improves the ability of the deep fusion network to deal with complex scenes, and is particularly suitable for the fusion of multi-spectral images and SAR images, which are heterogeneous data fusion tasks. In this task, the deep fusion network needs to process the complementary information of different source data and improve the overall information content while preserving the characteristics of the original data.

[0039] The structure consistency equation is used to evaluate the structure information preservation degree between the fused multi-spectral image and the high-resolution SAR image, the input includes the fused multi-spectral image gradient graph, the high-resolution SAR image gradient graph, the local window size, the edge intensity threshold, and the structure weight parameter, and the output is the structure consistency index;

[0040] The spectral fidelity equation is used to evaluate the spectral information preservation degree between the fused multi-spectral image and the original multi-spectral image, the input includes the fused multi-spectral image mean value of each band, the original multi-spectral image mean value of each band, the band weight vector, the spectral angle distance threshold, and the color distortion tolerance, and the output is the spectral fidelity index;

[0041] The texture enhancement equation is used to quantify the degree of texture detail enhancement introduced from the high-resolution SAR image in the fusion process, and the input includes local entropy values of the fused multi-spectral image, the original multi-spectral image, the high-resolution SAR image, a texture complexity parameter and a scale factor, and the output is a texture enhancement index.

[0042] The global correlation equation is used to evaluate the global correlation degree of the fused multi-spectral image with the original multi-spectral image and the high-resolution SAR image, and the input includes feature vectors of the fused multi-spectral image, the original multi-spectral image and the high-resolution SAR image, feature dimension weights and a correlation threshold, and the output is a global correlation index.

[0043] The fusion decision equation is used to calculate a final fusion quality decision value by comprehensively considering the structure consistency index, the spectral fidelity index, the texture enhancement index and the global correlation index, the fusion quality decision value is used as an optimization index for parameters of the feature fusion module in network training, and the higher the fusion quality decision value is, the better the fusion quality is.

[0044] In step S07, a multi-level dynamic cropping optimization step is further included, specifically, the multi-level dynamic cropping processing is performed on the multi-spectral image to be fused and the high-resolution SAR image, and the cropping tile size is dynamically adjusted according to the image content complexity to avoid the regional segmentation problem caused by fixed size cropping.

[0045] The multi-level dynamic cropping optimization step first identifies edge dense regions and texture rich regions in the multi-spectral image to be fused and the high-resolution SAR image through an edge detection algorithm, then determines a cropping strategy according to the fusion quality decision value, uses a smaller tile size for the edge dense regions and the texture rich regions to retain detailed information, and uses a larger tile size for homogeneous regions to maintain spatial coherence, and finally seamlessly splices the processed tiles to restore the complete fused multi-spectral image, effectively reduces tile boundary artifacts and improves overall fusion effect.

[0046] The edge dense region specifically refers to a region in the image with a higher gradient value and a density of edge feature points exceeding a preset threshold.

[0047] The texture rich region specifically refers to a region in the image with a local texture complexity calculated by a gray level co-occurrence matrix exceeding a preset threshold.

[0048] The homogeneous region specifically refers to a region in the image with a gentle gray scale change and a local standard deviation lower than a preset threshold.

[0049] The tile boundary artifact specifically refers to a brightness jump or texture discontinuity phenomenon at the splicing position of adjacent tiles due to inconsistent processing.

[0050] The specific implementation of the above steps is described in detail below.

[0051] The specific implementation of step S01 is to perform downsampling processing on the high-resolution SAR image to make its spatial resolution consistent with that of the multispectral image. This step first obtains the spatial resolution information of the original SAR image and the multispectral image, calculates the resolution ratio factor of the two, and then uses the bicubic interpolation algorithm to perform downsampling processing on the SAR image. The bicubic interpolation algorithm generates the target pixel value by weighted summation of 16 adjacent pixels of the original image. Compared with nearest neighbor interpolation and bilinear interpolation, it has better edge preservation ability and effectively reduces block artifacts. The downsampling ratio is usually an integer multiple, for example, from 3-meter resolution to 10-meter resolution, and the ratio factor is 10 / 3. This step aims to create multisource data pairs with the same spatial resolution, providing paired training samples for subsequent deep learning models, and ensuring that the network can learn the mapping relationship between multisensor data.

[0052] The specific implementation of step S02 is to perform quality reduction processing on the multispectral image to simulate low-quality data in actual applications. This step first performs downsampling operation on the original multispectral image, uses a Gaussian low-pass filter for smoothing, and then uses the bilinear interpolation algorithm to reduce the resolution to 1 / 2 to 1 / 4 of the original resolution. The downsampling ratio can be adjusted according to the actual application scenario, and is usually 2 or 4. Then, the multispectral image with reduced resolution is subjected to upsampling operation, and the bilinear interpolation is used to restore it to the original spatial resolution, but without increasing the actual information amount. This quality reduction process simulates the blurring and detail loss of the multispectral image in actual applications, enabling the network to learn how to recover high-quality images from low-quality inputs, and improving the robustness and generalization ability of the model.

[0053] The specific implementation of step S03 is to set the original high-quality multispectral image as the training label of the deep learning network. This step first preprocesses the original multispectral image, including radiation correction, atmospheric correction, and normalization processing, adjusts the pixel value range to the 0-1 interval, and eliminates sensor noise and atmospheric effects. Then, the processed original multispectral image is used as the network training label to calculate the loss function and guide the network parameter update. Through the supervised learning mechanism, the network learns the ability to reconstruct high-quality multispectral images from degraded multispectral images and low-resolution SAR images, achieving effective data fusion and image quality improvement. The loss function uses a weighted combination of mean square error and structural similarity index, and the weight ratio is usually set to 0.7:0.3 to balance the pixel-level accuracy and structural information preservation.

[0054] The specific implementation of step S04 is to perform spatial registration and cutting operation on low-resolution SAR image and degraded multi-spectral image to generate deep learning training samples. First, image spatial registration is performed based on geographic information system to ensure that the two data have the same geographic coordinate system and spatial range, and the registration accuracy should be controlled within 0.5 pixels. Then, the sliding window method is used to cut the registered image into fixed size tiles, and the tile size is usually set to 256x256 or 512x512 pixels, and the adjacent tile overlap rate is set to 25%-50% to increase the number of training samples and avoid edge effects. At the same time, considering the information amount of each tile, the tiles with cloud coverage of more than 10% or containing all invalid values are removed. After cutting, data enhancement operations are performed for each tile, including random rotation, flipping and brightness adjustment, to further expand the training set and improve the model generalization ability.

[0055] The specific implementation of step S05 is to design a deep fusion network architecture, including a feature extraction module, a feature fusion module and an image reconstruction module. The feature extraction module uses an improved residual network structure to extract multi-scale features from multi-spectral images and SAR images, and the network depth is set to 10-15 layers, and the convolution kernel size is 3x3, and the channel number increases from 64 to 256 layer by layer. The feature fusion module uses a dual attention network combining channel attention mechanism and spatial attention mechanism to perform adaptive weight allocation and fusion of different source data features, and the attention module contains 2-3 fully connected layers, and the activation function uses parameterized rectified linear unit. The image reconstruction module is composed of 3-5 deconvolution layers, which gradually restores the spatial resolution and reconstructs the complete multi-spectral image, and the last layer uses convolution to adjust the channel number to be consistent with the target multi-spectral band number. The overall network uses an encoder-fuser-decoder architecture, and residual connections are added between modules to alleviate the gradient vanishing problem and promote feature transmission.

[0056] Further, the implementation method of the fusion quality evaluation equation optimization step in step S05 is to construct a multi-objective quality evaluation system and dynamically optimize the feature fusion module. The structure consistency equation evaluates the structure information retention degree by calculating the correlation coefficient between the gradient graph of the fused multi-spectral image and the high-resolution SAR image. The local window size is set to 7x7 or 9x9 pixels, the edge intensity threshold is the front 30% percentile of the gradient amplitude, and the structure weight parameter is set to 0.8-0.9. The spectral fidelity equation calculates the similarity between the fused multi-spectral image and the original multi-spectral image based on the spectral angle mapping algorithm. The band weight vector is determined according to the information entropy of each band, the spectral angle distance threshold is set to 0.05 radian, and the color distortion tolerance is 0.1. The texture enhancement equation quantifies the texture details introduced from the SAR image using local entropy, and the local entropy calculation window size is 5x5 pixels. The texture complexity parameter is set to 0.7, and the scale factor is set to 1.5-3 according to the SAR and multi-spectral resolution ratio. The global correlation equation evaluates the overall correlation between the fusion result and the input data based on the canonical correlation analysis method. The feature dimension weight is inversely proportional to the principal component analysis feature variance contribution rate, and the correlation threshold is set to 0.6. The fusion judgment equation adopts a weighted sum form to integrate the four indexes. The weight coefficient is dynamically adjusted by the Bayesian optimization algorithm. The fusion quality judgment value ranges from 0 to 1. The higher the value, the better the fusion quality. The threshold is usually set to 0.75.

[0057] The implementation of step S06 is to train the deep fusion network and optimize the network parameters using tile training samples. First, the training sample set is divided into training set, validation set and test set according to the ratio of 7:2:1. Then, the Adam optimization algorithm is used for network training. The initial learning rate is set to 0.001, and the learning rate is reduced to 0.8 times of the original every 20 cycles. The batch size is set to 8-32 according to the hardware condition, and the total number of training cycles is 100-300. After each cycle, the model performance is evaluated using the validation set to calculate the peak signal-to-noise ratio, structural similarity index and spectral angle mapping error. When the validation accuracy improves by no more than 0.1% for 10 consecutive cycles or reaches the preset threshold of 0.85, the training is stopped in advance to prevent overfitting. At the same time, the gradient clipping technique is used to limit the maximum gradient norm to 10 to avoid unstable training. Finally, the test set is used for final performance evaluation to ensure that the model has good generalization ability.

[0058] The specific implementation of step S07 is to preprocess the to-be-fused multi-spectral image and the high-resolution SAR image in the prediction stage to make them have consistent spatial ranges. First, the geographic coordinate information and projection parameters of the two kinds of images are obtained respectively, and then the multi-spectral image is re-projected to the SAR image coordinate system based on the geographic registration technology. The re-projection process adopts an affine transformation algorithm, the control points are selected based on the mutual information maximization criterion, and the registration error is controlled within 0.3 pixels. After re-projection, the image cropping operation is used to ensure that the two kinds of data have completely consistent spatial ranges and coordinate reference systems, and finally the consistency of the grid size, row and column number and geographic range is verified to ensure that the spatial correspondence is accurate and correct. This step provides input data for subsequent fusion processing that are spatially matched, and is a prerequisite for accurate fusion.

[0059] Further, the multi-level dynamic cutting optimization step in step S07 has a specific implementation method of adaptively adjusting the cutting strategy according to the content characteristics of the image. First, the Canny edge detection algorithm is used to identify edge-intensive regions, and the high and low threshold values are set to 80% and 40% of the gradient amplitude histogram, respectively. The texture-rich region is identified by the entropy value and contrast of the gray level co-occurrence matrix, and the local entropy threshold value is set to 5.0 and the contrast threshold value is set to 0.6. The homogeneous region is identified by the local standard deviation, and the standard deviation threshold value is set to 15% of the original image standard deviation. Then, the cutting tile size is dynamically determined according to the region type, and small-size tiles of 128x128 or 192x192 pixels are used for edge-intensive regions and texture-rich regions, and large-size tiles of 384x384 or 512x512 pixels are used for homogeneous regions. After processing each tile, the Poisson image editing algorithm is used for seamless splicing, and the transition band width is set to 16-32 pixels. The transition region gradient field is kept continuous by solving the Poisson equation, effectively reducing tile boundary artifacts. This step balances the processing accuracy and efficiency by targeted processing of different complexity regions, and improves the overall fusion effect.

[0060] The specific implementation of step S08 is to simultaneously input the to-be-fused multi-spectral image and the high-resolution SAR image with consistent spatial ranges into the trained deep fusion network. First, the input data is preprocessed by normalization, and the pixel value is scaled to the interval of 0-1. The multi-spectral image is normalized by the maximum and minimum value, and the SAR image is normalized after logarithmic transformation to reduce the sample distribution difference. Then, a batch data structure is constructed, each batch contains multiple pairs of image tiles, and the batch size is usually set to 1-4, depending on the hardware memory capacity. During the network forward propagation process, the multi-spectral image and the SAR image pass through their respective feature extraction branches, then integrate complementary information in the feature fusion module, and finally output the fusion result through the image reconstruction module. The whole processing process is in an end-to-end manner, without human intervention, and the processing speed is related to the network complexity and hardware performance, with a typical processing time of 0.5-2 minutes per square kilometer.

[0061] The specific implementation of step S09 is to obtain a fused multi-spectral image with improved spatial resolution and complete spectral information through a deep fusion network. The network output is first subjected to inverse normalization processing to restore the pixel value from the 0-1 interval to the original dynamic range. Then, the output result is subjected to quality evaluation, including calculation of the spatial resolution gain coefficient, spectral distortion, edge preservation index, and noise suppression degree, and the evaluation thresholds are set to 1.5, 0.08, 0.8, and 0.75, respectively. For local abnormal regions, a post-processing strategy is adopted for optimization, including median filtering to eliminate isolated noise points, edge enhancement algorithm to improve profile clarity, and striping processing based on a variational model. Finally, the fused multi-spectral image is output, which has the same spatial resolution as the high-resolution SAR image, the spectral information is highly consistent with the original multi-spectral image, the spectral angle distance error is controlled within 0.05 radians, the peak signal-to-noise ratio is improved by 3-6 decibels, and the structural similarity index is increased by 0.05-0.15.

[0062] Further, the detailed structure of the deep fusion network is as follows: the network as a whole adopts a dual-flow encoder-fusor-single-flow decoder architecture. The multi-spectral flow encoder includes 5 residual blocks, each of which includes 2 layers of 3x3 convolution layers, batch normalization layers, and LeakyReLU activation functions, and the number of channels starts from 64 and increases by one time per layer until 512; the SAR flow encoder has a similar structure, but the first layer includes a logarithmic transformation layer to process the speckle noise characteristics of the SAR image. The feature fusion module consists of two parts: a cross-modal feature interaction unit and a multi-scale feature aggregation unit. The cross-modal feature interaction unit adopts a channel attention mechanism to calculate the feature channel importance weight, and the attention module includes a global average pooling layer, two fully connected layers, and a Sigmoid activation function; the multi-scale feature aggregation unit extracts and fuses features from different scales through a spatial pyramid pooling structure, and the pooling scales are 1x1, 2x2, 4x4, and 8x8, respectively, and then the channel dimension is adjusted through a 1x1 convolution. The image reconstruction module includes 4 transpose convolution layers, which gradually restore the spatial resolution, and the number of channels is halved per layer; at the same time, multi-level skip connections are introduced to splice the features of the corresponding layers of the encoder and the decoder, promoting detail recovery. A global residual connection is adopted at the end of the network to add the up-sampled input multi-spectral image to the network output, enhancing the stability of the network. The loss function consists of pixel-level L1 loss, structural similarity loss, spectral angle distance loss, and adversarial loss, and the weight proportions are 0.5:0.2:0.2:0.1, which comprehensively optimize the spatial details and spectral fidelity.

[0063] In the present application, the "performing down-sampling operation and then performing up-sampling operation on the multi-spectral image" and "using the original multi-spectral image as the model training label" constitute a supervised learning strategy of degradation-reconstruction, and the technical principle thereof lies in creating a self-supervised learning framework, so that the network can learn the mapping relationship from low-quality input to high-quality output. The down-sampling-up-sampling operation simulates the quality degradation process of the multi-spectral image in actual application through the reduction and then the promotion of the spatial resolution, and the degraded multi-spectral image produced retains the basic spectral characteristics of the original image but loses the spatial details; and using the original multi-spectral image without degradation as the training label provides the network with a "ideal output" supervision signal. This design enables the network to learn how to use the high-frequency structure information in the SAR image to make up for the lost spatial details in the multi-spectral image while maintaining the integrity of the spectral information during the training process. From the technical effect, this strategy solves the contradiction between spatial enhancement and spectral fidelity that cannot be reconciled by traditional fusion methods: first, through degradation sample training, the network learns to distinguish which information can be obtained from the SAR data and which information needs to be retained from the multi-spectral data; second, the supervision signal ensures that the spectral information is not replaced incorrectly during the reconstruction process; third, this simulated degradation-reconstruction training paradigm enhances the generalization ability of the model, enabling it to cope with input data of different quality levels; finally, this self-supervised mechanism reduces the dependence on large-scale labeled data and improves the practicality of the method. This optimization design essentially builds an end-to-end learning framework from degraded multi-source data to high-quality fusion data, achieving an optimal balance between spatial and spectral information.

[0064] Further, the technical principle of the fusion quality evaluation equation set optimization step in S05 is to construct a multi-dimensional and multi-objective evaluation system, and through the synergistic effect of the four equations of structural consistency, spectral fidelity, texture enhancement and global correlation, the overall constraint and dynamic optimization of the fusion process are realized. This method breaks through the limitation of the traditional single loss function, enables the feature fusion module to balance adjustment in different quality dimensions, and avoids the problem of performance decline caused by single index optimization. This multi-objective optimization mechanism can dynamically adjust the weights of the evaluation indexes according to different scene characteristics in the back propagation process, providing more accurate gradient guidance for the feature fusion module, so as to realize more intelligent parameter updating. The technical effects are: the fusion result meets the multiple requirements of structure preservation, spectral integrity, texture enhancement and global consistency; the network can adaptively adjust the fusion strategy according to different regions and scene characteristics; the model has stronger generalization ability and can adapt to different types of multispectral and SAR data combinations; and the generation of false information in the fusion process is significantly reduced. The technical principle of the multi-level dynamic cutting optimization step in S07 is to break through the limitation of the traditional fixed size tile processing, intelligently adjust the cutting strategy according to the content complexity of the image, use finer cutting size for edge-intensive areas and texture-rich areas to retain details, and use larger tiles for homogeneous areas to maintain continuity. This content-adaptive processing method cooperates with the aforementioned fusion quality evaluation equation set, and the technical effects are: effectively avoiding the problem that important features or texture features are divided at the tile boundary; reducing the splicing artifacts at the tile boundary; improving the fusion quality of different complexity areas; enhancing the system's ability to process large-scale remote sensing images; and ensuring the spatial continuity and integrity of the fusion result.

[0065] The second aspect of the present application provides a computer readable storage medium, wherein program instructions are stored in the computer readable storage medium, and the program instructions are used to execute the above-mentioned method for multispectral and SAR data fusion when running in a computer.

[0066] The third aspect of the present application provides a system for multispectral and SAR data fusion, which comprises the above-mentioned computer readable storage medium. The system is any one of a computer, a server or a single chip microcomputer. The computer readable storage medium is arranged in the system, and the system is provided with a microprocessor for executing the program instructions stored in the computer readable storage medium.

[0067] Specifically, the principle of the present application is that the technical principle of the present application is based on the feature extraction, fusion and reconstruction ability of deep learning, and the advantages of multi-source remote sensing data are complementary through the ingenious design of the network architecture and the training strategy. The core principle lies in solving the differences between multispectral and SAR two kinds of heterogeneous data in physical imaging mechanism, radiation characteristics and spatial resolution, etc., and establishing an effective information mapping and fusion mechanism.

[0068] In the feature extraction stage, the present application extracts multi-scale feature representations from multispectral and SAR data respectively. The multispectral branch mainly learns the associated information and basic spatial structure in the spectral dimension; the SAR branch focuses on extracting high-frequency texture and edge structure information. Through the hierarchical feature extraction capability of the convolutional neural network, the system can automatically learn the key features in the two kinds of data, avoiding the limitations of manual feature engineering in traditional methods.

[0069] The feature fusion module is the core of the present application, which realizes the adaptive fusion of two kinds of heterogeneous features by using the attention mechanism. The attention mechanism can dynamically adjust the fusion strategy according to the contribution of different regional and scale features by learning the feature importance weight, which not only retains the spectral information of multispectral data, but also integrates the high-resolution structural information of SAR data. This adaptive fusion method is superior to the simple weighted fusion or replacement operation in traditional methods.

[0070] The image reconstruction module converts the fused features into high-resolution multispectral output, and restores the spatial details through deconvolution or up-sampling convolution operation. In particular, the present application adopts a degradation-reconstruction supervised learning strategy, which enables the network to learn the mapping relationship from the degraded image to the original high-quality image, effectively avoiding the spectral distortion problem caused by direct fusion. By using the original multispectral image as a supervision signal, the system can maintain the integrity of the spectral information during the reconstruction process, while using the high-resolution structural information provided by the SAR data to enhance the expression of spatial details.

[0071] In addition, the network training adopts a tile sample construction strategy, which effectively handles the high-dimensional and large-size characteristics of remote sensing data, and improves the model training efficiency and generalization ability. In the prediction stage, spatial registration is used to ensure the consistency of multi-source data, providing a basic guarantee for fusion.

[0072] A specific embodiment 1 of the present application is provided below, and the specific implementation manner of each step in the embodiment 1 is described in detail as follows.

[0073] The specific implementation manner of step S01 is to perform down-sampling processing on the high-resolution SAR image, so that its spatial resolution is consistent with that of the multispectral image. First, the original SAR image resolution and the multispectral image resolution are obtained, and the resolution ratio factor Then, the SAR image is down-sampled by using a bicubic interpolation algorithm. The bicubic interpolation algorithm is based on a cubic convolution kernel function, which is expressed as follows:

[0074] ;

[0075] wherein, is the normalized distance from the center pixel, and the value range is .

[0076] The low-resolution SAR image after down-sampling is obtained by two-dimensional convolution of the original SAR image by using the convolution kernel function, and the calculation formula is as follows:

[0077] ;

[0078] wherein, is the original SAR image; is the low-resolution SAR image after down-sampling; is the target pixel coordinate; is the original pixel coordinate; is the convolution kernel function. The down-sampling process realizes the conversion from the high-resolution SAR image to the data with the same resolution as the multispectral image, thereby providing the paired training samples for the subsequent deep learning model.

[0079] The specific implementation of step S02 is to perform quality reduction processing on the multispectral image to simulate low-quality data in actual application. First, a down-sampling operation is performed on the original multispectral image, and then the resolution is reduced by using a bicubic interpolation algorithm after smoothing processing by using a Gaussian low-pass filter. The two-dimensional convolution kernel of the Gaussian low-pass filter is defined as:

[0080] ;

[0081] wherein, is the position of the pixel relative to the center of the convolution kernel; is the standard deviation of the Gaussian function, which controls the smoothing degree, and the value range is 1-3.

[0082] The mathematical expression of the down-sampling process is as follows:

[0083] ;

[0084] wherein, is the original multispectral image; is the multispectral image after down-sampling; is the target pixel coordinate; is the original pixel coordinate; represents the convolution operation. ​

[0085] Then, an upsampling operation is performed on the downsized multispectral image, and bilinear interpolation is used to restore it to its original spatial resolution. The bilinear interpolation formula is as follows:

[0086] ;

[0087] In the formula, The image is a degraded multispectral image; The floating-point coordinates of the target pixel; and They represent and The integer part; and , where is the weighting coefficient. This degradation process simulates the blurring and detail loss of multispectral images in real-world applications.

[0088] The specific implementation of step S03 involves setting the original high-quality multispectral image as the training label for the deep learning network. First, the original multispectral image is preprocessed, including radiometric correction, atmospheric correction, and normalization. Radiometric correction uses a linear stretching method.

[0089] ;

[0090] In the formula, The image is a radiometrically corrected multispectral image. This is the original multispectral image; These are pixel coordinates; For band indexing; and These are the minimum and maximum values ​​of the original data, respectively.

[0091] Atmospheric correction uses dark target subtraction:

[0092] ;

[0093] In the formula, This is an atmospherically corrected multispectral image; For the first The digital representation of dark targets in each band is obtained by statistically analyzing the average value of 1% of the darkest pixels in each band.

[0094] Normalization adjusts the pixel value range to the 0-1 interval:

[0095] ;

[0096] In the formula, The normalized multispectral images were used as training labels for the network. and They represent the first Minimum and maximum values ​​of the band.

[0097] Step S04 involves spatial registration and cropping of low-resolution SAR and degraded multispectral images to generate deep learning training samples. First, spatial registration is performed based on a geographic information system (GIS) to ensure that the two datasets have the same geographic coordinate system and spatial extent. The spatial registration uses an affine transformation model.

[0098] ;

[0099] In the formula, The coordinates are in the source coordinate system; The coordinates are in the target coordinate system; ~ The six parameters of the affine transformation matrix are calculated using the least squares method based on control point pairs. The transformation matrix is ​​solved using the normal equation:

[0100] ;

[0101] In the formula, This is a coefficient matrix composed of the coordinates of the control points; Transformation parameter vector; The target coordinate vector is used. Control points are selected based on feature matching, with matching accuracy controlled within 0.5 pixels.

[0102] Then, the registered image is cropped into tiles of fixed size using the sliding window method. The formula for calculating the tile position is:

[0103] ;

[0104] In the formula, Indicates the position index is A set of tile pixels; The sliding step size is typically 50% to 75% of the tile size; and These represent the width and height of the tile, typically set to 256×256 or 512×512 pixels.

[0105] The specific implementation of step S05 involves designing a deep fusion network architecture, including a feature extraction module, a feature fusion module, and an image reconstruction module. The feature extraction module employs an improved residual network structure to extract multi-scale features from both multispectral and SAR images. The mathematical expression of the residual block is as follows:

[0106] ;

[0107] In the formula, and respectively represent the feature map of the layer and the layer, respectively. represents the residual mapping function. represents the network parameters of the layer.

[0108] The residual mapping function is defined as a combination of convolution, batch normalization and activation function:

[0109] .

[0110] wherein, represents the convolution operation; represents the batch normalization operation; represents the activation function, which adopts a parameterized rectified linear unit (PReLU): wherein, is a learnable parameter, and the initial value is set to 0.2.

[0111] The feature fusion module adopts a dual attention network combining channel attention mechanism and spatial attention mechanism. The channel attention calculation formula is:

[0112] .

[0113] wherein, is the input feature map; and respectively represent the global average pooling and global maximum pooling operations; represents a multi-layer perceptron composed of two fully connected layers; represents a Sigmoid activation function; is the channel attention weight, which has the same size as the feature map channel number.

[0114] The spatial attention calculation formula is:

[0115] .

[0116] wherein, and respectively represent the average pooling and maximum pooling results along the channel dimension; represents a concatenation operation along the channel dimension; represents a convolution layer; is the spatial attention weight, which has the same size as the feature map spatial dimension.

[0117] The final feature fusion expression is:

[0118] .

[0119] wherein, and denote multispectral features and SAR features, respectively; denotes element-wise multiplication; is the fused feature.

[0120] In the optimization step of the fusion quality assessment equations, the mathematical expression of the structural consistency equation is:

[0121]

[0122] wherein, is the structural consistency index, with a value range of [0, 1], and the larger the value is, the higher the structural preservation degree is; and denote the gradient maps of the fused multispectral image and the high-resolution SAR image, respectively; is the position of the weight coefficient, which is determined by the edge intensity:

[0123]

[0124] wherein, is the edge intensity threshold, with a value of the first 30% percentile of the gradient amplitude; is the non-edge area weight, with a value range of 0.1-0.3.

[0125] The mathematical expression of the spectral fidelity equation is:

[0126]

[0127] wherein, is the spectral fidelity index, with a value range of [0, 1], and the larger the value is, the higher the spectral preservation degree is; and denote the mean values of each band of the fused multispectral image and the original multispectral image, respectively; is the band index; is the total number of bands; is the weight of the band, which is calculated according to the information entropy:

[0128]

[0129] wherein, is the information entropy of the band: wherein is the probability distribution of the gray value .

[0130] The mathematical expression of the texture enhancement equation is:

[0131] ​​​​ ;

[0132] wherein, is the texture enhancement index, and the value range is [0, 1], and the greater the value is, the higher the texture enhancement degree is; , and respectively represent the local entropy maps of the fused multi-spectral image, the original multi-spectral image and the high-resolution SAR image; is a small constant for preventing zero division, and the value is 0.001; is a scale factor, and the value is 1.5-3, which is determined according to the resolution ratio of SAR and multi-spectral image. The local entropy calculation formula is:

[0133] ;

[0134] wherein, is the normalized gray value of the pixel in the local window.

[0135] The mathematical expression of the global correlation equation is:

[0136] ;

[0137] wherein, is the global correlation index, and the value range is [0, 1], and the greater the value is, the higher the overall correlation degree is; represents the typical correlation coefficient; , and respectively represent the feature vectors of the fused multi-spectral image, the original multi-spectral image and the high-resolution SAR image; and are weight coefficients, and , and the weight coefficients are usually , . The typical correlation coefficient calculation formula is:

[0138] ;

[0139] wherein, represents the covariance of and ; and respectively represent the variance of and .

[0140] The mathematical expression of the fusion determination equation is:

[0141] ;

[0142] wherein, The fusion quality judgment value has a value range of [0, 1], and the higher the value, the better the fusion quality. 、 、 and are weight coefficients, and , the initial values are respectively set to 0.3, 0.3, 0.2 and 0.2, and are dynamically adjusted by a Bayesian optimization algorithm:

[0143] ;

[0144] In the formula, and respectively represent the weight vectors of the first and the first iteration; is a learning rate, and has a value range of 0.01-0.05; represents the gradient of the fusion quality judgment value to the weight vector.

[0145] The specific implementation of step S06 is to train the deep fusion network with tile training samples and optimize the network parameters. The Adam optimization algorithm is used for network training, and the update rule is:

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] ;

[0151] In the formula, represents the current gradient; and respectively represent the first-order momentum and the second-order momentum; and are the momentum after bias correction; is the updated network parameter; is a learning rate, and the initial value is set to 0.001; and are momentum decay coefficients, and have values of 0.9 and 0.999, respectively; is a small constant to prevent division by zero, and has a value of .

[0152] The learning rate is dynamically adjusted using the cosine annealing strategy:

[0153] ;

[0154] where, is the learning rate of the th epoch; and are the minimum and maximum learning rates, respectively, with values of 0.0001 and 0.001; is the total number of epochs, with values of 100-300.

[0155] The loss function is a weighted combination of the mean squared error and structural similarity index:

[0156]

[0157] where, is the mean squared error loss; is the structural similarity loss; is the spectral angle mapping loss; and are the weight coefficients, with values of 0.5, 0.3, and 0.2, respectively.

[0158] The specific calculation formulas for each loss term are:

[0159]

[0160]

[0161]

[0162] where, and represent the network prediction output and the true label, respectively, and represent the th network prediction output and the th true label, respectively; and represent the mean of and , respectively; and represent the standard deviation, respectively; represents the covariance; and are small constants to prevent division by zero; and represent the spectral vectors of the prediction output and the true label at position , respectively.

[0163] ​​​​​The specific implementation of step S07 is to preprocess the multispectral image and the high-resolution SAR image to be fused in the prediction stage, so that they have consistent spatial range. First, the geographic coordinate information and projection parameters of the two kinds of images are obtained respectively, and then the multispectral image is re-projected to the SAR image coordinate system based on geographic registration technology. The re-projection adopts an affine transformation model, the control points are selected based on the mutual information maximization criterion, and the registration error is controlled within 0.3 pixels. The mutual information calculation formula is:

[0164]

[0165] In the formula, denotes the mutual information between images and denotes the joint probability distribution; and denote the marginal probability distributions, respectively. In the registration process, the optimal control point pair is selected by maximizing the mutual information value.

[0166] The specific implementation method of the multi-level dynamic clipping optimization step is to adaptively adjust the clipping strategy according to the content characteristics of the image. First, the Canny edge detection algorithm is used to identify the edge dense area:

[0167]

[0168]

[0169] In the formula, denotes the gradient amplitude at position and denote the horizontal and vertical gradients, respectively; denotes the gradient direction.

[0170] The high and low thresholds of the Canny algorithm are set to be the 80% and 40% percentiles of the gradient amplitude histogram, respectively. The edge dense area judgment criterion is:

[0171]

[0172] In the formula, denotes the edge dense area judgment result; denotes the Canny edge detection result; denotes the statistical window size, which is 16 or 32; denotes the density threshold, which is 0.3-0.5.

[0173] The texture-rich area is identified by the entropy value and contrast of the gray level co-occurrence matrix:

[0174] ​​​​​​ ;

[0175] ;

[0176] where, and denote the entropy and the contrast of the gray level co-occurrence matrix respectively; denotes the joint probability from gray value to gray value ; denotes the number of gray levels, usually taking 16 or 32.

[0177] The criterion for the texture-rich region is:

[0178] ;

[0179] where, denotes the texture-rich region determination result; denotes the entropy threshold, taking 5.0; denotes the contrast threshold, taking 0.6.

[0180] The homogeneous region is identified by the local standard deviation:

[0181] ;

[0182] where, denotes the homogeneous region determination result; denotes the local standard deviation; denotes the standard deviation threshold, taking 15% of the original image standard deviation.

[0183] Finally, the cutting tile size is dynamically determined according to the region type:

[0184] ;

[0185] where, denotes the tile size at position ; , and denote the small, medium and large tile sizes respectively, taking 128x128, 256x256 and 512x512 pixels respectively.

[0186] The specific implementation of step S08 is to simultaneously input the to-be-fused multi-spectral image and the high-resolution SAR image in the same spatial range into the trained deep fusion network. First, the input data is normalized and preprocessed, and the pixel value is scaled to the interval of 0-1:

[0187] ;

[0188] wherein, denotes the normalized multispectral image; denotes the original multispectral image; denotes the pixel position; denotes the band index; and denote the minimum and maximum values of the band, respectively.

[0189] The SAR image is first log-transformed to mitigate speckle noise, and then normalized:

[0190] ;

[0191] ;

[0192] wherein, denotes the log-transformed SAR image; denotes the normalized SAR image.

[0193] Then a batch data structure is constructed, each batch containing multiple pairs of image tiles:

[0194] ;

[0195] wherein, denotes the data of a batch; denotes the batch size, taking values from 1 to 4, depending on the hardware memory capacity.

[0196] The mathematical expression of the network forward propagation process is:

[0197] ;

[0198] ;

[0199] ;

[0200] ;

[0201] wherein, and denote the multispectral encoder and the SAR encoder, respectively; and denote the extracted multispectral features and SAR features, respectively; denotes the feature fusion module; denotes the fused features; denotes the decoder; denotes the fusion output result.

[0202] The specific implementation of step S09 is to obtain a fused multispectral image with improved spatial resolution and complete spectral information through a deep fusion network. The network output is first subjected to inverse normalization processing to restore the pixel value from the 0-1 interval to the original dynamic range:

[0203] ;

[0204] In the formula, represents the inverse normalized fused multispectral image; represents the normalized fusion result of the network output.

[0205] Then, the output result is subjected to quality evaluation, including calculation of the spatial resolution gain coefficient, spectral distortion degree, edge preservation index, and noise suppression degree:

[0206] ;

[0207] In the formula, represents the spatial resolution gain coefficient, a value greater than 1 indicates that the spatial resolution is improved, and the ideal value is the square of the ratio of the SAR resolution to the multispectral resolution; and respectively represent the gradient map of the fusion result and the original multispectral image.

[0208] The spectral distortion degree calculation formula is:

[0209] ;

[0210] In the formula, represents the spectral distortion degree, and a smaller value indicates a higher spectral preservation degree; represents the original multispectral image subjected to interpolation up-sampling; represents the number of bands; represents the image size.

[0211] The edge preservation index calculation formula is:

[0212] ;

[0213] In the formula, represents the edge preservation index, with a value range of [0, 1], and a larger value indicates a higher edge preservation degree; and respectively represent the edge map of the fusion result and the high-resolution SAR image; represents the edge overlap operation.

[0214] The noise suppression degree calculation formula is:

[0215] ;

[0216] wherein, denotes the noise suppression degree, and the value range is [0, 1], and the larger the value is, the better the noise suppression effect is; and respectively denote the variance of the fusion result and the high-resolution SAR image in the homogeneous region.

[0217] The threshold values of the evaluation indexes are respectively set as: , , , If the evaluation indexes do not meet the requirements, the model needs to be retrained or the parameters need to be adjusted.

[0218] For the local abnormal region, a post-processing strategy is adopted for optimization. First, the abnormal region is detected through local standard deviation:

[0219] ;

[0220] wherein, denotes the abnormal region marking map; and respectively denote the local standard deviation and the mean value; denotes the abnormality detection threshold value, and the value is 3-5.

[0221] For the detected abnormal region, median filtering is applied for processing:

[0222] ;

[0223] wherein, denotes the fusion multispectral image after post-processing; denotes the median filtering operation; denotes the filtering window radius, and the value is 1 or 2.

[0224] Finally, an edge enhancement algorithm is realized based on a non-sharpening mask:

[0225] ;

[0226] wherein, denotes the final output fusion multispectral image; denotes the fusion image after Gaussian filtering; denotes the edge enhancement factor, and the value is 0.5-1.5.

[0227] The overall architecture of the deep fusion network adopts a double-flow encoder-fusion-single-flow decoder structure. The multispectral flow encoder and the SAR flow encoder are respectively responsible for extracting features from the multispectral image and the SAR image. The encoder is composed of multiple residual blocks, and each residual block is represented as:

[0228] ;

[0229] In the formula, and They represent the first Layer and first Features of the layer; Indicates the first The nonlinear mapping function of the layer. The specific implementation of the nonlinear mapping function is as follows:

[0230] ;

[0231] In the formula, and These represent the convolution operations in the first and second layers, respectively. This indicates a batch normalization operation; The activation function is represented by the parameterized corrected linear unit (PReLU).

[0232] Feature fusion module The mathematical expression of a dual attention network that combines channel attention and spatial attention mechanisms is as follows:

[0233] ;

[0234] ;

[0235] ;

[0236] ;

[0237] ;

[0238] In the formula, and These represent multispectral features and SAR features, respectively. and These represent the channel attention weights and spatial attention weights, respectively. and These represent the weighted multispectral features and SAR features, respectively. This indicates the characteristics after fusion.

[0239] decoder Composed of multiple transposed convolutional layers, it is used to transform fused features into high-resolution multispectral images. The transposed convolution operation is represented as:

[0240] ;

[0241] In the formula, Represents the transpose convolution operation. At the same time, multi-level skip connection is introduced to splice the encoder corresponding layer feature and the decoder feature:

[0242] ;

[0243] In the formula, represents the decoder first layer feature; represents the encoder corresponding layer feature; represents the feature splicing operation; represents the total number of network layers. The network end also adopts global residual connection to add the input multispectral image up-sampling and network output:

[0244] ;

[0245] In the formula, represents the up-sampling operation.

[0246] The loss function is composed of pixel-level L1 loss, structural similarity loss, spectral angle distance loss and adversarial loss:

[0247] ;

[0248] In the formula, represents the L1 loss; represents the structural similarity loss; represents the spectral angle distance loss; represents the adversarial loss; , , and are weight coefficients, and the values are 0.5, 0.2, 0.2 and 0.1 respectively.

[0249] The specific calculation formula of each loss term is:

[0250] ;

[0251] ;

[0252] ;

[0253] ;

[0254] In the formula, represents the fusion result of network output; represents the real label; and respectively represent the spectral vectors at positions ​The discriminator network is used to judge the consistency of the fusion result and the high-resolution SAR image.

[0255] In summary, the multispectral and SAR data fusion method realizes the effective combination of spatial resolution improvement and spectral information preservation through deep learning technology. In the training stage, low-resolution SAR images and degraded multispectral images are used as input, and the original multispectral image is used as label to learn the mapping relationship between multisensor data. In the prediction stage, the to-be-fused multispectral image and the high-resolution SAR image are input into the trained deep fusion network, and the fusion multispectral image with improved spatial resolution and complete spectral information is output. Through the optimization of the fusion quality evaluation equation set and the multi-level dynamic cutting optimization, the fusion effect and the generalization ability are further improved, and the method is suitable for remote sensing image fusion tasks in various complex scenes.

[0256] In order to better understand and implement the present application, the following provides an embodiment 2 of a specific application scenario of the present application: In order to solve the problems of insufficient spatial resolution of multispectral data and single spectral information of SAR data in agricultural remote sensing monitoring, researchers carried out a multispectral and SAR data fusion test in a farmland irrigation area. The area contains rice, rape, wheat and other crop types, with an area of about 25 square kilometers. The test uses GF6 multispectral data and GF3 SAR data, wherein the GF6 multispectral data has a spatial resolution of 8m and contains four bands of blue (450-520nm), green (520-590nm), red (630-690nm) and near-infrared (760-900nm), and the GF3 SAR data is C band with a spatial resolution of 3m. The purpose of fusion is to generate a fusion data product with high spatial resolution (3m) of GF3 and complete spectral information of GF6, to support farmland irrigation condition assessment and crop growth monitoring.

[0257] Step S01: Down-sampling the GF3 SAR data. First, measure the spatial resolution of the original GF3 SAR data and the GF6 multispectral data, calculate the resolution ratio factor α = 8 / 3 ≈ 2.67. Then use the bicubic interpolation algorithm to down-sample the GF3 SAR image, generate a low-resolution SAR image (8m) consistent with the spatial resolution of the GF6 multispectral image. The down-sampling uses a 3x3 pixel convolution window, and the convolution kernel is:

[0258] ;

[0259] The down-sampling results are shown in Table 1:

[0260] Table 1 Comparison of SAR data before and after down-sampling

[0261]

[0262] Step S02: Quality reduction processing is performed on the GF6 multispectral image. First, a Gaussian low-pass filter is used to smooth the original multispectral image, with the Gaussian kernel parameter σ set to 1.5. Then, bilinear interpolation is used to reduce the resolution to 1 / 3 of the original resolution (i.e., 24 m). Subsequently, the low-resolution multispectral image is upsampled back to the original resolution (8 m) using bilinear interpolation to form the degraded multispectral image. This process simulates the blurring and loss of detail that occurs in practical applications. The processing results are shown in Table 2:

[0263] Table 2: Effect of multispectral data degradation processing

[0264]

[0265] Step S03: The original GF6 multispectral image is set as the training label for the deep learning network. The original multispectral image is subjected to radiometric correction and atmospheric correction to eliminate sensor noise and atmospheric effects, and then normalized to the range of 0-1. The typical pixel value distribution of the original multispectral image is shown in Table 3:

[0266] Table 3: Typical surface reflectance of the original multispectral image

[0267]

[0268] Step S04: The low-resolution SAR image and the degraded multispectral image are cropped to the same geographic range as the training samples. First, spatial registration is performed using an affine transformation model with a registration accuracy of within 0.35 pixels. Then, the registered images are cropped into fixed-size tiles using a sliding window method, with a tile size of 256x256 pixels and a sliding step size of 192 pixels (25% overlap rate between adjacent tiles). Tiles with cloud coverage exceeding 10% and invalid values exceeding 5% are removed. The number of training samples generated is shown in Table 4:

[0269] Table 4: Number of training samples

[0270]

[0271] Step S05: The deep fusion network architecture is designed. The network includes a feature extraction module, a feature fusion module, and an image reconstruction module. The feature extraction module uses an improved residual network structure with a network depth of 12 layers and a convolution kernel size of 3x3. The number of channels increases from 64 to 256 layer by layer. The feature fusion module uses a dual attention network that combines channel attention and spatial attention mechanisms. The network parameter settings are shown in Table 5:

[0272] Table 5: Deep fusion network parameter settings

[0273]

[0274] The fusion quality evaluation equation set includes a structure consistency equation, a spectral fidelity equation, a texture enhancement equation, a global correlation equation, and a fusion determination equation. The edge intensity threshold of the structure consistency equation is set to the first 30% percentile of the gradient amplitude, and the non-edge area weight β is set to 0.2. The band weight of the spectral fidelity equation is calculated according to the information entropy, and the near-infrared band weight is the highest at 0.38. The scale factor γ of the texture enhancement equation is set to 2.5. The weight coefficients λ1 and λ2 of the global correlation equation are set to 0.6 and 0.4, respectively. The initial weights of the fusion determination equation are set to w1=0.3, w2=0.3, w3=0.2, and w4=0.2, respectively, and the learning rate η is set to 0.02.

[0275] Step S06: training the deep fusion network using the tile training sample. The Adam optimization algorithm is used for network training, the initial learning rate is set to 0.001, the cosine annealing strategy is used for dynamic adjustment, and the minimum learning rate is 0.0001. The batch size is set to 16, and the training period is 200. The loss function adopts a weighted combination of mean square error, structural similarity index, and spectral angle mapping, and the weight ratio is 0.5:0.3:0.2. The loss function changes in the training process are shown in Table 6:

[0276] Table 6 Loss function change table in network training process

[0277]

[0278] As shown in Figure 2 , the trend of L1 loss, SSIM loss, SAM loss, and total loss with the training period in the network training process is shown. It clearly shows how the loss value quickly decreases from the early training period, then gradually stabilizes, and stops training in the 183rd period due to reaching the preset precision threshold. The verification accuracy reaches the preset threshold of 0.82, and the training is stopped in the 183rd period to prevent overfitting. The final evaluation indicators on the test set are shown in Table 7:

[0279] Table 7 Test set evaluation index table

[0280]

[0281] Step S07: Data preprocessing in prediction stage. The GF6 multispectral image and GF3 SAR image to be fused are processed into consistent spatial range. First, the two kinds of data are spatially registered based on geographic information system, using affine transformation model, control points are selected based on mutual information maximization criterion, and the registration accuracy is controlled within 0.25 pixels. Then, multi-level dynamic cropping processing is performed, using Canny edge detection algorithm to identify edge dense areas, and the high and low threshold values are set to 78% and 38% percentiles of gradient amplitude histogram respectively. Texture-rich areas are identified by gray level co-occurrence matrix entropy, and the threshold value is set to 5.2. Homogeneous areas are identified by local standard deviation, and the threshold value is set to 13% of the original image standard deviation. The dynamic cropping allocation is shown in Table 8:

[0282] Table 8. Multi-level dynamic cropping statistics table

[0283]

[0284] Step S08: The GF6 multispectral image and GF3 SAR image to be fused are simultaneously input into the trained deep fusion network. The multispectral image is normalized by maximum and minimum value, and the SAR image is normalized after logarithmic transformation. The batch data structure is constructed, and the batch size is set to 2. The whole processing process is in an end-to-end manner, and the total processing time is 32 minutes, with an average processing time of 1.28 minutes per square kilometer.

[0285] Step S09: Obtain the fusion result and perform quality evaluation. The network output is first de-normalized to restore the pixel value from the 0-1 interval to the original dynamic range. Then the quality evaluation indicators are calculated: the spatial resolution gain coefficient is 2.56, the spectral distortion is 0.062, the edge preservation index is 0.863, and the noise suppression degree is 0.826, all of which meet the evaluation threshold requirements. For local abnormal areas, 3x3 median filtering is applied for post-processing, and the abnormal detection threshold is set to 4.2. Edge enhancement is achieved using a non-sharpening mask, and the edge enhancement factor is set to 1.2. The final fusion result is shown in Table 9:

[0286] Table 9. Comparison of indicators before and after fusion

[0287]

[0288] Compared with traditional multispectral and SAR data fusion methods, the deep learning method used in the embodiment has significant advantages. The traditional methods mainly include transform domain-based methods (such as high-pass filter injection, wavelet transform, principal component analysis, etc.) and statistical model-based methods (such as Bayesian fusion, sparse representation, etc.). These methods usually have problems such as serious spectral distortion, difficulty in simultaneously maintaining spatial and spectral quality, and difficulty in self-adaptive adjustment of algorithm parameters. For example, the wavelet transform method is prone to ring artifacts in processing high-frequency information such as farmland boundaries, and the general farmland boundary extraction accuracy can only reach about 85%; the principal component analysis method is prone to spectral distortion, and the NDVI accuracy usually decreases by 8% to 12%. In contrast, the deep fusion network of the present application dynamically optimizes the feature fusion module through a multi-objective fusion quality evaluation equation set, realizes the balance of structure information preservation, spectral information integrity, texture detail enhancement and global correlation, and at the same time, the multi-level dynamic cutting strategy adaptively adjusts the processing mode according to the content characteristics of the image, effectively reducing the edge artifact problem caused by the traditional fixed window processing. The experimental results in the farmland irrigation area of Jingzhou, Hubei Province show that, compared with traditional methods, the present application can maintain high spatial resolution (3m) while controlling the NDVI accuracy decrease within 2.2%, improving the farmland boundary extraction accuracy to 92.8%, and increasing the small farmland (area <0.5 hectares) recognition rate by 33.0%, thereby providing more accurate data support for fine agricultural management and farmland irrigation monitoring.

[0289] It should be noted that the variables involved in the present application are explained in detail as shown in Tables 10, 11, 12 and 13.

[0290] Table 10 Variable explanation table (first part)

[0291]

[0292] Table 11 Variable explanation table (second part)

[0293]

[0294] Table 12 Variable explanation table (third part)

[0295]

[0296] Table 13 Variable explanation table (fourth part)

[0297]

[0298] The above description is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for fusing multispectral and SAR data, characterized in that, include: When constructing the deep learning training set, SAR images are downsampled to the same spatial resolution as multispectral images to obtain low-resolution SAR images; multispectral images are first downsampled and then upsampled to obtain degraded multispectral images; the original multispectral images are used as model training labels; low-resolution SAR images and degraded multispectral images are cropped into fixed-size tile training samples; a deep fusion network is designed, including a feature extraction module, a feature fusion module, and an image reconstruction module; the deep fusion network is trained using tile training samples, and the feature fusion module is dynamically optimized during the network training phase using a multi-objective fusion quality evaluation equation set; during prediction, the multispectral images to be fused and the high-resolution SAR images are processed to have the same spatial range and input into the trained deep fusion network; the fused multispectral image is output through the deep fusion network; wherein... The fusion quality evaluation equation set includes a structural consistency equation, a spectral fidelity equation, a texture enhancement equation, a global correlation equation, and a fusion decision equation. The training process of the deep fusion network includes: firstly, extracting multiple feature representations from the fused multispectral image, including the gradient map of the fused multispectral image, the mean value of each band of the fused multispectral image, the local entropy value of the fused multispectral image, and the feature vector of the fused multispectral image; then, calculating the extracted multiple feature representations with the corresponding features of the original multispectral image and the high-resolution SAR image through the fusion quality evaluation equation set to obtain the structural consistency index, spectral fidelity index, texture enhancement index, and global correlation index; finally, integrating the four indices obtained through the fusion decision equation into a fusion quality decision value, which serves as the parameter optimization index for the feature fusion module in the network training; The structural consistency equation is used to evaluate the degree of structural information preservation between the fused multispectral image and the high-resolution SAR image. The inputs include the gradient map of the fused multispectral image, the gradient map of the high-resolution SAR image, the local window size, the edge intensity threshold, and the structural weight parameters. The output is the structural consistency index. The spectral fidelity equation is used to evaluate the degree of spectral information preservation between the fused multispectral image and the original multispectral image. The inputs include the mean value of each band of the fused multispectral image, the mean value of each band of the original multispectral image, the band weight vector, the spectral angular distance threshold, and the color distortion tolerance. The output is the spectral fidelity index. The texture enhancement equation is used to quantify the degree of texture detail enhancement introduced from the high-resolution SAR image during the fusion process. The inputs include the local entropy value of the fused multispectral image, the local entropy value of the original multispectral image, the local entropy value of the high-resolution SAR image, the texture complexity parameter, and the scale factor. The output is the texture enhancement index. The global correlation equation is used to evaluate the global correlation between the fused multispectral image and the original multispectral image and the high-resolution SAR image. The input includes the feature vector of the fused multispectral image, the feature vector of the original multispectral image, the feature vector of the high-resolution SAR image, the feature dimension weights, and the correlation threshold. The output is the global correlation index.

2. The method according to claim 1, characterized in that, The downsampling operation specifically involves reducing the spatial resolution of a high-resolution image using an interpolation algorithm; the upsampling operation specifically involves increasing the spatial resolution of a low-resolution image using an interpolation algorithm.

3. The method according to claim 2, characterized in that, The tile training samples are specifically formed by labeling large-size remote sensing images and then cropping them into fixed-size pixel blocks.

4. The method according to claim 3, characterized in that, The fused multispectral image is specifically a multispectral image with improved spatial resolution while maintaining the integrity of spectral information.

5. The method according to claim 4, characterized in that, The feature extraction module specifically uses a convolutional neural network to extract multi-scale feature representations from multispectral and SAR images; the feature fusion module specifically combines multi-source features effectively through attention mechanisms or other fusion algorithms, retaining their respective advantages; the image reconstruction module specifically restores the fused features into a high-resolution multispectral image output through deconvolution or upsampling convolution.

6. The method according to claim 5, characterized in that, In the step of processing the multispectral image to be fused and the high-resolution SAR image to have the same spatial extent, the multispectral image and the SAR image are adjusted to the same geographic coordinate system and spatial extent through geometric transformation.

7. The method according to claim 6, characterized in that, The step of processing the multispectral image to be fused and the high-resolution SAR image to have the same spatial range also includes a multi-level dynamic cropping optimization step, which specifically involves performing multi-level dynamic cropping processing on the multispectral image to be fused and the high-resolution SAR image, and dynamically adjusting the size of the cropped tiles according to the complexity of the image content.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed in a computer, perform a method for fusing multispectral and SAR data as described in any one of claims 1-7.

9. A system for fusing multispectral and SAR data, characterized in that, The system includes the computer-readable storage medium of claim 8, wherein the system is any one of a computer, a server, or a microcontroller, the computer-readable storage medium is disposed within the system, and the system is provided with a microprocessor that executes the program instructions stored in the computer-readable storage medium.

Images