A fusion method and system for multi-source remote sensing images

Abstract

The invention relates to the technical field of image fusion, and in particular to a fusion method and system for multi-source remote sensing images. The method comprises: constructing a fusion model based on a generative adversarial network, training the fusion model according to a sample data set, and obtaining a trained fusion model; wherein the fusion model comprises an adaptive resolution convolutional Transformer generator and a convolutional Transformer discriminator, and the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator are deep supervised generative adversarial networks constructed by convolutional Transformer modules; performing fusion processing on a fine-resolution image and a coarse-resolution image to be fused based on the trained fusion model, and obtaining a fused image; the invention can simultaneously realize the spatial-spectral fusion and the spatiotemporal fusion of multi-source remote sensing images and improve the fusion ability and the generalization ability.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a fusion method and system for multi-source remote sensing images. Background Art

[0002] Images obtained by a single sensor cannot simultaneously meet the requirements of high spatial resolution, high spectral resolution, and high temporal resolution. Multi-source remote sensing image fusion technology is an effective way to obtain multispectral images with high spatial resolution (spatial-spectral fusion) and multispectral images with high temporal and spatial resolution (spatial-temporal fusion). However, most current deep learning models can only achieve spatial-spectral fusion or spatial-temporal fusion.

[0003] Therefore, how to simultaneously achieve spatial-spectral fusion and spatiotemporal fusion of multi-source remote sensing images has become an urgent problem to be solved. Summary of the invention

[0004] The present invention aims to provide a multi-source remote sensing image fusion method and system, which can simultaneously realize the spatial-spectral fusion and spatiotemporal fusion of multi-source remote sensing images and improve the fusion ability and generalization ability.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a method for fusing multi-source remote sensing images, the method comprising the following steps:

[0007] S100, obtaining a sample data set, wherein the sample data set includes a space-spectrum fusion data set and a space-time fusion data set; wherein the space-spectrum fusion data set includes a plurality of panchromatic images with high spatial resolution and multispectral images with low spatial resolution, and the space-time fusion data set includes a plurality of multispectral images with low temporal and high spatial resolution and multispectral images with high temporal and low spatial resolution;

[0008] S200, constructing a fusion model based on a generative adversarial network, and training the fusion model according to the sample data set to obtain a trained fusion model; wherein the fusion model includes an adaptive resolution convolutional Transformer generator and a convolutional Transformer discriminator, and the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator are deep supervised generative adversarial networks constructed by convolutional Transformer modules;

[0009] S300, performing a fusion process on the fine-resolution image and the coarse-resolution image to be fused based on the trained fusion model to obtain a fused image.

[0010] Optionally, the adaptive resolution convolutional Transformer generator includes an encoder, a decoder and a reconstruction unit; in S200, the fusion model is trained according to the sample data set to obtain a trained fusion model, including:

[0011] S210, inputting the sample data set into an encoder, adjusting the resolution level and resolution factor of the encoder according to the spatial resolution difference between the high-resolution image and the low-resolution image in the sample data set, and extracting multi-scale local-global features of the fine-resolution image and the coarse-resolution image respectively in a manner from fine resolution to coarse resolution and from coarse resolution to fine resolution;

[0012] S220, fusing complementary local-global features of different levels and resolutions step by step through a decoder, hierarchically reconstructing to generate multispectral images of different resolutions, and reconstructing the multispectral images through a reconstruction unit to generate a fused image;

[0013] S230, adversarially learning local-global structures and spectral information of different resolutions through the convolutional Transformer discriminator and feeding back the output discrimination result to the adaptive resolution convolutional Transformer generator;

[0014] S240, determining a loss value between a source image in the sample data set and a target image generated by the fusion model for the source image, iteratively training the fusion model according to the loss value until the loss value is lower than a loss threshold, and obtaining a trained fusion model.

[0015] Optionally, the convolutional Transformer discriminator includes a plurality of discriminators of different resolutions. In S230, the adversarial learning of local-global structures and spectral information of different resolutions by the convolutional Transformer discriminator and feeding back the output discrimination result to the adaptive resolution convolutional Transformer generator includes:

[0016] S231, the convolutional Transformer discriminator sets a plurality of discriminators of corresponding resolutions according to the number of levels of the adaptive resolution convolutional Transformer generator, and the input of each of the discriminators is a down-resolution image and a predicted image of a corresponding resolution level, respectively; wherein the down-resolution image is generated by downsampling the reference image; and the predicted image includes a fused image of the corresponding resolution level generated by the adaptive resolution convolutional Transformer generator;

[0017] S232, the first stage of the n-th resolution discriminator in the convolutional Transformer discriminator extracts the primary local-global features of the n-th resolution reference image and the predicted image by corresponding convolution and MDC T, the middle stage extracts the deep features of the n-th resolution reference image and the predicted image by corresponding convolution and MDC T and reduces the size of the features in turn, and the last stage outputs the discrimination result by convolution and Sigmoid function;

[0018] S233, feeding back the discrimination result to the adaptive resolution convolutional Transformer generator to generate a more refined fused image.

[0019] Optionally, in S240, the loss value is determined in the following manner:

[0020] Construct a hybrid supervised loss function, which is:

[0021]

[0022] in, represents the mixed supervision loss function of the fusion model, represents the adversarial loss function between the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator, is the feature loss function on the convolutional Transformer discriminator at the nth resolution, is the perceptual loss function, represents the spatial consistency loss function, represents the spectral consistency loss function, μ1, ρ2, λ3 are weight coefficients;

[0023] The loss value of the source image and the image generated by the fusion model is calculated by the hybrid supervised loss function.

[0024] Optionally, the adversarial loss function between the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator is:

[0025]

[0026] And there are:

[0027]

[0028] Among them, L CTUGAN (G,CTD n ) represents the adversarial learning loss between the generator of the fusion model and the discriminator of the nth resolution, G represents the adaptive resolution convolutional Transformer generator of the fusion model, CTD nrepresents the convolutional Transformer discriminator of the nth resolution, represents the adversarial loss between the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator, represents the total adversarial loss of the convolutional Transformer discriminator, represents the relative probability obtained by the convolutional Transformer discriminator CTDn, σ represents the Sigmoid activation function, C n (·) represents the untransformed convolutional Transformer discriminator output, and R represents the reference image R at the nth resolution n The distribution of R0 image has the highest spatial resolution, and P represents the generated image P of the nth resolution. n The distribution of and Represents the average operation of all reference images and generated images in the batch data.

[0029] Optionally, the feature loss function on the convolutional Transformer discriminator is:

[0030]

[0031] Among them, L FM (G,CTD n ) is the CTD of the convolutional Transformer discriminator n The feature loss function on , Z is the total number of layers, is the CTD of the convolutional Transformer discriminator n The first layer features, M l is the total number of features in layer l.

[0032] Optionally, the perceptual loss function is:

[0033]

[0034] Where S is the batch size, f(R0) and f(P0) represent the feature maps of the reference image and generated image of the ideal resolution on the pre-trained model VGG-19, R0 represents the reference image of the ideal resolution, and P0 represents the generated image of the ideal resolution.

[0035] Optionally, the spatial consistency loss function is:

[0036]

[0037] in, represents the spatial consistency loss function, h() and ▽() represent high-pass filtering and gradient operations to extract high-frequency information and gradient information of the image, ||·|| F represents the F-norm.

[0038] Optionally, the spectral consistency loss function is:

[0039]

[0040] in, represents the spectral consistency loss function, and It is the reference image and generated image used for spectral information discrimination after conversion.

[0041] In a second aspect, an embodiment of the present invention provides a fusion system for multi-source remote sensing images, the system comprising:

[0042] at least one processor;

[0043] at least one memory for storing at least one program;

[0044] When the at least one program is executed by the at least one processor, the at least one processor implements the multi-source remote sensing image fusion method as described in any one of the above.

[0045] The beneficial effects of the present invention are as follows: the present invention discloses a method and system for fusing multi-source remote sensing images. The fusion model proposed in the present invention is based on a deep supervised generative adversarial network composed of convolutional Transformer modules. By constructing potential relationships between different source data, spatial-spectral fusion and spatiotemporal fusion are achieved, and the fusion ability and generalization ability are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A flow chart of a multi-source remote sensing image fusion method in an embodiment of the present invention is shown.

[0048] Figure 2 The structure diagram of the fusion model in the embodiment of the present invention is shown.

[0049] Figure 3 A structural diagram of an adaptive resolution convolutional Transformer generator in an embodiment of the present invention is shown.

[0050] Figure 4 A structural diagram of a convolutional Transformer discriminator in an embodiment of the present invention is shown.

[0051] Figure 5 A block diagram of a multi-source remote sensing image fusion system in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0052] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0053] In related technologies, remote sensing images with high temporal and spatial resolution play a very important role in monitoring coastal ecological environment changes, marine resource monitoring, and urban dynamic planning. However, due to the limitations of sensor technology and other factors, the products obtained are mutually restricted in temporal, spatial, and spectral resolution, which to a certain extent limits the application of remote sensing images. Temporal resolution and spatial resolution restrict each other. For example, the revisit period of Landsat satellite is about 16 days, and the spatial resolution of multispectral images is 30m; while the period for obtaining MODIS images is 1 day, and the spatial resolution is 500m. Spectral resolution and temporal resolution restrict each other. For example, the spatial resolution of multispectral images obtained by Gaofen-2 satellite is about 4m, and the spectral band is 4; while the corresponding panchromatic image is single-band and the spatial resolution is 1m. Multi-source remote sensing fusion technology integrates multi-source complementary information to provide a convenient and effective solution for generating high-resolution images. At present, a large number of remote sensing image fusion methods have been developed, which can be divided into space-spectrum fusion, space-time fusion, and space-time-spectrum fusion.

[0054] Spatial-spectral fusion aggregates the complementary spatial and spectral information between multi-source remote sensing images to generate images with high spatial and spectral resolution. Spatial-spectral fusion based on convolutional neural networks has achieved certain results. In recent years, Transformer has also been gradually applied to spatial-spectral fusion. However, most models are trained on down-resolution data. Although they have achieved good results on down-resolution data, the fusion results on full-resolution data are not ideal, which is not conducive to practical applications. For the fusion results on full-resolution data, there is still a lot of room for improvement in the spectrum and detail preservation of spatial-spectral fusion. Unsupervised models based on generative adversarial networks and Transformer still have great research potential to improve the spectrum and detail preservation capabilities.

[0055] Spatiotemporal fusion aggregates low temporal-high spatial resolution images and high temporal-low spatial resolution images to obtain high temporal-spatial resolution images. Although the current spatiotemporal fusion technology is good at predicting phenological changes, it is still challenging to predict changes in surface cover, which is still a difficult problem that needs to be solved; moreover, the huge resolution difference makes it difficult for spatiotemporal fusion to predict the details of spectral information changes, and the accuracy of the model is greatly affected by the number of reference images and the degree of information change, making it difficult to apply in actual scenarios.

[0056] The above-mentioned space-spectrum fusion and space-time fusion are implemented independently. Based on this, an integrated space-time spectral fusion framework is developed to realize space-spectrum fusion, space-time fusion and space-time spectral fusion. Although few integrated fusion models have been developed, remote sensing image fusion still faces the following challenges:

[0057] 1. The relationship between multi-source remote sensing images is complex, and the existing integrated fusion methods are limited. Deep learning models have strong nonlinear expression capabilities, and few deep learning fusion models can achieve spatial-spectral fusion and spatial-temporal fusion. They lack applicability and general functionality, and few studies have been conducted on models that simultaneously achieve spatial-spectral fusion and spatial-temporal fusion to meet the needs of different tasks such as classification, recognition, and dynamic monitoring.

[0058] 2. The huge difference in resolution between multi-source remote sensing images and sensor bias make it difficult to reconstruct the texture, details and change information of low-resolution images. For example, the resolution ratio between Landsat images and MODIS images in spatiotemporal fusion is 16, while the resolution ratio of image super-resolution is about 4.

[0059] 3. There is a lack of reference images for supervised training of deep learning models. For example, the spatial-spectral fusion model lacks high-spatial-resolution multispectral images as references, and the down-resolution training model leads to unsatisfactory fusion performance on full-resolution data.

[0060] Taking the above challenges into consideration, the present invention proposes a solution of spatial-spectral fusion and spatiotemporal fusion by generating an adversarial multi-source remote sensing image integrated fusion model based on convolutional Transformer.

[0061] The present invention realizes space-spectrum fusion and space-time fusion by constructing potential relationships between different source data to meet the needs of high-resolution images for different tasks.

[0062] Combination Figure 1 and Figure 2 The embodiment of the present invention provides a method for fusing multi-source remote sensing images, the method comprising the following steps:

[0063] S100, obtaining a sample data set, wherein the sample data set includes a space-spectrum fusion data set and a space-time fusion data set; wherein the space-spectrum fusion data set includes a plurality of panchromatic images with high spatial resolution and multispectral images with low spatial resolution, and the space-time fusion data set includes a plurality of multispectral images with low temporal and high spatial resolution and multispectral images with high temporal and low spatial resolution;

[0064] Specifically, paired remote sensing images collected by satellites including Gaofen-2 and QuickBird are used to form a spatial-spectral fusion dataset. The spatial resolution of multispectral / panchromatic images collected by Gaofen-2 and QuickBird is 4m / 1m and 2.4m / 0.6m, and the number of bands is 4 bands / 1 band. The spatiotemporal fusion dataset uses the publicly available Coleambly Irrigation Area (CIA) remote sensing dataset with a large number of seasonal phenological changes and the Lower GWydir Catchment (LGC) remote sensing dataset with a large number of changes in land cover types. The CIA and LGC datasets are composed of paired Landsat-5 / 7 and MODIS images.

[0065] S200, constructing a fusion model based on a generative adversarial network, and training the fusion model according to the sample data set to obtain a trained fusion model; wherein the fusion model includes an adaptive resolution convolutional Transformer generator and a convolutional Transformer discriminator, and the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator are deep supervised generative adversarial networks constructed by convolutional Transformer modules;

[0066] S300, performing a fusion process on the fine-resolution image and the coarse-resolution image to be fused based on the trained fusion model to obtain a fused image.

[0067] In the embodiments provided by the present invention, in view of the complex relationship between multi-source remote sensing images, the limited capabilities of existing integrated fusion methods, and the fact that few deep learning-based fusion models can achieve spatial-spectral fusion and spatiotemporal fusion, the present invention proposes a convolutional Transformer-generated adversarial multi-source remote sensing image integrated fusion model, which can achieve spatial-spectral fusion and spatiotemporal fusion, and improve the image fusion quality and the generalization ability of the fusion model.

[0068] refer to Figure 2 , Figure 3 and Figure 4 In the figure, FI represents the fine-resolution image, CI represents the coarse-resolution image, and GI represents the image generated by the proposed model. n and C n(n=0,1,……,N) represent the features of the nth scale extracted, the size of F0 is the same as that of FI (the dimension of the maximum resolution: H×W×B), F N The size of F0 is the same as that of CI (dimensions of minimum resolution: h×w×b). n The ratio of the resolution is τ n , τ is the downsampling factor each time and τ=2 is set. n Represents the generated intermediate image of the nth scale. HI n represents the generated image of the nth scale, and HI0 is the final fused image.

[0069] RMDCT stands for Residual Multi-Head Deep Convolutional Transformer Module, MFF stands for Multi-Resolution Feature Fusion Module, BALGE stands for Bidirectional Adaptive Local-Global Feature Encoder, MCAFD stands for Multi-Head Cross Attention Fusion Decoder, RU stands for Reconstruction Unit; MCAFT stands for Multi-Head Cross Attention Fusion Transformer Module; CTD stands for Convolutional Transformer Discriminator, and MDCT stands for Multi-Head Deep Convolutional Transformer Module.

[0070] In some embodiments, the adaptive resolution convolutional Transformer generator includes an encoder, a decoder, and a reconstruction unit; in S200, the fusion model is trained according to the sample data set to obtain a trained fusion model, including:

[0071] S210, inputting the sample data set into an encoder, adjusting the resolution level and resolution factor of the encoder according to the spatial resolution difference between the high-resolution image and the low-resolution image in the sample data set, and extracting multi-scale local-global features of the fine-resolution image and the coarse-resolution image respectively in a manner from fine resolution to coarse resolution and from coarse resolution to fine resolution;

[0072] The structure of the adaptive resolution convolutional Transformer generator is as follows Figure 3 As shown, in the present invention, the adaptive resolution convolutional Transformer generator includes a bidirectional adaptive local-global feature encoder (BALGE), a multi-head cross attention fusion decoder (MCAFD) and a reconstruction unit (RU). The encoder includes a branch for extracting multi-resolution features from high resolution to low resolution for encoding fine resolution images and a branch for extracting multi-resolution features from low resolution to high resolution for encoding coarse resolution images. The encoder extracts local-global features of different multi-resolutions of fine resolution images and coarse resolution images respectively.

[0073] In order to reduce the spectral information loss and detail distortion that may be caused by increasing the resolution multiple by a large factor at one time, it is considered to use small factors to gradually increase the spatial resolution, and the magnification can be adaptively adjusted according to the resolution of the coarse-fine image. This paper proposes a residual multi-head deep convolution Transformer module (RMDCT) to enhance the expression of multi-scale local and global features of remote sensing images. Figure 3 The branch for extracting the multi-resolution local-global features of the fine-resolution image in the encoder includes a dilated convolution module (MDconv), N RMDCT modules and downsampling, and the branch for extracting the multi-resolution local-global features of the coarse-resolution image includes MDconv, N RMDCT modules and upsampling.

[0074] S220, fusing complementary local-global features of different levels and resolutions step by step through a decoder, hierarchically reconstructing to generate multispectral images of different resolutions, and reconstructing the multispectral images through a reconstruction unit to generate a fused image;

[0075] Specifically, the decoder is used to fuse multi-source multi-resolution features and recovery information. The reconstruction unit is used to reconstruct the recovery information and generate the final fusion result. The decoder aggregates the local-global features of different levels and resolutions extracted from the encoder that are complementary to the fusion step by step to restore effective information resolution by resolution. The decoder uses the multi-resolution feature fusion (MFF) module and the multi-head cross attention fusion Transformer (MCAFT) module to reduce the image attribute deviation caused by different sensor mechanisms, aggregate and enhance the complementary features of multi-source images, and gradually enhance the reconstruction of local-global information. The reconstruction unit reconstructs the intermediate results obtained by the decoder to generate multi-resolution fusion results, and builds deep supervision with the convolutional Transformer discriminator.

[0076] S230, adversarially learning local-global structures and spectral information of different resolutions through the convolutional Transformer discriminator and feeding back the output discrimination result to the adaptive resolution convolutional Transformer generator;

[0077] S240, determining a loss value between a source image in the sample data set and a target image generated by the fusion model for the source image, iteratively training the fusion model according to the loss value until the loss value is lower than a loss threshold, and obtaining a trained fusion model.

[0078] Specifically, in response to the huge resolution differences between multi-source remote sensing images and the difficulty in reconstructing the texture, details and change information of low-resolution images caused by sensor bias, the present invention constructs a bidirectional adaptive local-global feature encoder and a multi-head cross-attention fusion decoder. The bidirectional adaptive local-global feature encoder adjusts the resolution level and resolution factor of the encoder according to the resolution difference between high-resolution and low-resolution images, and extracts multi-scale local-global features of fine-resolution images and coarse-resolution images from fine to coarse resolution and coarse to fine resolution respectively; it uses multi-scale convolution to enhance the expression of local information and uses the RMDCT module to enhance the expression of local and global information to better extract spectral, texture, and detail information.

[0079] The decoder uses the multi-resolution feature fusion (MFF) module and the multi-head cross-attention fusion Transformer model (MCAFT) module to aggregate and enhance the complementary features of multiple source images to gradually enhance the reconstruction of local-global information. In order to solve the problem of lack of reference images for supervised training of deep learning models, pixel-level supervised loss and multi-scale pixel-feature-level unsupervised loss are introduced according to the resolution relationship between the source image and the generated image, guiding the model to learn multi-resolution local-global information adversarially.

[0080] Because the sizes of objects in remote sensing images are different and local-global information is also introduced in the adaptive resolution convolutional Transformer generator, in order to combat learning of scene targets of different scales, the present invention designs a convolutional Transformer discriminator to combat learning of structures, spectra and other information of different resolutions to improve the authenticity of the generated images.

[0081] like Figure 4 As shown, in some embodiments, the convolutional Transformer discriminator includes multiple discriminators of different resolutions. In S230, the convolutional Transformer discriminator is used to learn the local-global structure and spectral information of different resolutions and the output discrimination result is fed back to the adaptive resolution convolutional Transformer generator, including:

[0082] S231, the convolutional Transformer discriminator sets a plurality of discriminators of corresponding resolutions according to the number of levels of the adaptive resolution convolutional Transformer generator, and the input of each of the discriminators is a down-resolution image and a predicted image of a corresponding resolution level, respectively; wherein the down-resolution image is generated by downsampling the reference image; and the predicted image includes a fused image of the corresponding resolution level generated by the adaptive resolution convolutional Transformer generator;

[0083] Specifically, N+1 (N=4) discriminators CTD0 to CTD are set according to the number of levels of the adaptive resolution convolutional Transformer generator. N , discriminator CTD0 to CTD N The inputs are R0 / HI0, ..., R N / HI N , where R n (n=0, ..., N) represents the 2 n The reduced resolution image generated by downsampling; H n (n=0, ..., N) represents the predicted image of the nth resolution of the adaptive resolution convolutional Transformer generator;

[0084] S232, the first stage of the n-th resolution discriminator in the convolutional Transformer discriminator extracts primary local-global features of the n-th resolution reference image and the predicted image by corresponding convolution and MDCT, the middle stage extracts deep features of the n-th resolution reference image and the predicted image by corresponding convolution and MDCT and reduces the size of the features in sequence, and the last stage outputs the discrimination result by convolution and Sigmoid function;

[0085] S233, feeding back the discrimination result to the adaptive resolution convolutional Transformer generator to generate a more refined fused image.

[0086] Specifically, CTD0 first uses convolution with a kernel of 3×3 and MDCT to extract primary local-global features, then uses three convolutions with a kernel of 4×4 and a stride of 2 and MDCT to extract deep features and reduce the size of the features in turn, and finally outputs a discrimination result with a size of 30×30×1 through two convolutions with a kernel of 4×4 and a stride of 1 and a Sigmoid function; CTD1 and CTD2 have the same structure as CTD0, but the weights are not shared, and the discrimination results with sizes of 14×14×1 and 6×6×1 are output respectively. Because the size of the input image is small, CTD3 and CTD4 reduce one convolution with a kernel of 4×4 and a stride of 2 and MDCT, and the other parts have the same structure as CTD0. CTD3 and CTD4 output discrimination results with sizes of 6×6×1 and 2×2×1 respectively. At the same time, in order to accelerate the convergence of the model and improve the stability and performance, the spectral normalization operation is introduced after each convolution operation. In addition, the relative average least squares discriminator is used to judge the relative authenticity of the input image to improve the stability and performance of the fusion model.

[0087] In some embodiments, in S240, the loss value is determined by:

[0088] Construct a hybrid supervised loss function, which is:

[0089]

[0090] in, represents the mixed supervision loss function of the fusion model, represents the adversarial loss function between the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator, is the feature loss function on the convolutional Transformer discriminator at the nth resolution, is the perceptual loss function, represents the spatial consistency loss function, represents the spectral consistency loss function, μ1, ρ2, λ3 are weight coefficients;

[0091] The loss value of the source image and the image generated by the fusion model is calculated by the hybrid supervised loss function.

[0092] In this embodiment, in order to make the fusion model converge quickly, stabilize the training and optimize the fusion performance of the fusion model, in addition to using the relative average least squares adversarial loss of the generative adversarial model, spatial consistency loss and spectral information consistency loss optimization model are introduced, and feature layer loss is also used to optimize the model. The feature layer loss includes the intermediate layer feature matching loss of the convolutional Transformer discriminator and the perceptual loss on the VGG-19 model.

[0093] In some embodiments, the adversarial loss for multi-resolution learning of the adaptive-resolution convolutional Transformer generator and the convolutional Transformer discriminator is:

[0094]

[0095] And there are:

[0096]

[0097] Among them, L CTUGAN (G,CTD n ) represents the adversarial learning loss between the generator of the fusion model and the discriminator of the nth resolution, G represents the adaptive resolution convolutional Transformer generator of the fusion model, CTD n represents the convolutional Transformer discriminator of the nth resolution, represents the adversarial loss between the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator, represents the total adversarial loss of the convolutional Transformer discriminator, represents the relative probability obtained by the convolutional Transformer discriminator CTDn, σ represents the Sigmoid activation function, C n (·) represents the untransformed convolutional Transformer discriminator output, and R represents the reference image R at the nth resolution n The distribution of R0 image has the highest spatial resolution, and P represents the generated image P of the nth resolution. n The distribution of and Represents the average operation of all reference images and generated images in the batch data.

[0098] Introduced a feature matching loss on the convolutional Transformer discriminator, using R n The L1 distance representation of the output feature map of the generated image on the convolutional Transformer discriminator is expressed as;

[0099]

[0100] Among them, L FM (G,CTD n ) is the CTD of the convolutional Transformer discriminator n The feature loss function on , Z is the total number of layers, is the CTD of the convolutional Transformer discriminator n The first layer features, M l is the total number of features in layer l.

[0101] In order to minimize the distance between two features before the activation function, the perceptual loss The expression is;

[0102]

[0103] Where S is the batch size, f(R0) and f(P0) represent the feature maps of the reference image and generated image of the ideal resolution on the pre-trained model VGG-19, R0 represents the reference image of the ideal resolution, and P0 represents the generated image of the ideal resolution.

[0104] In order to make the spatial structure information of the generated image consistent with that of the real image, the spatial consistency loss is;

[0105]

[0106] in, represents the spatial consistency loss function, h() and ▽() represent high-pass filtering and gradient operations to extract high-frequency information and gradient information of the image, ||·|| F represents the F-norm.

[0107] In order to make the spectral information of the generated image consistent with that of the real image, the spectral consistency loss function is defined as:

[0108]

[0109] in, represents the spectral consistency loss function, and It is the reference image and generated image used for spectral information discrimination after conversion.

[0110] In spatiotemporal fusion, the reference image R0 is the Landsat image of the predicted date; and In spatial-spectral fusion, the reference image R0 is the acquired PAN image; and The spectral resolution of the collected MS image is reduced and the expression is as follows:

[0111]

[0112] in, Represented by multispectral image The full-color image of the nth resolution obtained by reducing the spectral resolution, B is The number of bands of the image, and Represents the weight coefficient and bias of the nth scale. and The panchromatic image is obtained by reducing the spectral resolution of the generated multispectral image HI2.

[0113] and Figure 1 Corresponding to the method, refer to Figure 5 The embodiment of the present invention provides a multi-source remote sensing image fusion system, including:

[0114] at least one processor;

[0115] at least one memory for storing at least one program;

[0116] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0117] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0118] In addition, an embodiment of the present invention further discloses a computer program product or a computer program, which is stored in a computer-readable storage medium. A processor of a computer device can read the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above method. Similarly, the contents of the above method embodiment are all applicable to the storage medium embodiment, and the functions specifically implemented by the storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0119] It will be appreciated by those skilled in the art that all or some of the methods disclosed above, the system can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0120] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for fusing multi-source remote sensing images, characterized in that: The method comprises the following steps: S100, obtaining a sample data set, wherein the sample data set includes a space-spectrum fusion data set and a space-time fusion data set; wherein the space-spectrum fusion data set includes a plurality of panchromatic images with high spatial resolution and multispectral images with low spatial resolution, and the space-time fusion data set includes a plurality of multispectral images with low temporal and high spatial resolution and multispectral images with high temporal and low spatial resolution; S200, constructing a fusion model based on a generative adversarial network, and training the fusion model according to the sample data set to obtain a trained fusion model; wherein the fusion model includes an adaptive resolution convolutional Transformer generator and a convolutional Transformer discriminator, and the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator are deep supervised generative adversarial networks constructed by convolutional Transformer modules; S300, performing fusion processing on the fine-resolution image and the coarse-resolution image to be fused based on the trained fusion model to obtain a fused image; The adaptive resolution convolutional Transformer generator includes an encoder, a decoder and a reconstruction unit; in S200, the fusion model is trained according to the sample data set to obtain a trained fusion model, including: S210, inputting the sample data set into an encoder, adjusting the resolution level and resolution factor of the encoder according to the spatial resolution difference between the high-resolution image and the low-resolution image in the sample data set, and extracting multi-scale local-global features of the fine-resolution image and the coarse-resolution image respectively in a manner from fine resolution to coarse resolution and from coarse resolution to fine resolution; S220, fusing complementary local-global features of different levels and resolutions step by step through a decoder, hierarchically reconstructing to generate multispectral images of different resolutions, and reconstructing the multispectral images through a reconstruction unit to generate a fused image; S230, adversarially learning local-global structures and spectral information of different resolutions through the convolutional Transformer discriminator and feeding back the output discrimination result to the adaptive resolution convolutional Transformer generator; S240, determining a loss value between a source image in the sample data set and a target image generated by the fusion model from the source image, iteratively training the fusion model according to the loss value until the loss value is lower than a loss threshold, thereby obtaining a trained fusion model; In S240, the loss value is determined by: Construct a hybrid supervised loss function, which is: Among them, L A represents the mixed supervision loss function of the fusion model, represents the adversarial loss function between the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator, L FM (G,CTD n ) is the feature loss function on the convolutional Transformer discriminator at the nth resolution, L PL is the perceptual loss function, L SaC represents the spatial consistency loss function, L SeC represents the spectral consistency loss function, μ1, ρ2, λ3 are weight coefficients; The loss value of the source image and the image generated by the fusion model is calculated by the hybrid supervised loss function.

2. The method according to claim 1, characterized in that The convolutional Transformer discriminator includes a plurality of discriminators of different resolutions. In S230, the convolutional Transformer discriminator is used to antagonistically learn the local-global structure and spectral information of different resolutions and the output discrimination result is fed back to the adaptive resolution convolutional Transformer generator, including: S231, the convolutional Transformer discriminator sets a plurality of discriminators of corresponding resolutions according to the number of levels of the adaptive resolution convolutional Transformer generator, and the input of each of the discriminators is a down-resolution image and a predicted image of a corresponding resolution level, respectively; wherein the down-resolution image is generated by downsampling the reference image; and the predicted image includes a fused image of the corresponding resolution level generated by the adaptive resolution convolutional Transformer generator; S232, the first stage of the n-th resolution discriminator in the convolutional Transformer discriminator extracts primary local-global features of the n-th resolution reference image and the predicted image by corresponding convolution and MDCT, the middle stage extracts deep features of the n-th resolution reference image and the predicted image by corresponding convolution and MDCT and reduces the size of the features in sequence, and the last stage outputs the discrimination result by convolution and Sigmoid function; S233, feeding back the discrimination result to the adaptive resolution convolutional Transformer generator to generate a more refined fused image through adversarial learning.

3. The method according to claim 1, characterized in that The adversarial loss function between the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator is: And there are: Among them, L CTUGAN (G,CTD n ) represents the adversarial learning loss between the generator of the fusion model and the discriminator of the nth resolution, G represents the adaptive resolution convolutional Transformer generator of the fusion model, CTD n represents the convolutional Transformer discriminator of the nth resolution, represents the adversarial loss between the adaptive resolution convolutional Transformer generator and the convolutional Transformer discriminator, represents the total adversarial loss of the convolutional Transformer discriminator, Represents the convolutional Transformer discriminator CTD n The relative probability obtained, σ represents the Sigmoid activation function, C n (·) represents the untransformed convolutional Transformer discriminator output, and R represents the reference image R at the nth resolution n The distribution of R0 image has the highest spatial resolution, and P represents the generated image P of the nth resolution. n The distribution of and Represents the average operation of all reference images and generated images in the batch data.

4. The method according to claim 3, characterized in that The feature loss function on the convolutional Transformer discriminator is: Among them, L FM (G,CTD n ) is the feature loss function on the CTDn of the convolutional Transformer discriminator, Z is the total number of layers, is the CTD of the convolutional Transformer discriminator n The first layer features, M l is the total number of features in layer l.

5. A multi-source remote sensing image fusion system, characterized in that: The system comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the multi-source remote sensing image fusion method as described in any one of claims 1 to 4.

Images