Multi-task collaborative remote sensing image time-space-spectrum fusion method

Abstract

The invention relates to a multi-task cooperative remote sensing image time-space-spectrum fusion method, which comprises the following steps: establishing a coupled space-spectrum joint enhancement task and a time change prediction task, and carrying out initial fusion on original hyperspectral and multispectral images to obtain initial features; constructing a spatial enhancement unit, an error correction unit and an error correction unit, carrying out a space-spectrum joint enhancement task, constructing an alternate scale projection module, injecting multispectral information into a hyperspectral image, and carrying out space-spectrum joint enhancement to obtain a fused image; a time change prediction task is carried out, a spectrum fine tuning network is constructed, a hyperspectral image is modulated into a target time domain, change area perception is enhanced by means of a differential image, and a final time-space-spectrum fusion result is reconstructed. The method has the beneficial effects that the information reduction capability under heterogeneous land coverage is enhanced, the cooperative processing requirements of time, space and spectral information in a complex scene are met, and remote sensing image data with high time-space-spectral resolution is produced.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing, and in particular relates to a multi-task collaborative remote sensing image time-space-spectral fusion method. Background Art

[0002] my country's launch of earth observation remote sensing satellites has entered a period of rapid growth. Remote sensing images with high spatial, high spectral and high temporal resolution play an important role in environmental monitoring, smart agriculture, resource survey and other fields. However, due to the limitations of satellite sensor hardware and imaging conditions, there are inherent contradictions in the spatial, spectral and temporal resolution of a single remote sensing system, which limits its dynamic and precise monitoring potential in complex surface environments. Image fusion technology aims to integrate the complementary advantages of multi-source remote sensing data resolution, generate remote sensing images with high spatial, high spectral and high temporal resolution, and provide accurate and reliable data support for observation applications.

[0003] At present, a large number of remote sensing image fusion methods have been proposed, most of which can be basically divided into three categories: space-spectrum fusion, time-space fusion, and time-space-spectrum fusion. The first two methods have been extensively studied and have improved the quality of remote sensing imaging to a certain extent, but they only take into account the pairwise fusion between spatial, temporal and spectral resolutions. The development of various fusion technologies is relatively independent, which makes it difficult to meet the needs of complementary observations between multiple sensors.

[0004] The time-space-spectral fusion technology can comprehensively utilize multi-source and multi-temporal data to generate remote sensing images with high time-space-spectral resolution. It is mainly divided into two types of fusion methods: step-by-step time-space-spectral fusion method and integrated time-space-spectral fusion method. The step-by-step time-space-spectral fusion method is simple and intuitive. Its core is to execute multiple independent enhancement or fusion methods, and finally achieve a comprehensive improvement in spatial, temporal and spectral resolution by gradually improving the resolution. However, the step-by-step method is prone to error accumulation between subtasks, and it is difficult to obtain the global optimal solution. There is still a lot of room for improvement in the framework. The integrated time-space-spectral fusion method establishes a complete mapping relationship between multi-source observation images and ideal images, and simultaneously improves the spatial, temporal and spectral resolution of the images. The method is more efficient and has certain advantages over the distributed method. However, the existing remote sensing image time-space-spectral fusion method performs poorly in solving nonlinear spatial, temporal and spectral fidelity, is limited to processing seasonal spectral changes, and is difficult to apply to the reconstruction of heterogeneous surface observation information. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a multi-task collaborative remote sensing image time-space-spectral fusion method.

[0006] This multi-task collaborative remote sensing image time-space-spectral fusion method includes the following steps:

[0007] S1. Establish a multi-source and multi-temporal time-space-spectral fusion dataset;

[0008] S2, establish a coupled spatial-spectral joint enhancement task and a temporal change prediction task, and initially fuse the original hyperspectral and multispectral images to obtain initial features;

[0009] S3, construct spatial enhancement unit, error correction unit and error correction unit, perform spatial-spectral joint enhancement task, construct alternating scale projection module, inject multispectral information into hyperspectral image, and obtain spatial-spectral joint enhanced fused image;

[0010] S4. Carry out the task of temporal change prediction, build a spectral fine-tuning network, modulate the hyperspectral image into the target time domain, and use the differential image to enhance the perception of the changed area and reconstruct the final fusion result of time, space and spectrum.

[0011] Preferably, step S3 includes, S3.1, a spatial enhancement unit upsamples the initial features and cascades them with multi-spectral features, utilizes a channel attention mechanism to simulate spectral features, and then enhances the channel features through weighted vectors to reconstruct reasonable spectral information, and updates them through dense connections to obtain refined features; S3.2, an error correction unit resamples the refined features to obtain a low-resolution version and a blurred version, generates high-frequency texture details by difference, and downsamples them and feeds them back to the original refined feature map, and updates them through dense connections to obtain optimized fusion results; S3.3, an error correction unit obtains high-resolution features, connects the high-resolution features as residual components with the bicubic interpolation image, and obtains the result of spatial-spectral joint enhancement.

[0012] Preferably, in step S4, a weight generator of an attention module with residual blocks and convolution blocks is constructed to learn the deep low-rank spectral information of the target phase multi-spectrum, and then a global average pooling layer is used to abstract the affine function parameters from the feature domain, and the high-dimensional redundant features of the spatial-spectral joint enhancement output are spectrally reduced and expanded to the original dimension after affine transformation.

[0013] Preferably, in step S4, overlapping intervals and non-overlapping intervals are divided according to the spectral coverage, and the highly complementary spectral information dimension is expanded in the overlapping interval by gating, and the extended features are projected to the non-overlapping interval. The hyperspectral image is modulated into the target time domain through the spectral fine-tuning network, and cascaded with the multispectral image of the target phase to obtain a preliminary fusion result of the spatial and spectral information; the differential image is introduced into the preliminary fusion result, and the Sigmoid nonlinear function is added to suppress the low-correlated background noise in the differential image to obtain an optimized fusion result.

[0014] Preferably, in step S4, the optimized fusion result is projected into the deep feature space through three groups of densely connected convolution blocks, the attention module of the convolution block is further used to refine the spatial and spectral information, and the spectral channel is reconstructed using a convolution layer of size 1×1, so as to obtain the fusion result of time change prediction, and the fusion result of time change prediction is the final fusion result of time-space-spectrum.

[0015] Preferably, there is a step S5 after step S4. Specifically, step S5 is to optimize the final fusion result of the time-space-spectrum reconstructed in step S4 by using a weighted loss function.

[0016] The beneficial effects of the present invention are:

[0017] 1) The present invention constructs a coupling framework of the space-spectrum joint enhancement task and the time change prediction task, and proposes a dual-task collaborative time-space-spectrum fusion model for remote sensing images. This model can overcome the technical bottleneck of time-space-spectrum fusion in existing methods that is difficult to achieve in scenes with complex time transformation relationships, enhance the ability to restore information under heterogeneous land cover, meet the needs of coordinated processing of time, space and spectral information in complex scenes, and produce remote sensing image data with high time-space-spectrum resolution.

[0018] 2) The dual-task collaborative time-space-spectral fusion model of remote sensing images constructed by the present invention can effectively improve the fidelity of time-space-spectral fusion in heterogeneous surface coverage. By constructing an alternating scale projection module in the space-spectral joint enhancement task, multispectral information is injected into the hyperspectral image with low loss, generating a fused image with high spatial and high spectral resolution. Secondly, a spectral fine-tuning network is constructed in the time change prediction task to modulate the hyperspectral image into the target time domain, and the perception of the changed area is enhanced with the help of differential images to reconstruct a remote sensing image with high time-space-spectral resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of the time-space-spectral fusion method of remote sensing images for dual-task collaboration;

[0020] Figure 2 It is a structural diagram of the alternating scale projection module;

[0021] Figure 3 This is a schematic diagram of the structure of the spectrum fine-tuning network module;

[0022] Figure 4 This is the fusion result diagram on the DY simulation data set;

[0023] Figure 5 This is the fusion result diagram on the YRE simulation dataset;

[0024] Figure 6 This is the fusion result diagram on the DY real data set;

[0025] Figure 7 This is the fusion result diagram on the YRE real dataset. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0027] Embodiment 1

[0028] As an example, Figures 1 to 3 As shown in the figure, this multi-task collaborative remote sensing image time-space-spectral fusion method enhances the information restoration capability under heterogeneous land cover by coupling the space-spectrum joint enhancement task and the time change prediction task, and realizes the time-space-spectrum fusion under complex time transformation relationship scenes. The system used establishes a multi-source and multi-temporal time-space-spectral fusion dataset, which contains multispectral and hyperspectral images with different imaging time differences; including:

[0029] Establish a module for coupling the spatial-spectral joint enhancement task and the temporal change prediction task, and perform initial fusion of raw hyperspectral and multispectral images;

[0030] The alternating scale projection module is used to inject multispectral information into hyperspectral images with low loss in the spatial-spectral joint enhancement task, generating fused images with high spatial and spectral resolution;

[0031] The spectral fine-tuning network module is used to modulate the hyperspectral image into the target time domain in the task of temporal change prediction, and to enhance the perception of the changing area with the help of differential images to reconstruct high temporal-spatial-spectral resolution remote sensing images.

[0032] This multi-task collaborative remote sensing image time-space-spectral fusion method includes the following steps:

[0033] Step S1: establishing a multi-source and multi-temporal time-space-spectrum fusion dataset, wherein the time-space-spectrum fusion dataset includes multispectral and hyperspectral images with different imaging time differences.

[0034] Specifically, the present invention establishes a new time-space-spectral fusion dataset, by removing low-quality bands from the original images and degrading the original multispectral and hyperspectral images, the original dataset images are trimmed into 72×72 multispectral image blocks and 24×24 hyperspectral image blocks, and a dataset that can be used for network learning is constructed.

[0035] Step S2: establish a coupled spatial-spectral joint enhancement task and a temporal change prediction task, and perform an initial fusion of the original hyperspectral and multispectral images.

[0036] like Figure 1 As shown, the spatial-spectral joint enhancement task SSF assists the phase to generate a remote sensing image F1 with high spatial and high spectral resolution, while the temporal change prediction task TSF generates a remote sensing image F2 with high temporal-spatial-spectral resolution by interacting the fused image with the multispectral of the target phase. For the temporal-spatial-spectral fusion task, the spatial-spectral joint enhancement task aims to enhance the spatial quality under high spectral fidelity. Therefore, for the consideration of computational complexity and spectral quality, the present invention conservatively injects spatial information at the original high spectral resolution, downsamples the original scale multispectral convolution to high spectral resolution, and then maps the feature map to a feature space of size h×w×B, and the hyperspectral also performs similar operations.

[0037] The expression of step S2 is:

[0038] F1=SSF(MS1,HS1)

[0039] F2=TSF(MS1,MS2,F1)

[0040] F m =Cov 3,3 (δ(d(MS1)))

[0041] F h =Cov 3,3 (δ(Cov 3,3 (HS1)))

[0042] L0=Cov 3,3 (Concat(F h ,F m ))

[0043] Among them, SSF(·) and TSF(·) represent the spatial-spectral joint enhancement and temporal change prediction tasks, respectively, F1 and F2 represent the corresponding output results; MS1 and MS2 represent multispectral images, HS1 represents hyperspectral images, d(·) represents convolution downsampling, Cov 3,3 (·) represents a 3×3 convolutional layer, δ(·) represents a linear rectification function, and F m and F h represents the features generated by multispectral and hyperspectral; Concat(·) represents the feature cascade of channel dimension, and L0 represents the initial features generated by fusion.

[0044] Step S3: In the spatial-spectral joint enhancement task, an alternating scale projection module is constructed to inject multispectral information into the hyperspectral image with low loss, and a fused image with high spatial and high spectral resolution is generated as the output result F1 of the spatial-spectral joint enhancement.

[0045] Specifically, referring to the idea of ​​deep back-projection network, the present invention designs an alternating scale projection module SAP to realize cross-scale fusion iteration, retain the real space-spectrum structure and learn the scale degradation relationship. Among them, L0 inputs the SAP module, and in the forward propagation, the spatial details of the multi-spectrum are injected into the feature domain through the spatial enhancement unit SEU; in the backward propagation, the error correction unit ECU provides error feedback and scale degradation perception for the fusion. Taking the nth iteration as an example, the structural details of SEU and ECU are as follows Figure 2 shown.

[0046] Step S4: In the task of temporal change prediction, a spectral fine-tuning network is constructed to modulate the hyperspectral image into the target time domain, and the perception of the changing area is enhanced with the help of differential images to reconstruct high temporal-spatial-spectral resolution remote sensing images.

[0047] Spectral changes reflect the disorder and order of temporal changes under the influence of surface cover changes and seasonal factors. For orderliness, the output result F1 of the joint spatial-spectral enhancement is used as a spectral prior to provide high-dimensional spectral structure information; for disorder, the multi-spectrum of the target phase provides real spatial information and temporal change information. In addition, since the spectral coverage of the multi-spectrum does not completely overlap with the hyperspectrum, unknown spectral content can be inferred from the overlapping spectral region. The temporal change prediction task utilizes the low-rank information of the multi-spectrum of the target phase to generate a learnable affine function to modulate the F1 spectral structure value, whose structure is as follows: Figure 3 shown.

[0048] Embodiment 2

[0049] As another embodiment, based on the first embodiment, the second embodiment proposes a more specific multi-task collaborative remote sensing image time-space-spectral fusion method, and step S3 is specifically as follows:

[0050] Step S3 includes:

[0051] Step S3.1, construct a spatial enhancement unit, upsample the initial features generated by fusion and cascade them with multi-spectrum to utilize the overall information. However, traditional spatial information replacement will cause spectral distortion, so the spatial enhancement unit uses the channel attention mechanism to simulate spectral features and enhance the region of interest in the channel dimension of the fused features, and then injects the spatial components of the multi-spectrum into the fused features as supplementary information, and enhances the channel features through weighted vectors to reconstruct reasonable spectral information. In addition, the module imitates the idea of ​​dense connection and regards the output of the spatial enhancement unit in the previous stage as a residual output to ensure the stable propagation of deep gradients.

[0052] The expression of step S3.1 is:

[0053] f p =Cov 3,3 (Concat(MS1,u(Ln -1 )))

[0054] w=σ(f c2 (δ(f c1 (gap(L n-1 )))))

[0055] H n =w·(f p +1)

[0056] H n =H1+H2+…+H n-1 +H n

[0057] Among them, u(·) represents the upsampling operator, f p represents the fusion feature after the initial feature and the multispectral image cascade; w = {w1,w2,…,w B}∈[0,1] represents the weight of each channel, σ(·) represents the Sigmoid nonlinear function, gap(·) represents the global average pooling layer, and f c1 (·) and f c2 (·) indicates a fully connected mapping; H n Represented as features refined by attention.

[0058] Step S3.2: construct an error correction unit, use the error back propagation mechanism, and realize self-correction of the fusion feature by perceiving the high-frequency information of the enhanced area. Specifically, the refined features generated in step S2.1 are resampled to obtain a low-resolution version and a blurred version, and high-frequency texture details are generated by subtraction, and then downsampled and fed back to the original refined feature map to obtain an optimized fusion result, which is also updated through dense connections.

[0059] The expression of step S3.2 is:

[0060] H n ↓=d(H n )

[0061] H n ↓↑=d(δ(H n ↓))

[0062] r n =H n -H n ↓↑

[0063] L n =H n ↓+d(r n )

[0064] L n =L1+L2+…+L n-1 +L n

[0065] Among them, H n ↓ and H n ↓↑ respectively represent H n Low-resolution and blurred versions of n Represents high-frequency texture details; L n Represents the optimized fusion result.

[0066] Step S3.3, after the alternating scale projection module iterates n times to obtain the optimized fusion result, a separate spatial enhancement unit is executed to obtain high-resolution features, which are connected to the bicubic interpolation image as residual components to reduce the spectral deviation of the fused image, and finally the result of spatial-spectral joint enhancement is obtained.

[0067] The expression of step S3.3 is:

[0068] F1=Cov 1,1 (H n+1 )+HS1↑

[0069] Among them, F1 represents the output result of spatial-spectral joint enhancement, H n+1 Represented as high-resolution features, Cov 1,1 (·) indicates a 1×1 convolutional layer, and HS1↑ indicates a bicubic interpolated image.

[0070] It should be noted that the parts in this embodiment that are the same or similar to those in the first embodiment can be referenced to each other and will not be described in detail in this application.

[0071] Embodiment 3

[0072] As another embodiment, this embodiment 3 proposes a more specific multi-task collaborative remote sensing image time-space-spectral fusion method based on the first and second embodiments, and step S4 is specifically as follows:

[0073] Step S4.1: Construct a weight generator of the attention module with residual blocks and convolution blocks to learn the deep low-rank spectral information of the target phase multi-spectrum, and then use the global average pooling layer to abstract the affine function parameters from the feature domain. Considering the low rank of the hyperspectral, the high-dimensional redundant features of the spatial-spectral joint enhancement output are spectrally reduced and expanded to the original dimension after affine transformation.

[0074] The expression of step S4.1 is:

[0075]

[0076] f(F1)=Cov 1,1 ((α+1)·Cov 1,1 (F1)+β)

[0077] Among them, Res1(·) and Res2(·) represent residual convolution, CBAM1 and CBAM2 represent attention modules, represents the generated feature map; ρ(·) represents the channel alignment segmentation, α and β represent the scaling factor and displacement factor of the affine function respectively; Cov 1,1 (·) represents the channel expansion and compression convolution of size 1×1, and f(F1) represents the hyperspectral time transformation of the output result of the joint spatial-spectral enhancement.

[0078] Step S4.2: Divide the spectral coverage into overlapping and non-overlapping intervals, use the correlation between spectra to infer the non-overlapping content, and then introduce the hyperspectral prior information of the auxiliary phase to solve the orderly seasonal spectral changes. Specifically, in the overlapping interval, the highly complementary spectral information dimension is expanded through gating, and then the expanded features are projected to the non-overlapping interval, and then the fine-tuned features are cascaded with the multispectral image of the target phase to obtain the preliminary fusion result of spatial and spectral information.

[0079] The expression of step S4.2 is:

[0080]

[0081] in, and represent overlapping and non-overlapping spectral bands respectively, represents the projection of the extended feature on the non-overlapping interval; g(·) represents the nonlinear time transformation, represents the spectral band of the output result of the overlapped spatial-spectral joint enhancement, Represents the preliminary fusion results of time-varying predictions.

[0082] Step S4.3: In order to perceive irregular changes in ground object types, differential images are introduced into the preliminary fusion results to highlight the changing areas between time phases and further improve the details of the spectrum. Considering that the differential image has a low signal-to-noise ratio and artifacts are easily generated in small units, the Sigmoid nonlinear function is added to suppress the low-correlation background noise in the differential image to obtain the optimized fusion result.

[0083] The expression of step S4.3 is:

[0084] s=Cov 3,3 (σ(MS2-MS1))

[0085]

[0086] Where s represents the weight for guiding the perception of surface changes, Represents the optimized fusion result.

[0087] Step S4.4: Project the optimized fusion result obtained in step S3.3 into the deep feature space through three groups of densely connected convolution blocks. In order to reconstruct a fine image from the semantic features, the attention module of the convolution block is further used to refine the spatial and spectral information, and a convolution layer of size 1×1 is used to reconstruct the spectral channel, thereby obtaining the fusion result of time change prediction, which is also the final fusion result of time-space-spectrum.

[0088] The expression of step S4.4 is:

[0089]

[0090] Among them, Dens i (·), i = {1, 2, 3} is a densely connected convolution block with a kernel size of 3, and F2 represents the final fusion result of the time-varying prediction task and the time-space-spectrum.

[0091] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 and 2 can be referenced to each other and will not be described in detail in this application.

[0092] Embodiment 4

[0093] As another embodiment, this embodiment 4 uses this multi-task collaborative remote sensing image time-space-spectral fusion method on the basis of embodiments 1 to 3 to process two basic data sets.

[0094] The two basic datasets are: the Dongying DY dataset located in the lower reaches of the Yellow River in Shandong Province and the YRE dataset located at the mouth of the Yellow River in Shandong Province, both of which include Sentinel-2 multispectral images on February 10, 2021 and December 8, 2019; and ZY-1 02D hyperspectral images on February 10, 2021 and December 11, 2019. After removing low-quality bands from the original images, 4-band 10-meter multispectral images and 151-band 30-meter hyperspectral images were obtained.

[0095] Step S1: establishing a multi-source and multi-temporal time-space-spectrum fusion dataset, wherein the time-space-spectrum fusion dataset includes multispectral and hyperspectral images with different imaging time differences.

[0096] Step S2: The newly created data set is cropped, normalized, and other processes are performed to expand the data set, so as to avoid problems such as overfitting and gradient explosion caused by too large original data and small data volume during deep learning training.

[0097] Specifically, in this embodiment, multispectral and hyperspectral images for training and testing are simulated on single hyperspectral images of DY and YRE based on the Wald protocol, and the original hyperspectral images are used as reference. Taking the DY dataset as an example, in the spatial dimension, the original multispectral and hyperspectral images are degraded, blurred using a Gaussian filter with a kernel size of 7×7 and a standard deviation of 5, and then downsampled at 3 times the spatial resolution. Finally, in this embodiment, about 4,500 pairs of multispectral / hyperspectral image blocks are collected in the DY dataset, and nearly 2,000 pairs of image blocks are sorted in the YRE dataset. In this embodiment, training and testing are performed on NVIDIA GeForce RTX 3090 GPU and Intel i7-11700K CPU.

[0098] Step S3, since the true spectrum of the target phase is unknown, it can only refer to low-rank multispectral images or hyperspectral images of the auxiliary phase, so the time-space-spectral fusion is decomposed into the interaction of two tasks: the joint enhancement and migration of the space-spectral information of the auxiliary phase and the spectral super-resolution of the target phase. Among them, the multispectral image is not full rank in the spectral dimension, and a large amount of missing information depends on the latter to be completed. The present invention establishes a coupled space-spectral joint enhancement task and a time change prediction task, divides the time change problem into ordered seasonal spectral changes and disordered surface cover changes, and performs initial fusion of the original hyperspectral and multispectral images.

[0099] Step S4: In the spatial-spectral joint enhancement task, an alternating scale projection module is constructed to achieve high-fidelity hyperspectral image spatial information enhancement. First, a spatial enhancement unit is constructed to upsample the initial features generated by fusion and concatenate them with multi-spectral cascades, and the channel attention mechanism is used to simulate the spectral features. Then, the channel features are enhanced to reconstruct reasonable spectral information to obtain refined features; secondly, an error correction unit is constructed to resample the refined features to obtain low-resolution versions and blurred versions, and then optimize them by feedback fusion errors; finally, a separate error correction unit is executed to obtain high-resolution features, which are connected to the bicubic interpolation image as residual components to generate a fused image with high spatial and high spectral resolution.

[0100] Step S5: In the task of temporal change prediction, a spectral fine-tuning network is constructed, and the low-rank information of the multi-spectral image of the target phase is utilized. The spectral numerical distribution generated by the spatial-spectral fusion in step S4 is modulated with learnable weights, and the spectral dimension reduction is performed through global average pooling; secondly, the spectrum is divided into overlapping and non-overlapping areas, and the information is expanded in the overlapping area, and projected to the non-overlapping area and cascaded with the multi-spectral image of the target phase to obtain a preliminary fusion result; differential images are introduced to highlight the changing areas between phases for further optimization; finally, deep features are projected through dense convolution blocks to refine spatial and spectral information, and a 1×1 convolution layer is used to reconstruct the spectral channel to obtain the final fusion result.

[0101] Step S6: According to the supervised deep network training strategy, a weighted loss model is established using mean absolute error loss, spatial loss and spectral loss functions to optimize the network. First, the mean absolute error loss is used to measure the absolute average of the difference between the fused image and the real image, so that the spatial texture details of the fusion result are closer to the real value; secondly, in order to better reconstruct the edge details of the image, the model introduces spatial loss to retain more high-frequency information; finally, the spectral loss constrains the similarity of spectral vectors to ensure the color authenticity and spectral structure of the fused image in the spectral dimension.

[0102] In step S6, the expression of the weighted loss function is:

[0103] L MAE1 =‖X1-F1‖1

[0104] L MAE2 =‖X2-F2‖1

[0105] L spa =‖g(X2)-g(F2)‖2

[0106]

[0107] L=λ1L MAE1 +λ2L MAE2+λ3L spa +λ4L spe

[0108] Among them, L MAE1 and L MAE2 All represent mean absolute error loss, X1 and X2 represent real images, F1 and F2 represent fused images output by spatial-spectral joint enhancement and temporal change prediction tasks; L spa represents the spatial loss, g(·) is the 3×3 Laplace operator; L spe represents spectral loss, H and W represent the number of horizontal and vertical pixels, <·,·> represents the vector inner product, and represents the spectral vector of the pixel (h, w) position; λ1, λ2, λ3 and λ4 represent the weights of each constraint, which are set to 0.2, 1, 0.0001 and 0.001 respectively.

[0109] In this embodiment, this method is verified through experiments:

[0110] The comparative experiments were conducted on low-resolution simulated datasets and full-resolution real datasets according to the different resolutions of the datasets. Three common spectral super-resolution methods (Arad, A+ and J-SLoL) and two deep learning-based spatiotemporal fusion methods (GLMM-STSF and USTSFM) were compared with the proposed method. The results are shown in Figure 2. Figures 4 to 7 shown.

[0111] First, we conduct experiments on a simulated dataset. Figure 4 and Figure 5 The following are the fusion results on the DY simulated dataset and the YRE simulated dataset. Figure 4 and Figure 5 (a) and (h) are the original hyperspectral images of the auxiliary phase and the target phase, respectively, and (b)-(g) are the fused images based on Arad, A+, J-SLoL, GLMM-STSF, USTSFM and the method proposed in this invention, respectively.

[0112] Figure 4 It can be seen that Arad has serious spatial blur, while A+ and USTSFM are accompanied by obvious spectral differences. J-SLoL and GLMM-STSF have poor reconstruction accuracy in vegetation areas. In this embodiment, the visual error between the fusion result of the proposed method and the real image is very ideal, and the drastic and irregular spectral variation within the stage can be accurately predicted. Figure 5It can be seen that Arad, J-SLoL and the proposed method can still show relatively fine spectra, but Arad cannot solve the blurring problem, while other methods will cause obvious distortion. To a certain extent, the former proves its better robustness. The method proposed in this paper not only achieves spectral fidelity across time and space, but also effectively utilizes the spectral prior of the auxiliary phase as a constraint.

[0113] Secondly, experiments are conducted on real datasets. Figure 6 and Figure 7 The fusion results are respectively performed on the DY simulated dataset and the YRE simulated dataset. The specific images represented by (a)-(h) are consistent with those of the simulated dataset.

[0114] Figure 6 and Figure 7 It can be clearly seen that, whether it is the spectral super-resolution method or the method based on deep learning, it is difficult to simultaneously achieve spatial detail reconstruction and spectral fidelity of the fused image in the face of scenes with complex temporal transformation relationships. The method proposed in the present invention can comprehensively improve the image quality and maintain stable performance in heterogeneous object coverage. The comprehensive results show that the dual-task collaborative time-space-spectral fusion method proposed in the present invention can meet the needs of coordinated processing of time, space and spectral information in complex scenes, and has strong practicality.

[0115] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 to 3 can be referenced to each other and will not be described in detail in this application.

[0116] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

Claims

1. A multi-task collaborative remote sensing image time-space-spectral fusion method, characterized in that: The following steps are involved: S1. Establish a multi-source and multi-temporal time-space-spectral fusion dataset; S2, establish a coupled spatial-spectral joint enhancement task and a temporal change prediction task, and initially fuse the original hyperspectral and multispectral images to obtain initial features; S3, construct spatial enhancement unit, error correction unit and error correction unit, perform spatial-spectral joint enhancement task, construct alternating scale projection module, inject multispectral information into hyperspectral image, and obtain spatial-spectral joint enhanced fused image; S4. Carry out the task of temporal change prediction, build a spectral fine-tuning network, modulate the hyperspectral image into the target time domain, and use the differential image to enhance the perception of the changed area and reconstruct the final fusion result of time, space and spectrum.

2. The multi-task collaborative remote sensing image time-space-spectral fusion method according to claim 1, characterized in that: Step S3 includes: S3.1, the spatial enhancement unit upsamples the initial features and cascades them with multi-spectral features, uses the channel attention mechanism to simulate the spectral features, and then enhances the channel features through weighted vectors to reconstruct reasonable spectral information, and updates them through dense connections to obtain refined features; S3.2, the error correction unit resamples the refined features to obtain a low-resolution version and a blurred version, generates high-frequency texture details by difference, and downsamples them and feeds them back to the original refined feature map, and updates them through dense connections to obtain optimized fusion results; S3.3, the error correction unit obtains high-resolution features, connects the high-resolution features as residual components with the bicubic interpolation image, and obtains the result of spatial-spectral joint enhancement.

3. The multi-task collaborative remote sensing image time-space-spectral fusion method according to claim 1, characterized in that: In step S4, a weight generator of an attention module with residual blocks and convolution blocks is constructed to learn the deep low-rank spectral information of the target phase multi-spectrum, and then a global average pooling layer is used to abstract the affine function parameters from the feature domain, and the high-dimensional redundant features of the spatial-spectral joint enhancement output are spectrally reduced and expanded to the original dimension after affine transformation.

4. The multi-task collaborative remote sensing image time-space-spectral fusion method according to claim 1, characterized in that: In step S4, overlapping intervals and non-overlapping intervals are divided according to the spectral coverage. The highly complementary spectral information dimension is expanded in the overlapping interval by gating, and the extended features are projected to the non-overlapping interval. The hyperspectral image is modulated into the target time domain through the spectral fine-tuning network, and cascaded with the multispectral image of the target phase to obtain a preliminary fusion result of spatial and spectral information; the differential image is introduced into the preliminary fusion result, and the Sigmoid nonlinear function is added to suppress the low-correlated background noise in the differential image to obtain the optimized fusion result.

5. The multi-task collaborative remote sensing image time-space-spectral fusion method according to claim 4, characterized in that: In step S4, the optimized fusion result is projected into the deep feature space through three groups of densely connected convolution blocks, and the attention module of the convolution block is further used to refine the spatial and spectral information, and the spectral channel is reconstructed using a convolution layer of size 1×1 to obtain the fusion result of time change prediction. The fusion result of time change prediction is the final fusion result of time-space-spectrum.

6. The multi-task collaborative remote sensing image time-space-spectral fusion method according to claim 1, characterized in that: There is also a step S5 after step S4. Specifically, step S5 is to optimize the final fusion result of the time-space-spectrum reconstructed in step S4 through a weighted loss function.

Images