A multi-dimensional remote sensing data fusion method based on multi-task learning

Abstract

The invention discloses a multi-dimensional remote sensing data fusion method based on multi-task learning, and relates to the field of remote sensing technology. The multi-dimensional remote sensing data fusion method based on multi-task learning comprises the following steps: Step 101, based on a time series high spatial resolution data set, a time series high temporal resolution data set and a time series high spectral resolution data set to be fused, the data are sorted and stored according to the time dimension, the spatial dimension X, the spatial dimension Y and the spectral dimension. Compared with the prior art, the beneficial effects of the invention are as follows: the invention constructs a spatial information extraction model, a temporal information extraction model and a spectral information extraction model by constructing four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data, and utilizes a multi-task learning multi-feature loss function design to realize the joint extraction and fusion of time, space and spectral dimension information, thereby realizing integrated fusion in multiple dimensions and improving the efficiency and speed of data fusion.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a multi-dimensional remote sensing data fusion method based on multi-task learning. Background Art

[0002] With the continuous development of remote sensing technology, a large number of remote sensing data sets with different temporal, spatial and spectral resolutions have been accumulated, providing a solid data foundation for applications. Among them, hyperspectral has been widely used in the field due to its rich spectral information. However, due to the limitations of satellite sensors, remote sensing data sets cannot have high temporal, spatial and spectral resolutions at the same time. Especially for hyperspectral data, its temporal resolution and spatial resolution are often low, which greatly limits the application of hyperspectral data in long time series. Therefore, how to combine the advantages of different remote sensing products to obtain a data set with high temporal, spatial and spectral resolution has become an urgent problem to be solved.

[0003] In order to obtain a data set with high spatiotemporal resolution, data fusion is an effective choice to achieve this goal. Current data fusion methods include spatiotemporal fusion methods, space-spectrum fusion methods, and space-time spectrum fusion methods. Spatiotemporal fusion methods usually include weighted function methods, linear optimization decomposition methods, and nonlinear optimization methods; space-spectrum data fusion methods usually include linear optimization decomposition methods and nonlinear optimization methods. Spatiotemporal spectrum fusion methods currently include spatiotemporal spectrum fusion methods based on maximum a posteriori estimation and spatiotemporal spectrum fusion methods based on coupled sparse tensor decomposition.

[0004] However, existing data fusion methods often perform fusion and reconstruction through a single phase or multiple discrete phases, and are unable to realize integrated extraction and fusion of the time dimension, spatial dimension, and spectral dimension of the data set to be fused in a multidimensional data structure. Therefore, the fusion speed and efficiency are insufficient, and it is difficult to apply to the generation of large-scale long-time series hyperspectral data sets, which needs to be improved. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-dimensional remote sensing data fusion method based on multi-task learning to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-dimensional remote sensing data fusion method based on multi-task learning includes the following steps:

[0008] Step 101, based on the time series high spatial resolution data set to be fused, the time series high temporal resolution data set and the time series high spectral resolution data set are sorted and stored according to the time dimension, spatial dimension X, spatial dimension Y and spectral dimension, four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data are obtained, and temporal phase encoding is performed; the data source includes the time series high spatial resolution data, the time series high temporal resolution data and the time series high spectral resolution data, wherein the time series high spatial resolution data refers to the data with the highest spatial resolution among the three data sources, the time series high temporal resolution data refers to the data with the highest temporal resolution among the three data sources, and the time series high spectral resolution data refers to the data with the highest spectral resolution among the three data sources;

[0009] Step 102, the four-dimensional high time resolution data to be fused is downsampled in time dimension to construct corresponding four-dimensional time dimension downsampled multispectral data;

[0010] Step 103, based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spatial resolution data, the four-dimensional time dimension downsampled multispectral data and the four-dimensional high temporal resolution data, and the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spectral resolution data, respectively construct spatial, temporal, and spectral information extraction models;

[0011] Step 104, combining the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model through multi-task learning to jointly train and learn multi-source time-space-spectral information;

[0012] Step 105, connecting the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model in series, inputting the data to be fused, and obtaining four-dimensional high-temporal-high-spatial-high-spectral data;

[0013] In the time, space, and spectrum, the "high" refers to the data with the highest resolution among the three types of data, the "medium" refers to the data with the second highest resolution among the three types of data, and the "low" refers to the data with the third highest resolution among the three types of data.

[0014] As a further solution of the present invention: the step 101, based on the time series high spatial resolution data set to be fused, the time series high temporal resolution data set and the time series high spectral resolution data set are sorted and stored according to the time dimension, the spatial dimension X, the spatial dimension Y and the spectral dimension to obtain four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data, before phase encoding, also includes:

[0015] Step 1001, using a panchromatic or multispectral, multispectral and hyperspectral sensor to acquire original panchromatic or multispectral data with a time series of low temporal-high spatial resolution, original multispectral data with a time series of high temporal-medium spatial-low spectral resolution, and original hyperspectral data with a time series of low temporal-low spatial-high spectral resolution;

[0016] Step 1002, preprocessing the original panchromatic or multispectral data of the time series low temporal-high spatial resolution, the original multispectral data of the time series high temporal-medium spatial-low spectral resolution, and the original hyperspectral data of the time series low temporal-low spatial-high spectral resolution, respectively, to obtain corresponding multispectral / panchromatic data, multispectral reflectance data, and hyperspectral reflectance data;

[0017] Step 1003, accurately align all multispectral or panchromatic data, multispectral reflectance data and hyperspectral reflectance data to obtain the four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data.

[0018] As a further solution of the present invention: all four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data are phase-encoded, specifically including:

[0019] E standard =[1,2,...,L];

[0020]

[0021] Among them, E standard is the phase coding set of the time series data with the highest time resolution containing L scenes, and the imaging time of the mth scene image is Encoded as The phase encoding set of time series data from other sources is E N , the imaging time of the nth image is Encoded as

[0022] As a further solution of the present invention: using the phase coding set, the four-dimensional high time resolution data to be fused is quickly downsampled in time dimension to construct the corresponding four-dimensional time dimension downsampled multispectral data, specifically including:

[0023] For four-dimensional remote sensing data Use the following formula to calculate the corresponding time dimension downsampling data Right now

[0024]

[0025] Among them, E standardis the phase encoding set of time series data with the highest time resolution, E N Four-dimensional remote sensing data The phase coding set, x∈W, y∈H, b∈B.

[0026] As a further solution of the present invention: based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spatial resolution data, a spatial information extraction model is constructed, which specifically includes:

[0027] Spatial Dimension Mapping Features Extraction and utilization of four-dimensional high spatial resolution data and 4D time dimension downsampled multispectral data The relationship between spatial Extract

[0028]

[0029] As a further solution of the present invention: based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high time resolution data, a time information extraction model is constructed, which specifically includes:

[0030] Time dimension mapping features Extraction of high temporal resolution data using four-dimensional and 4D time dimension downsampled multispectral data The relationship between temporal To extract,

[0031]

[0032] As a further solution of the present invention: based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spectral resolution data, a spectral information extraction model is constructed, which specifically includes:

[0033] Spectral Dimension Mapping Features Extracting multispectral data using downsampling of the four-dimensional time dimension and four-dimensional high spectral resolution data The relationship between spectral To extract,

[0034]

[0035] As a further solution of the present invention: the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are combined by multi-task learning to jointly learn multi-source time-space-spectral information, specifically including:

[0036] The multi-feature loss function design for multi-task learning optimizes the combination of the target loss functions of all single-task feature network branches to achieve joint feature extraction and fusion of multi-task learning.

[0037] As a further solution of the present invention: the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are connected in series to obtain four-dimensional high-temporal-high-spatial-high-spectral data, specifically including:

[0038] Using four-dimensional high spatial resolution data and 4D time dimension downsampled multispectral data Obtain the four-dimensional high-temporal-high-spatial-high-spectral data Z to be reconstructed 4D .

[0039]

[0040] As a further solution of the present invention: multi-feature loss function design for multi-task learning, optimizing and combining the target loss functions of all single-task feature network branches to achieve joint feature extraction and fusion of multi-task learning, specifically including:

[0041] Through the design of multi-feature loss functions based on multi-task learning, the target loss functions of all single-task feature network branches are optimized and combined to achieve joint feature extraction and fusion of multi-task learning. Assume that the set of learnable parameters of the multi-task learning network is in Represents all weight parameters in the network. All input sets of the multi-task learning network are x, and the label set is All predicted label sets are For each single-task feature network branch, its loss function is L τ (x,y τ ,y' τ ;w τ ),in The total loss function of the multi-task learning network is designed as a linear combination of each single-task feature network branch, and a regularization term is added to obtain

[0042]

[0043] Among them, c τ is the weight of the learnable τ task feature network branch loss function.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention constructs four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data, constructs spatial information extraction models, temporal information extraction models and spectral information extraction models, and utilizes multi-task learning and multi-feature loss function design to realize the joint extraction and fusion of temporal, spatial and spectral dimension information, realizes integrated fusion in multiple dimensions, and improves the efficiency and speed of data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The schematic diagram of a multi-dimensional remote sensing data fusion method based on multi-task learning.

[0046] Figure 2 A schematic diagram of an embodiment of building a mapping model based on a multi-layer 3D convolutional neural network.

[0047] Figure 3 A schematic diagram of an embodiment of building a mapping model based on a multi-layer 2D convolutional neural network. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] See also Figure 1 , a multi-dimensional remote sensing data fusion method based on multi-task learning, comprising the following steps:

[0050] Step 101, based on the time series high spatial resolution data set to be fused, the time series high temporal resolution data set and the time series high spectral resolution data set, they are sorted and stored according to the time dimension, spatial dimension X, spatial dimension Y and spectral dimension to obtain four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data, and perform phase encoding; the time series high spatial resolution data refers to the data with the highest spatial resolution among the three data sources, which is low time, high space, low spectral resolution / full color data, the time series high temporal resolution data refers to the data with the highest temporal resolution among the three data sources, which is high time, medium space, low spectral resolution data, and the time series high spectral resolution data refers to the data with the highest spectral resolution among the three data sources, which is low space, low time, high spectral resolution data;

[0051] Step 102, the four-dimensional high time resolution data to be fused is downsampled in time dimension to construct corresponding four-dimensional time dimension downsampled multispectral data;

[0052] Step 103, based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spatial resolution data, the four-dimensional time dimension downsampled multispectral data and the four-dimensional high temporal resolution data, and the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spectral resolution data, respectively construct spatial, temporal, and spectral information extraction models;

[0053] Step 104, combining the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model through multi-task learning to jointly train and learn multi-source time-space-spectral information;

[0054] Step 105 , connect the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model in series, input the data to be fused, and obtain four-dimensional high-temporal-high-spatial-high-spectral data.

[0055] In the time, space, and spectrum, the "high" refers to the data with the highest resolution among the three types of data, the "medium" refers to the data with the second highest resolution among the three types of data, and the "low" refers to the data with the third highest resolution among the three types of data.

[0056] Specifically, the time series high spatial resolution data set, the time series high temporal resolution data set and the time series high spectral resolution data set to be fused are sorted and stored according to the time dimension, the space dimension and the spectrum dimension to obtain four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data. Among them, the time series high spatial resolution data set is low time, high spatial resolution, low spectral resolution / full color, the time series high temporal resolution data set is high time, medium space, low spectral resolution, and the time series high spectral resolution data set is low space, low time, high spectral resolution; all four-dimensional high spatial resolution data, four-dimensional high spatial resolution data and four-dimensional high spectral resolution data are phase encoded.

[0057] Specifically, the time series multispectral data and time series hyperspectral data can be integrated and stored separately to facilitate data extraction. For any multidimensional remote sensing data, it is divided by band to obtain the pixel data of each band, and then the pixel data of each band is stored in the order of the band. For the pixel data of any band, the pixel data of all times of the band are stored in sequence in the order of spatial rows first and columns later.

[0058] Taking the first band as an example, the pixel data of all times of the band can be stored in the order of spatial rows first and columns later. When storing, the pixel data of the earlier time is stored first, and the pixel data of the later time is stored later. For example, the pixel data of the first row and the first column can be stored in time order, and then the pixel data of the first row and the second column can be stored until the pixel data of all columns of the row are stored, and then the pixel data of the pixel data of the second row and the first column can be stored, and so on.

[0059] The fused four-dimensional high temporal resolution data set is downsampled in time dimension to construct the corresponding four-dimensional time dimension downsampled multispectral data. For example, the four-dimensional high temporal resolution data contains data from days 1 to 9, while the four-dimensional high spectral resolution data contains data from days 1, 3, 5 and 7. Then the time downsampling can be achieved by selecting the data from days 1, 3, 5 and 7 in the four-dimensional multispectral data.

[0060] Specifically, based on the mapping relationship between the four-dimensional time dimension down-sampled multispectral data and the four-dimensional high spatial resolution data, a spatial information extraction model is constructed; based on the mapping relationship between the four-dimensional time dimension down-sampled multispectral data and the four-dimensional high spatial resolution data, a time information extraction model is constructed; based on the mapping relationship between the four-dimensional time dimension down-sampled multispectral data and the four-dimensional high spectral resolution data, a spectral information extraction model is constructed.

[0061] Specifically, the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are combined through multi-task learning. Through the design of a multi-feature loss function of multi-task learning, the target loss functions of all single-task feature network branches are optimized and combined, and multi-source time-space-spectral information is learned together to achieve joint feature extraction fusion of multi-task learning.

[0062] Then, the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are connected in series to obtain four-dimensional high temporal-high spatial-high spectral resolution data. For example, the four-dimensional high spatial resolution data is input into the series model, and the temporal, spatial, and spectral resolutions of the data set can be improved through the spatial information extraction, temporal information extraction, and spectral information extraction models, so that four-dimensional high temporal-high spatial-high spectral resolution data can be obtained.

[0063] In this example: See Figure 1 The step 101, based on the time series high spatial resolution data set to be fused, the time series high temporal resolution data set and the time series high spectral resolution data set are sorted and stored according to the time dimension, the spatial dimension X, the spatial dimension Y and the spectral dimension to obtain four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data, before performing phase encoding, also includes:

[0064] Step 1001, using a panchromatic or multispectral, multispectral and hyperspectral sensor to acquire original panchromatic or multispectral data with a time series of low temporal-high spatial resolution, original multispectral data with a time series of high temporal-medium spatial-low spectral resolution, and original hyperspectral data with a time series of low temporal-low spatial-high spectral resolution;

[0065] Step 1002, preprocessing the original panchromatic or multispectral data of the time series low temporal-high spatial resolution, the original multispectral data of the time series high temporal-medium spatial-low spectral resolution, and the original hyperspectral data of the time series low temporal-low spatial-high spectral resolution, respectively, to obtain corresponding multispectral / panchromatic data, multispectral reflectance data, and hyperspectral reflectance data;

[0066] Step 1003, accurately align all multispectral or panchromatic data, multispectral reflectance data and hyperspectral reflectance data to obtain the four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data.

[0067] Specifically, panchromatic or multispectral, multispectral and hyperspectral sensors are used to obtain original high spatial resolution data with a time series of low temporal-high spatial resolution, original high temporal resolution data with a time series of high temporal-medium spatial-low spectral resolution, and original high spectral resolution data with a time series of low temporal-low spatial-high spectral resolution. For example, the ZY1-02D ​​multispectral sensor, the GF-6 wide-width multispectral sensor and the ZY1-02D ​​hyperspectral sensor are used to obtain the original high spatial resolution data of time series low temporal-high spatial resolution, the original high temporal resolution data of time series high temporal-medium spatial-low spectral resolution and the original high spectral resolution data of time series low temporal-low spatial-high spectral resolution. The original high spatial resolution data is the time series ZY1-02D ​​multispectral data, whose spatial resolution is 10m, the temporal resolution is 55 days, and the number of bands is 8 bands; the original high temporal resolution data is the time series GF-6 wide-width multispectral data, whose spatial resolution is 16m, the theoretical temporal resolution is 4 days, and the number of bands is 8 bands; the original high spectral resolution data is the time series ZY1-02D ​​hyperspectral data, whose spatial resolution is 30m, the theoretical temporal resolution is 4 days, and the number of bands is 166 bands.

[0068] The original time series multispectral data and the original time series multispectral data are preprocessed respectively to obtain the corresponding first reflectivity data and second reflectivity data. Among them, radiation calibration can be performed first, and then the Quac fast atmospheric correction module of ENVI is used to perform reflectivity inversion to obtain the original time series panchromatic / multispectral reflectivity data, the original time series multispectral reflectivity data and the original time series hyperspectral reflectivity data.

[0069] Then, all time-series panchromatic / multispectral reflectance data, time-series multispectral reflectance data and time-series hyperspectral reflectance data are precisely aligned and resampled to the same spatial resolution to obtain time-series panchromatic / multispectral reflectance geometrically precisely corrected data, time-series multispectral reflectance geometrically precisely corrected data and time-series hyperspectral reflectance geometrically precisely corrected data. For example, all time-series panchromatic / multispectral reflectance data, time-series multispectral reflectance data and time-series hyperspectral reflectance data can be aligned, and the alignment accuracy can be within 1 pixel. Then, the above data are resampled, for example, bilinear interpolation can be used for spatial resampling, so that the spatial resolution of the sampled data is the same, for example, 10m. After resampling, time-series panchromatic / multispectral reflectance geometrically precisely corrected data, time-series multispectral reflectance geometrically precisely corrected data and time-series hyperspectral reflectance geometrically precisely corrected data are obtained.

[0070] In this example: See Figure 1 , phase-encode all four-dimensional high spatial resolution data, four-dimensional high temporal resolution data, and four-dimensional high spectral resolution data, including:

[0071] E standard =[1,2,...,L];

[0072]

[0073] Among them, E standard is the phase coding set of the time series data with the highest time resolution containing L scenes, and the imaging time of the mth scene image is Encoded as The phase encoding set of time series data from other sources is E N , the imaging time of the nth image is Encoded as

[0074] In this example: See Figure 1 , using the phase coding set, fast time dimension downsampling is performed on the four-dimensional high time resolution data to be fused, and corresponding four-dimensional time dimension downsampling multispectral data is constructed, specifically including:

[0075] For four-dimensional remote sensing data Use the following formula to calculate the corresponding time dimension downsampling data Right now

[0076]

[0077] Among them, E standard is the phase encoding set of time series data with the highest time resolution, E N Four-dimensional remote sensing data The phase coding set, x∈W, y∈H, b∈B.

[0078] In this example: See Figure 1 Based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spatial resolution data, a spatial information extraction model is constructed, which specifically includes:

[0079] Spatial Dimension Mapping Features Extraction and utilization of four-dimensional high spatial resolution data and 4D time dimension downsampled multispectral data The relationship between spatial Extract

[0080]

[0081] In this example: See Figure 1 Based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high time resolution data, a time information extraction model is constructed, which specifically includes:

[0082] Time dimension mapping features Extraction of high temporal resolution data using four-dimensional and 4D time dimension downsampled multispectral data The relationship between temporal To extract,

[0083]

[0084] In this example: See Figure 1 Based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spectral resolution data, a spectral information extraction model is constructed, which specifically includes:

[0085] Spectral Dimension Mapping Features Extracting multispectral data using downsampling of the four-dimensional time dimension and four-dimensional high spectral resolution data The relationship between spectral To extract,

[0086]

[0087] In this example: See Figure 1 The spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are combined in a multi-task learning manner to jointly learn multi-source time-space-spectral information, specifically including:

[0088] The multi-feature loss function design for multi-task learning optimizes the combination of the target loss functions of all single-task feature network branches to achieve joint feature extraction and fusion of multi-task learning.

[0089] In this example: See Figure 1 , the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are connected in series to obtain four-dimensional high-temporal-high-spatial-high-spectral data, specifically including:

[0090] Using four-dimensional high spatial resolution data and 4D time dimension downsampled multispectral data Obtain the four-dimensional high-temporal-high-spatial-high-spectral data Z to be reconstructed 4D .

[0091]

[0092] In this example: See Figure 1 , multi-feature loss function design for multi-task learning, optimize the combination of target loss functions of all single-task feature network branches, and realize joint feature extraction and fusion of multi-task learning, including:

[0093] Through the design of multi-feature loss functions based on multi-task learning, the target loss functions of all single-task feature network branches are optimized and combined to achieve joint feature extraction and fusion of multi-task learning. Assume that the set of learnable parameters of the multi-task learning network is in Represents all weight parameters in the network. All input sets of the multi-task learning network are x, and the label set is All predicted label sets are For each single-task feature network branch, its loss function is L τ (x,y τ ,y' τ ;w τ ),in The total loss function of the multi-task learning network is designed as a linear combination of each single-task feature network branch, and a regularization term is added to obtain

[0094]

[0095] Among them, c τ is the weight of the learnable τ task feature network branch loss function.

[0096] Based on any of the above embodiments, step 103 specifically includes:

[0097] Based on the mapping relationship between four-dimensional time dimension downsampled multispectral data and four-dimensional high spatial resolution data, a spatial information extraction model is constructed.

[0098] Specifically, in order to learn and establish the mapping relationship between four-dimensional time dimension downsampled multispectral data and four-dimensional high spatial resolution data, a deep learning mechanism can be used to construct a mapping model, and the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spatial resolution data can be used as training data to learn and establish the mapping relationship between the two.

[0099] Based on any of the above embodiments, the mapping model is a multi-branch, multi-layer 3D convolutional neural network;

[0100] The structure of each layer of the network can be expressed as:

[0101]

[0102] in, and is the i-th 3D feature data in the l-th layer and the k-th 3D feature data in the first l-1 layers, (x, y, z) and (u, v, w) represent the different points of the feature data; is a 3D convolution filter, is the 3D bias and σ(·) is the nonlinear activation function.

[0103] Specifically, a mapping model can be constructed based on a multi-layer 3D convolutional neural network, where each branch has convolution filters of different sizes, such as 1, 3, 5, and 7. The model can be expressed as Figure 2 .

[0104] Based on any of the above embodiments, step 103 specifically includes:

[0105] Based on the mapping relationship between four-dimensional time dimension downsampled multispectral data and four-dimensional high time resolution data, a temporal information extraction model is constructed.

[0106] Specifically, in order to learn and establish the mapping relationship between four-dimensional time dimension downsampled multispectral data and four-dimensional high time resolution data, a deep learning mechanism can be used to construct a mapping model, and the four-dimensional time dimension downsampled multispectral data and the four-dimensional high time resolution data can be used as training data to learn and establish the mapping relationship between the two.

[0107] Based on any of the above embodiments, the mapping model is a multi-branch, multi-layer 3D convolutional neural network;

[0108] The structure of each layer of the network can be expressed as:

[0109]

[0110] in, and is the i-th 3D feature data in the l-th layer and the k-th 3D feature data in the first l-1 layers, (x, y, z) and (u, v, w) represent the different points of the feature data; is a 3D convolution filter, is the 3D bias and σ(·) is the nonlinear activation function.

[0111] Specifically, a mapping model can be constructed based on a multi-layer 3D convolutional neural network, where each branch has convolution filters of different sizes, such as 1, 3, 5, and 7. The model can be expressed as Figure 2 .

[0112] Based on any of the above embodiments, step 103 specifically includes:

[0113] Based on the mapping relationship between four-dimensional time dimension downsampled multispectral data and four-dimensional high spectral resolution data, a spectral information extraction model is constructed.

[0114] Specifically, in order to learn and establish the mapping relationship between four-dimensional time dimension downsampled multispectral data and four-dimensional high time resolution data, a deep learning mechanism can be used to construct a mapping model, and the four-dimensional time dimension downsampled multispectral data and the four-dimensional high time resolution data can be used as training data to learn and establish the mapping relationship between the two.

[0115] Based on any of the above embodiments, the mapping model is a multi-branch, multi-layer TimeDistributed2D convolutional neural network, which simultaneously performs 2D convolution operations on the data of each time phase;

[0116] The structure of each layer of the network can be expressed as:

[0117]

[0118] in, and is the i-th 2D feature data in the l-th layer and the k-th 2D feature data in the first l-1 layers, (x, y) and (u, v) represent the differences between the feature data; is a 2D convolution filter, is the 3D bias and σ(·) is the nonlinear activation function.

[0119] Specifically, a mapping model can be constructed based on a multi-layer 2D convolutional neural network, where each branch has convolution filters of different sizes, such as 1, 3, 5, and 7. The model can be expressed as Figure 3 In fact, f can choose any neural network structure. Figure 2 , Figure 3 This is just a partial example of the invention.

[0120] Based on any of the above embodiments, step 104 specifically includes:

[0121] The spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are combined through multi-task learning to jointly train and learn multi-source time-space-spectral information.

[0122] Specifically, through the design of multi-feature loss functions of multi-task learning, the target loss functions output by the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are optimized and combined to achieve joint feature extraction fusion of multi-task learning.

[0123] Based on any of the above embodiments, the multi-feature loss function is designed as follows, assuming that the set of learnable parameters of the multi-task learning network is in Represents all weight parameters in the network. All input sets of the multi-task learning network are x, and the label set is All predicted label sets are For each single-task feature network branch, its loss function is L τ (x,y τ ,y' τ ;w τ ),in The total loss function of the multi-task learning network is designed as a linear combination of each single-task feature network branch, and a regularization term is added to obtain

[0124]

[0125] Among them, c τ is the weight of the learnable τ task feature network branch loss function.

[0126] Specifically, through the spatial information extraction model, the temporal information extraction model and the spectral information extraction model, the four-dimensional time dimension downsampled multispectral data is taken as input and the four-dimensional hyperspectral data is taken as the target, the initial model is trained to obtain the mapping model.

[0127] When training the mapping model, since the objects that the mapping model expects to learn are the mapping relationship between four-dimensional time dimension downsampling multispectral data and four-dimensional high spatial data, the mapping relationship between four-dimensional time dimension downsampling multispectral data and four-dimensional high temporal data, and the mapping relationship between four-dimensional time dimension downsampling multispectral data and four-dimensional high spectral data, the four-dimensional time dimension downsampling multispectral data can be used as the input of the initial model, and the initial model will generate corresponding outputs according to the input, and then the four-dimensional high spatial data, four-dimensional high temporal data, and four-dimensional high spectral data are used as training targets, and the error between the output of the mapping model and the training target is calculated through the multi-feature loss function, and the internal parameters of the model are continuously adjusted to gradually reduce the error between the model output and the training target. After the initial model training converges, the trained spatial information extraction model, temporal information extraction model, and spectral information extraction model can be obtained.

[0128] Based on the above embodiment, step 105 specifically includes:

[0129] The spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are connected in series, and the data to be fused are input to obtain four-dimensional high-temporal-high-spatial-high-spectral data.

[0130] Specifically, after the training of the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model is completed, the models are connected in series end to end, and the four-dimensional high spatial data is input into the mapping model. The mapping model will map and transform the four-dimensional high spatial data according to the spatial, temporal and spectral mapping relationship learned by itself to obtain its corresponding four-dimensional high-temporal, high-spatial and high-spectral data.

[0131] The present invention constructs four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data, constructs spatial information extraction models, temporal information extraction models and spectral information extraction models, and utilizes multi-task learning and multi-feature loss function design to realize the joint extraction and fusion of temporal, spatial and spectral dimension information, realizes integrated fusion in multiple dimensions, and improves the efficiency and speed of data fusion.

[0132] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered as exemplary and non-restrictive in all respects.

[0133] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multi-dimensional remote sensing data fusion method based on multi-task learning, characterized in that: The multi-dimensional remote sensing data fusion method based on multi-task learning comprises the following steps: Step 101, based on the time series high spatial resolution data set to be fused, the time series high temporal resolution data set and the time series high spectral resolution data set are sorted and stored according to the time dimension, spatial dimension X, spatial dimension Y and spectral dimension, four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data are obtained, and temporal phase encoding is performed; the data source includes the time series high spatial resolution data, the time series high temporal resolution data and the time series high spectral resolution data, wherein the time series high spatial resolution data refers to the data with the highest spatial resolution among the three data sources, the time series high temporal resolution data refers to the data with the highest temporal resolution among the three data sources, and the time series high spectral resolution data refers to the data with the highest spectral resolution among the three data sources; Step 102, the four-dimensional high time resolution data to be fused is downsampled in time dimension to construct corresponding four-dimensional time dimension downsampled multispectral data; Step 103, based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spatial resolution data, the four-dimensional time dimension downsampled multispectral data and the four-dimensional high temporal resolution data, and the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spectral resolution data, respectively construct spatial, temporal, and spectral information extraction models; Step 104, combining the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model through multi-task learning to jointly train and learn multi-source time-space-spectral information; Step 105, connecting the spatial information extraction model, the temporal information extraction model, and the spectral information extraction model in series, inputting the data to be fused, and obtaining four-dimensional high-temporal-high-spatial-high-spectral data; In the time, space and spectrum, the "high" refers to the data with the highest resolution among the three types of data, the "medium" refers to the data with the second highest resolution among the three types of data, and the "low" refers to the data with the third highest resolution among the three types of data; Based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spatial resolution data, a spatial information extraction model is constructed, which specifically includes: Spatial Dimension Mapping Features Extraction and utilization of four-dimensional high spatial resolution data and 4D time dimension downsampled multispectral data The relationship between spatial Extract Based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high time resolution data, a time information extraction model is constructed, which specifically includes: Time dimension mapping features Extraction of high temporal resolution data using four-dimensional and 4D time dimension downsampled multispectral data The relationship between temporal To extract, Based on the mapping relationship between the four-dimensional time dimension downsampled multispectral data and the four-dimensional high spectral resolution data, a spectral information extraction model is constructed, which specifically includes: Spectral Dimension Mapping Features Extracting multispectral data using downsampling of the four-dimensional time dimension and four-dimensional high spectral resolution data The relationship between spectral To extract, 2. The multi-dimensional remote sensing data fusion method based on multi-task learning according to claim 1 is characterized in that: The step 101, based on the time series high spatial resolution data set to be fused, the time series high temporal resolution data set and the time series high spectral resolution data set are sorted and stored according to the time dimension, the spatial dimension X, the spatial dimension Y and the spectral dimension to obtain four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data, before performing phase encoding, also includes: Step 1001, using a panchromatic or multispectral, multispectral and hyperspectral sensor to acquire original panchromatic or multispectral data with a time series of low temporal-high spatial resolution, original multispectral data with a time series of high temporal-medium spatial-low spectral resolution, and original hyperspectral data with a time series of low temporal-low spatial-high spectral resolution; Step 1002, preprocessing the original panchromatic or multispectral data of the time series low temporal-high spatial resolution, the original multispectral data of the time series high temporal-medium spatial-low spectral resolution, and the original hyperspectral data of the time series low temporal-low spatial-high spectral resolution, respectively, to obtain corresponding multispectral / panchromatic data, multispectral reflectance data, and hyperspectral reflectance data; Step 1003, accurately align all multispectral / panchromatic data, multispectral reflectance data and hyperspectral reflectance data to obtain the four-dimensional high spatial resolution data, four-dimensional high temporal resolution data and four-dimensional high spectral resolution data.

3. The multi-dimensional remote sensing data fusion method based on multi-task learning according to claim 1 is characterized in that: Phase encoding is performed on all four-dimensional high spatial resolution data, four-dimensional high temporal resolution data, and four-dimensional high spectral resolution data, including: AND standard =[1,2,...,L]; Among them, E standard is the phase coding set of the time series data with the highest time resolution containing L scenes, and the imaging time of the mth scene image is Encoded as The phase encoding set of time series data from other sources is E N , the imaging time of the nth image is Encoded as 4. The multi-dimensional remote sensing data fusion method based on multi-task learning according to claim 3 is characterized in that: Using the phase coding set, the four-dimensional high time resolution data to be fused is quickly downsampled in time dimension to construct corresponding four-dimensional time dimension downsampled multispectral data, specifically including: For four-dimensional remote sensing data Use the following formula to calculate the corresponding time dimension downsampling data Right now Among them, E standard is the phase encoding set of time series data with the highest time resolution, E N Four-dimensional remote sensing data The phase coding set, x∈W, y∈H, b∈B.

5. The multi-dimensional remote sensing data fusion method based on multi-task learning according to claim 1, characterized in that: The spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are combined in a multi-task learning manner to jointly learn multi-source time-space-spectral information, specifically including: The multi-feature loss function design for multi-task learning optimizes the combination of the target loss functions of all single-task feature network branches to achieve joint feature extraction and fusion of multi-task learning.

6. The multi-dimensional remote sensing data fusion method based on multi-task learning according to claim 1, characterized in that: The spatial information extraction model, the temporal information extraction model, and the spectral information extraction model are connected in series to obtain four-dimensional high-temporal-high-spatial-high-spectral data, specifically including: Using four-dimensional high spatial resolution data and 4D time dimension downsampled multispectral data Obtain the four-dimensional high-temporal-high-spatial-high-spectral data Z to be reconstructed 4D , 7. The multi-dimensional remote sensing data fusion method based on multi-task learning according to claim 5, characterized in that: The multi-feature loss function design for multi-task learning optimizes the combination of the target loss functions of all single-task feature network branches to achieve joint feature extraction and fusion of multi-task learning, including: Through the design of multi-feature loss function based on multi-task learning, the target loss functions of all single-task feature network branches are optimized and combined to achieve joint feature extraction and fusion of multi-task learning. Assume that the learnable parameter set of the multi-task learning network is in Represents all weight parameters in the network, all input sets of the multi-task learning network are x, and the label set is All predicted label sets are For each single-task feature network branch, its loss function is L τ (x,y τ ,y' τ ;w τ ),in The total loss function of the multi-task learning network is designed as a linear combination of each single-task feature network branch, and a regularization term is added to obtain Among them, c τ is the weight of the learnable τ task feature network branch loss function.

Images