A deep spatiotemporal super-resolution mapping method integrating historical high-resolution land cover data

Abstract

The present invention discloses a deep spatiotemporal super-resolution mapping method that integrates historical high-resolution land cover data. The method constructs a deep spatiotemporal super-resolution mapping model structure composed of a super-resolution adversarial network (SRGAN) structure and a MUNet network structure in series. It performs a mapping of low-resolution images to high-resolution land cover category probabilities through joint super-resolution reconstruction and semantic segmentation. The method uses a historical high-resolution land cover category map of the target area as auxiliary information. By comparing the category information of low-resolution images from the historical period and the target period, it distinguishes between changed and unchanged categories. Based on this, a sub-pixel category allocation optimization model is constructed that integrates the historical high-resolution land cover classification map and the predicted high-resolution category probability map based on change detection, thereby completing deep spatiotemporal super-resolution mapping of low-resolution remote sensing images. The present invention has the advantages of high simulation accuracy and stability, and is suitable for surface information extraction tasks such as classification, land cover / utilization, and change detection when remote sensing images are insufficient.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a deep spatiotemporal super-resolution mapping method integrating historical high-resolution land cover data. Background Art

[0002] Land cover data is a crucial input for a wide range of earth science applications. Extracting land cover information from remote sensing imagery is an efficient and widely adopted method. Due to sensor limitations, low- and medium-resolution remote sensing images contain mixed pixels of multiple land cover types, posing a challenge to pixel-based classification. Super-resolution mapping (SPM) (also known as sub-pixel mapping or sub-pixel localization) is a mainstream approach to addressing the mixed pixel problem encountered in traditional hard and soft classification (Atkinson, 1997). SPM divides each pixel into sub-pixels and assigns a land cover type to each sub-pixel within a mixed pixel while maintaining the area ratio of each land cover type. Traditional hard classification, however, assigns a single land cover type to each pixel, resulting in inaccurate representation of mixed pixels. Soft classification methods can generate a score map for each land cover type, but cannot clearly define the spatial distribution of land cover types within a pixel. In contrast, super-resolution mapping obtains hard classification results for each subpixel within a mixed pixel under the pixel-level category ratio constraints provided by soft classification. This effectively combines the advantages of both methods, allowing for in-depth mining of mixed pixels and achieving a precise and accurate representation of the objects within them, thus providing richer object classification details (Chen et al., 2015; Ge et al., 2009; Mertens et al., 2006).

[0003] Sub-pixel mapping (SPM), also known as super-resolution mapping, is a mainstream method for solving the problem of mixed pixels in traditional hard classification and soft classification. SPM methods are mainly divided into four categories: (1) SPM methods based on spatial dependence; (2) SPM methods based on geostatistical priors; (3) SPM methods based on spatial regularization; and (4) SPM methods based on auxiliary data (Wang et al., 2022). The first three methods mainly obtain the spatial relationship of sub-pixels in mixed pixels from low-resolution remote sensing images, and their prediction diagram space is very large; the rich information provided by auxiliary data can reduce the uncertainty of SPM results. The proposal of spatiotemporal sub-pixel mapping (STSPM) extends the traditional SPM and uses multi-temporal images to enhance SPM results. Currently, the main STSPM methods are based on spatiotemporal dependence and change detection, but these models are too simple and limited by the quality of soft classification results.

[0004] In recent years, with the rapid development of deep learning technology, many deep learning-based SPM methods have emerged. These methods are data-driven and use the network's end-to-end mapping and feature expression capabilities to learn complex land cover spatial distribution relationships from training samples (Ma et al., 2020; Zhang et al., 2023; Zhang et al., 2021; He et al., 2023; Wang et al., 2019). However, these methods still face problems such as the blurring of spatial detail features by stacked convolutional layers and the underutilization of auxiliary data (such as high-resolution land cover classification maps from historical periods). Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing sub-pixel mapping technology and provide a deep spatiotemporal super-resolution mapping method that integrates historical high-resolution land cover data. The method is based on the traditional model-driven spatiotemporal sub-pixel mapping method, and combines super-resolution reconstruction with semantic segmentation to map low-resolution images to high-resolution land cover category probabilities. The historical high-resolution land cover category map of the target area is used as auxiliary information, and the type information of the low-resolution images of the historical period and the target period is compared to distinguish between changed and unchanged categories. A sub-pixel category allocation optimization model based on change detection is constructed by integrating the historical high-resolution land cover classification map and the predicted high-resolution category probability map to obtain a high-resolution land cover classification map of the target area for the target period.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A deep spatiotemporal super-resolution mapping method integrating historical high-resolution land cover data, the method comprising the following steps:

[0008] Step 1: obtaining low-resolution remote sensing images, high-resolution remote sensing images, and high-resolution land cover classification maps of the target area in historical periods, and performing preprocessing and radiation normalization on the remote sensing images;

[0009] Step 2: Construct an STSPMNet model structure consisting of a super-resolution adversarial network (SRGAN) structure and a MUNet network structure in series; the SRGAN structure includes a generator network structure and a discriminator network structure, and the MUNet network structure is a UNet network structure with an added atrous spatial pyramid pooling module (ASSP);

[0010] Step 3: The remote sensing image and high-resolution land cover classification map processed in step 1 are cropped according to a preset size to obtain each low-resolution remote sensing image block, each corresponding high-resolution remote sensing image block, and each high-resolution land cover classification image block. The image blocks at the corresponding positions constitute samples, and then constitute a sample set; the low-resolution image blocks and the high-resolution land cover classification image blocks are used as the input of the generator network structure of the SRGAN structure, the corresponding high-resolution remote sensing image blocks are used as the input of the discriminator network structure of the SRGAN structure, the corresponding high-resolution remote sensing impact image blocks combined with the land cover category information output by the SRGAN structure are used as the input of the MUNet network structure, and the corresponding high-resolution land cover category probability image blocks are used as the output of the MUNet network structure. The STSPMNet model structure is trained, and the trained SRGAN generator network model and MUNet network model constitute a deep spatiotemporal super-resolution mapping model;

[0011] Step 4: Using each image block of the low-resolution remote sensing image of the target area during the target period as input, a deep spatiotemporal super-resolution mapping model is applied to obtain the corresponding high-resolution land cover category probability image blocks. The obtained high-resolution land cover category probability image blocks are spliced ​​according to geographic coordinates to obtain a high-resolution land cover category probability map of the target area during the target period.

[0012] Step 5: Compare the land cover categories in each pixel of the low-resolution remote sensing images of the target area in the historical period and the target period, calculate the category scores of each land cover category in each pixel, and obtain the information of the change of the category scores of each land cover category in each pixel over time, which is called change detection constraint information;

[0013] Step 6: Construct a sub-pixel category allocation optimization model based on change detection constraint information. The sub-pixel category allocation optimization model combines the high-resolution land cover classification map of the target area in the historical period, assigns the land cover category of each pixel in the high-resolution land cover category probability map of the target area in the target period, and obtains the high-resolution land cover classification map of the target area in the target period.

[0014] Furthermore, the generator network structure of the SRGAN structure consists of a shallow feature extraction layer consisting of a convolutional layer and a PReLU activation layer, a deep feature extraction layer consisting of 16 residual modules, a sub-pixel convolution layer responsible for upsampling, and a reconstruction layer; the discriminator network structure consists of a shallow feature extraction layer consisting of a convolutional layer and a PReLU activation layer, a deep feature extraction layer consisting of 7 modules consisting of a convolutional layer, a batch normalization layer (BN), and an LReLU activation layer, an average pooling layer, a fully connected layer, and a sigmoid activation layer;

[0015] The MUNet network structure includes an encoder and a decoder. The encoder includes three downsampling operation modules and an ASSP module. The decoder includes four upsampling operation modules and uses jump connections to fully fuse the features obtained by the downsampling module with the features of the feature map of the same size obtained during the upsampling process.

[0016] Furthermore, the downsampling operation module of the encoder of the MUNet network structure consists of a convolution layer, a batch normalization layer, a ReLU activation layer, and a maximum pooling layer; the ASSP module is a parallel processing module consisting of four dilated convolutions with dilation rates of 1, 6, 12, and 18; the upsampling operation module of the decoder consists of a convolution layer, a batch normalization layer, a ReLU activation layer, and an upsampling layer for upsampling by interpolation.

[0017] Furthermore, the mathematical expression of the generator network structure of the SRGAN structure for super-resolution reconstruction is:

[0018] H SR =G(Y)

[0019] Among them, G() represents the generator network structure of SRGAN structure, H SR The super-resolution image is generated by the generator network structure based on the input low-resolution remote sensing image;

[0020] The discriminator network structure of the SRGAN structure distinguishes the super-resolution image generated by the generator network structure from the real high-resolution remote sensing image, and its judgment method is:

[0021] D(H)=σ(C(H))→1 Real data?

[0022] D(H SR )=σ(C(H SR ))→0 Fake data?

[0023] Where D() is the discriminator network structure of the SRGAN structure, H is the real high-resolution remote sensing image, σ() and C() represent the non-transformed output of the sigmoid function and the discriminator network structure respectively;

[0024] The discriminator network structure makes the probability of the discrimination result of the real image close to 1; makes the probability of the discrimination result of the generated image closer to 0; when it is difficult to judge whether the image is real or fake, the probability output is 0.5;

[0025] The generator network structure and the discriminator network structure of the SRGAN structure are subjected to adversarial learning, and the performance is optimized by the Optimal function. The specific formula is as follows:

[0026]

[0027] L SRGAN =L G +L D

[0028] Among them, G is the generator network structure, D is the discriminator network structure, L SRGAN is the generator network structure loss function L G and the discriminator network structure loss function L D During the learning process, G generates a predicted image, and D distinguishes the predicted image from the real image. When it is difficult to distinguish the predicted image from the real image, the training ends and the reconstructed super-resolution image is output.

[0029] Furthermore, the MUNet network structure in step 2 includes an ASSP module composed of dilated convolutions with different dilation rates, which can perform multi-scale feature extraction on the input feature map. The calculation formula is as follows:

[0030]

[0031] Among them, i represents the index position on the input feature map, r represents the hole rate, w and k represent the convolution kernel and the size of the convolution kernel, x i+r×k Represents the input feature map of index position i under different void rates, y i represents the output feature map at index position i.

[0032] Furthermore, the following loss function is used when training the STSPMNet model structure in step 3:

[0033] The loss function of the SRGAN structure consists of three parts: content loss, perceptual loss, and adversarial loss:

[0034]

[0035] Where B represents the number of channels in the high-resolution image, VGG19(H) ijb and VGG19(H SR ) ijb They represent the VGG19 network in super-resolution image H SR And the feature output of the position (i, j) of channel b of the real high-resolution remote sensing image H, h 40 、W 40 and B 40 Represent the height, width and number of channels of the 40th layer feature respectively; D(H SR ) is the super-resolution image H SR The output probability of the discriminator network structure; D(H) is the output probability of the discriminator network structure for the real high-resolution remote sensing image H;

[0036] The formula of the cross entropy loss function of the MUNet network structure is as follows:

[0037]

[0038] Among them, C is the total number of land cover categories, S is the downscaling factor, that is, each pixel in the low spatial resolution remote sensing image is divided into S×S sub-pixels, and the high-resolution land cover classification map contains I×S×S×J pixels. is the predicted probability of land cover class c at location (i, j);

[0039] The formula of the overall loss function is as follows:

[0040]

[0041] Among them, α and β represent the hyperparameters that control the weight of each loss; ω and Represent the weight coefficients of the MUNet network structure and the SRGAN structure respectively; during the training process, the weight coefficients of the MUNet network structure and the SRGAN structure are adjusted according to the loss function.

[0042] Furthermore, the sub-pixel category allocation optimization model in step 6 adopts the maximum criterion:

[0043]

[0044] Among them, z is the objective function and st is the constraint function, X ijc Indicates whether the high-resolution pixel (i, j) belongs to the land cover category c, F c represents the portion of low-resolution pixels associated with land cover type c, N uc and N rc They respectively represent the number of high-resolution pixels whose categories remain unchanged and the number of remaining high-resolution pixels in the low-resolution pixels of land cover category c; it should be noted that in order to reduce the calculation time, the calculation of the first two constraints in the objective function and constraint function are only for the remaining high-resolution pixels in each low-resolution pixel.

[0045] Compared with the prior art, the present invention has the following technical effects:

[0046] (1) The present invention integrates historical high-resolution land cover data, combines super-resolution reconstruction with semantic segmentation, and realizes the mapping from low-resolution remote sensing images to high-resolution category probabilities;

[0047] (2) The present invention utilizes auxiliary information from high-resolution land cover classification maps of historical periods to improve the accuracy of sub-pixel spatial relationships within mixed pixels, whereas traditional super-resolution mapping methods have a large prediction space;

[0048] (3) The method of the present invention uses change detection to compare the category information of low-resolution images of the historical period and the low-resolution images of the target period, distinguishing between changed and unchanged categories to reduce the uncertainty of super-resolution mapping results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The main flow chart of the method provided by the present invention;

[0050] Figure 2 The STSPMNet architecture diagram provided by the present invention;

[0051] Figure 3 Comparison of traditional super-resolution mapping and super-resolution mapping results integrated with historical high-resolution land cover classification data; (a) is the locally enlarged area of ​​the ESPCN mapping result, (b) is the locally enlarged area corresponding to the STSPMNet mapping result, and (c) is the locally enlarged area corresponding to the standard high-resolution land cover classification map. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] like Figure 1 As shown, the present invention provides a deep spatiotemporal super-resolution mapping method that integrates historical high-resolution land cover data, which includes the following steps:

[0054] Step 1: Obtain low-resolution remote sensing images, high-resolution remote sensing images, and high-resolution land cover classification maps of the target area and target historical period, pre-process and radiometrically normalize the remote sensing images, and develop the required land classification system, including: built-up areas, unused land, cultivated land, forests, grasslands, and water bodies;

[0055] Step 2: Construct an STSPMNet model structure consisting of a super-resolution adversarial network (SRGAN) structure and a MUNet network structure in series; the SRGAN structure includes a generator network structure and a discriminator network structure, and the MUNet network structure is a UNet network structure with an added atrous spatial pyramid pooling module (ASSP);

[0056] Step 3: The remote sensing image and high-resolution land cover classification map processed in step 1 are cropped according to a preset size to obtain each low-resolution remote sensing image block, each corresponding high-resolution remote sensing image block, and each high-resolution land cover classification image block. The image blocks at the corresponding positions constitute samples, and then constitute a sample set; the low-resolution image blocks and the high-resolution land cover classification image blocks are used as the input of the generator network structure of the SRGAN structure, the corresponding high-resolution remote sensing image blocks are used as the input of the discriminator network structure of the SRGAN structure, the corresponding high-resolution remote sensing impact image blocks combined with the land cover category information output by the SRGAN structure are used as the input of the MUNet network structure, and the corresponding high-resolution land cover category probability image blocks are used as the output of the MUNet network structure. The STSPMNet model structure is trained, and the trained SRGAN generator network model and MUNet network model constitute a deep spatiotemporal super-resolution mapping model;

[0057] Step 4: Using each image block of the low-resolution remote sensing image of the target area during the target period as input, a deep spatiotemporal super-resolution mapping model is applied to obtain the corresponding high-resolution land cover category probability image blocks. The obtained high-resolution land cover category probability image blocks are spliced ​​according to geographic coordinates to obtain a high-resolution land cover category probability map of the target area during the target period.

[0058] Step 5: Compare the land cover categories in each pixel of the low-resolution remote sensing images of the target area in the historical period and the target period, calculate the category scores of each land cover category in each pixel, and obtain the information of the change of the category scores of each land cover category in each pixel over time, which is called change detection constraint information;

[0059] Step 6: Construct a sub-pixel category allocation optimization model based on change detection constraint information. The sub-pixel category allocation optimization model combines the high-resolution land cover classification map of the target area in the historical period, assigns the land cover category of each pixel in the high-resolution land cover category probability map of the target area in the target period, and obtains the high-resolution land cover classification map of the target area in the target period.

[0060] like Figure 2 As shown in the figure, the generator network structure of the SRGAN structure consists of a shallow feature extraction layer consisting of a convolution layer and a PReLU activation layer, a deep feature extraction layer consisting of 16 residual modules, a sub-pixel convolution layer responsible for upsampling, and a reconstruction layer; the discriminator network structure consists of a shallow feature extraction layer consisting of a convolution layer and a PReLU activation layer, a deep feature extraction layer consisting of 7 modules consisting of a convolution layer, a batch normalization layer (BN), and an LReLU activation layer, an average pooling layer, a fully connected layer, and a Sigmoid activation layer;

[0061] The MUNet network structure includes an encoder and a decoder. The encoder includes three downsampling operation modules and an ASSP module; the downsampling operation module consists of a convolutional layer, a batch normalization layer, a ReLU activation layer, and a maximum pooling layer; the ASSP module is a parallel processing module consisting of four dilated convolutional layers with dilation rates of 1, 6, 12, and 18; the decoder includes four upsampling operation modules, and uses skip connections to fully integrate the features obtained by the downsampling module with the features of the feature map of the same size obtained during the upsampling process; the upsampling module of the decoder consists of a convolutional layer, a batch normalization layer, a ReLU activation layer, and an upsampling layer that performs upsampling by interpolation.

[0062] The mathematical expression of the generator network of the SRGAN structure mapping the low-resolution remote sensing image to its high-resolution remote sensing image is:

[0063] H SR =G(Y)

[0064] Among them, G() represents the generator network structure of SRGAN structure, H SR The super-resolution image is generated by the generator network structure based on the input low-resolution remote sensing image;

[0065] The discriminator network structure of the SRGAN structure distinguishes the super-resolution image generated by the generator network structure from the real high-resolution remote sensing image, and its judgment method is:

[0066] D(H)=σ(C(H))→1 Real data?

[0067] D(H SR )=σ(C(H SR ))→0 Fake data?

[0068] Where D() is the discriminator network structure of the SRGAN structure, H is the real high-resolution remote sensing image, σ() and C() represent the non-transformed output of the sigmoid function and the discriminator network structure respectively;

[0069] The discriminator network structure makes the probability of the discrimination result of the real image close to 1; makes the probability of the discrimination result of the generated image closer to 0; when it is difficult to judge whether the image is real or fake, the probability output is 0.5;

[0070] The generator network structure and the discriminator network structure of the SRGAN structure are subjected to adversarial learning, and the performance is optimized by the Optimal function. The specific formula is as follows:

[0071]

[0072] L SRGAN =L G +L D

[0073] Among them, G is the generator network structure, D is the discriminator network structure, L SRGAN is the generator network structure loss function L G and the discriminator network structure loss function L D During the learning process, G generates a predicted image and D distinguishes the predicted image from the real image. When it is difficult to distinguish the predicted image from the real image, the training ends.

[0074] The MUNet network structure in step 2 includes an ASSP module composed of dilated convolutional layers with different dilation rates, which can perform multi-scale feature extraction on the input feature map. The calculation formula is as follows:

[0075]

[0076] Among them, i represents the index position on the input feature map, r represents the hole rate, w and k represent the convolution kernel and the size of the convolution kernel, x i+r×k Represents the input feature map of index position i under different void rates, y i represents the output feature map at index position i.

[0077] The following loss function is used when training the STSPMNet model structure in step 3:

[0078] The loss function of the SRGAN structure consists of three parts: content loss, perceptual loss, and adversarial loss:

[0079]

[0080] Where B represents the number of channels in the high-resolution image, VGG19(H) ijb and VGG19(H SR ) ijb They represent the VGG19 network in super-resolution image H SR And the feature output of the position (i, j) of channel b of the real high-resolution remote sensing image H, h 40 、W 40 and B 40 Represent the height, width and number of channels of the 40th layer feature respectively; D(H SR ) is the super-resolution image H SR The output probability of the discriminator network structure; D(H) is the output probability of the discriminator network structure for the real high-resolution remote sensing image H;

[0081] The formula of the cross entropy loss function of the MUNet network structure is as follows:

[0082]

[0083] Where C is the total number of land cover types, S is the downscaling factor, that is, each pixel in the low spatial resolution remote sensing image is divided into S×S pixels, and the high-resolution land cover classification map contains I×S×S×J pixels. is the predicted probability of land cover type c at location (i, j);

[0084] The formula of the overall loss function is as follows:

[0085]

[0086] Among them, α and β represent the hyperparameters that control the weight of each loss, and the empirical value is 10 -3 and 2×10 -6 ;ω and They represent the weight coefficients of the MUNet network structure and the SRGAN network structure, which are set to 0.77 and 0.23 respectively.

[0087] The sub-pixel category allocation optimization model in step 6 adopts the maximum criterion:

[0088]

[0089] Among them, z is the objective function, st is the constraint function, X ijc Indicates whether the high-resolution pixel (i, j) belongs to the land cover category c, F c represents the portion of low-resolution pixels related to land cover category c, N uc and N rc They respectively represent the number of high-resolution pixels whose categories remain unchanged and the number of remaining high-resolution pixels in the low-resolution pixels of land cover category c; it should be noted that in order to reduce the calculation time, the calculation of the first two constraints in the objective function and constraint function are only for the remaining high-resolution pixels in each low-resolution pixel.

[0090] Step 3 is based on the training sample set and combined with the loss function. The image block pair consisting of the low-resolution remote sensing image of the target area in the target historical period and the high-resolution land cover classification map is used as the input of the generator network structure of the SRGAN structure. The result generated by the generator network structure is input into the discriminator network structure. The discriminator network structure compares the result generated by the generator network structure with the input high-resolution remote sensing image block of the target area in the target historical period. The loss function value is fed back to the generator network structure to achieve super-resolution reconstruction of the low-resolution remote sensing image block; the high-resolution remote sensing image block reconstructed by the SRGAN structure and combined with the land cover category information is input into the MUNet network structure. The MUNet network structure extracts the land cover category features, realizes semantic segmentation, and obtains the predicted high-resolution land cover category probability image block; when the loss function converges, the training ends, and the trained SRGAN generator network model and MUNet network model constitute a deep spatiotemporal super-resolution mapping model.

[0091] Step 4 uses the image blocks of the low-resolution remote sensing image of the target area during the target period as input, applies the deep spatiotemporal super-resolution mapping model, obtains the high-resolution land cover category probability image blocks corresponding to the target area during the target period, and splices the obtained high-resolution land cover category probability image blocks according to geographic coordinates to obtain a high-resolution land cover category probability map of the target area during the target period.

[0092] In summary, in steps 3 and 4, based on deep learning methods, SRGAN is used to map low-resolution images to high-resolution feature maps, providing MUNet with more detailed texture features. Using MUNet with ASPP for image classification can fully integrate shallow semantic features and deep semantic features, and by adjusting the weight parameters of the SRGAN structure and MUNet network structure, a more accurate high-resolution land cover category probability map can be obtained.

[0093] In step 6, a temporal constraint module based on land cover class change detection is proposed, which can better constrain the unchanged land cover classes, thereby preserving the information characteristics provided by the high-resolution land cover classification map in the historical period.

[0094] In order to compare the performance of the STSPMNet method proposed in this paper with the traditional super-resolution mapping method in the experiment, a comparison was conducted in the experiment. The experimental results are as follows: Figure 3 As shown, Figure 3 (a) is the local enlarged area of ​​ESPCN mapping results. Figure 3 (b) is the locally enlarged area corresponding to the STSPMNet mapping result, and (c) is the locally enlarged area corresponding to the standard high-resolution land cover classification map.

[0095] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of the claims.

Claims

1. A deep spatiotemporal super-resolution mapping method integrating historical high-resolution land cover data, characterized in that: The method comprises the following steps: Step 1: obtaining low-resolution remote sensing images, high-resolution remote sensing images, and high-resolution land cover classification maps of the target area in historical periods, and performing preprocessing and radiation normalization on the remote sensing images; Step 2: Construct an STSPMNet model structure consisting of a super-resolution adversarial network (SRGAN) structure and a MUNet network structure in series; the SRGAN structure includes a generator network structure and a discriminator network structure, and the MUNet network structure is a UNet network structure with an added atrous spatial pyramid pooling module (ASSP); Step 3: The remote sensing image and high-resolution land cover classification map processed in step 1 are cropped according to a preset size to obtain each low-resolution remote sensing image block, each corresponding high-resolution remote sensing image block, and each high-resolution land cover classification image block. The image blocks at the corresponding positions constitute samples, and then constitute a sample set; each low-resolution image block and high-resolution land cover classification image block are used as the input of the generator network structure of the SRGAN structure, each corresponding high-resolution remote sensing image block is used as the input of the discriminator network structure of the SRGAN structure, each high-resolution remote sensing image block output by the SRGAN structure that combines land cover category information is used as the input of the MUNet network structure, and each corresponding high-resolution land cover category probability image block is used as the output of the MUNet network structure. The STSPMNet model structure is trained, and the trained SRGAN generator network model and MUNet network model constitute a deep spatiotemporal super-resolution mapping model; Step 4: Using each image block of the low-resolution remote sensing image of the target area during the target period as input, a deep spatiotemporal super-resolution mapping model is applied to obtain the corresponding high-resolution land cover category probability image blocks. The obtained high-resolution land cover category probability image blocks are spliced ​​according to geographic coordinates to obtain a high-resolution land cover category probability map of the target area during the target period. Step 5: Compare the land cover categories in each pixel of the low-resolution remote sensing images of the target area in the historical period and the target period, calculate the category scores of each land cover category in each pixel, and obtain the information of the change of the category scores of each land cover category in each pixel over time, which is called change detection constraint information; Step 6: Construct a sub-pixel category allocation optimization model based on change detection constraint information. The sub-pixel category allocation optimization model combines the high-resolution land cover classification map of the target area in the historical period, assigns the land cover category of each pixel in the high-resolution land cover category probability map of the target area in the target period, and obtains the high-resolution land cover classification map of the target area in the target period.

2. The method for deep spatiotemporal super-resolution mapping by integrating historical high-resolution land cover data according to claim 1, characterized in that: The generator network structure of the SRGAN structure consists of a shallow feature extraction layer consisting of a convolutional layer and a PReLU activation layer, a deep feature extraction layer consisting of 16 residual modules, a sub-pixel convolution layer responsible for upsampling, and a reconstruction layer; the discriminator network structure consists of a shallow feature extraction layer consisting of a convolutional layer and a PReLU activation layer, a deep feature extraction layer consisting of 7 modules consisting of a convolutional layer, a batch normalization layer (BN), and an LReLU activation layer, an average pooling layer, a fully connected layer, and a sigmoid activation layer; The MUNet network structure includes an encoder and a decoder. The encoder includes three downsampling operation modules and an ASSP module. The decoder includes four upsampling operation modules and uses jump connections to fully fuse the features obtained by the downsampling module with the features of the feature map of the same size obtained during the upsampling process.

3. The method for deep spatiotemporal super-resolution mapping by integrating historical high-resolution land cover data according to claim 2, characterized in that: The downsampling operation module of the encoder of the MUNet network structure consists of a convolutional layer, a batch normalization layer, a ReLU activation layer, and a maximum pooling layer; the ASSP module is a parallel processing module consisting of four dilated convolutional layers with dilation rates of 1, 6, 12, and 18; the upsampling operation module of the decoder consists of a convolutional layer, a batch normalization layer, a ReLU activation layer, and an upsampling layer for upsampling by interpolation.

4. A deep spatiotemporal super-resolution mapping method integrating historical high-resolution land cover data according to any one of claims 1 to 3, characterized in that: The mathematical expression of the generator network structure of the SRGAN structure for super-resolution reconstruction is: H SR =G(Y) Among them, G() represents the generator network structure of SRGAN structure, H SR The super-resolution image is generated by the generator network structure based on the input low-resolution remote sensing image; The discriminator network structure of the SRGAN structure distinguishes the super-resolution image generated by the generator network structure from the real high-resolution remote sensing image, and its judgment method is: D(H)=σ(C(H))→1 Real data? D(H SR )=σ(C(H SR ))→0 Fake date? Where D() is the discriminator network structure of the SRGAN structure, H is the real high-resolution remote sensing image, σ() and C() represent the non-transformed output of the sigmoid function and the discriminator network structure respectively; The discriminator network structure makes the probability of the discrimination result of the real image close to 1; makes the probability of the discrimination result of the generated image closer to 0; when it is difficult to judge whether the image is real or fake, the probability output is 0.5; The generator network structure and the discriminator network structure of the SRGAN structure are subjected to adversarial learning, and the performance is optimized by the Optimal function. The specific formula is as follows: L SRGAN =L G +L D Among them, G is the generator network structure, D is the discriminator network structure, L SRGAN is the generator network structure loss function L G and the discriminator network structure loss function L D During the learning process, G generates predicted images and D distinguishes the predicted images from the real images.

5. The method for deep spatiotemporal super-resolution mapping by integrating historical high-resolution land cover data according to claim 3, characterized in that: The MUNet network structure in step 2 can perform multi-scale feature extraction on the input feature map. The calculation formula is as follows: Among them, i represents the index position on the input feature map, r represents the hole rate, w and k represent the convolution kernel and the size of the convolution kernel, x i+r×k Represents the input feature map of index position i under different void rates, y i represents the output feature map at index position i.

6. The method for deep spatiotemporal super-resolution mapping by integrating historical high-resolution land cover data according to claim 1, characterized in that: The following loss function is used when training the STSPMNet model structure in step 3: The loss function of the SRGAN structure consists of three parts: content loss, perceptual loss, and adversarial loss: Where B represents the number of channels in the high-resolution image, VGG19(H) ijb and VGG19(H SR ) ijb They represent the VGG19 network in super-resolution image H SR And the feature output of the position (i, j) of channel b of the real high-resolution remote sensing image H, h 40 、w 40 and B 40 Represent the height, width and number of channels of the 40th layer feature respectively; D(H SR ) is the super-resolution image H SR The output probability of the discriminator network structure; D(H) is the output probability of the discriminator network structure for the real high-resolution remote sensing image H; The formula of the cross entropy loss function of the MUNet network structure is as follows: Among them, C is the total number of land cover categories, S is the downscaling factor, that is, each pixel in the low spatial resolution remote sensing image is divided into S×S sub-pixels, and the high-resolution land cover classification map contains I×S×S×J pixels. is the predicted probability of land cover class c at location (i, j); The formula of the overall loss function is as follows: Among them, α and β represent the hyperparameters that control the weight of each loss; ω and Represent the weight coefficients of the MUNet network structure and the SRGAN structure respectively; during the training process, the weight coefficients of the MUNet network structure and the SRGAN structure are adjusted according to the loss function.

7. The method for deep spatiotemporal super-resolution mapping by integrating historical high-resolution land cover data according to claim 1, characterized in that: The sub-pixel category allocation optimization model in step 6 adopts the maximum criterion: Among them, z is the objective function, st is the constraint function, X ijc Indicates whether the high-resolution pixel (i, j) belongs to the land cover category c, F c represents the portion of low-resolution pixels related to land cover category c, N uc and N rc They respectively represent the number of high-resolution pixels whose categories remain unchanged and the number of remaining high-resolution pixels in the low-resolution pixels of land cover category c; and the calculation of the first two constraints in the objective function and constraint function are only for the remaining high-resolution pixels in each low-resolution pixel.

Images