Remote sensing image spatial-spectral fusion method, device, medium and product

Abstract

The present invention discloses a remote sensing image spatial-spectral fusion method, device, medium, and product, relating to the field of remote sensing fine identification of crops. The method comprises: using a training data set and a resolution loss function to train and optimize an enhanced super-resolution generative adversarial network model; the training data set includes Sentinel-2 images and high-resolution images corresponding to the Sentinel-2 images; inputting the Sentinel-2 images of the crop area to be identified into the optimized enhanced super-resolution generative adversarial network model, obtaining a high-resolution image of the crop area to be identified output by the model; and using a multiple linear regression method to inherit the bands of the Sentinel-2 images of the crop area to be identified from the high-resolution image of the crop area to be identified, thereby obtaining a high-resolution multi-band image for crop identification in the crop area to be identified. The present invention can both improve the spatial resolution of the image and fully inherit multiple bands of the medium-resolution image, thereby improving the recognition accuracy of the crop.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing fine identification of crops, and in particular to a remote sensing image space-spectrum fusion method, device, medium and product. Background Art

[0002] High-spatial-resolution remote sensing imagery is of great significance for monitoring crop growth, estimating grain yields, and mapping fire areas. It also provides guidance for policymaking. However, due to the trade-offs between scanning width and pixel width, current remote sensing sensors produce high-resolution (HR) imagery with a long revisit period and a small number of bands. For example, Sentinel-2 images have a spatial resolution of 10 meters and a revisit period of 5-7 days, while GaoFen (GF) images have a spatial resolution of 2 meters and a revisit period of 15-20 days. These images struggle to meet the critical need for ground observation based on remote sensing data. However, many studies have designed image super-resolution methods to produce large-scale remote sensing imagery with high spatial and temporal resolution, reducing the cost and difficulty of data acquisition and becoming an important remote sensing technology.

[0003] Super-resolution technology is a new image processing method that can generate high-spatial-resolution images from medium-spatial-resolution images. Since the 1980s, this technology has been widely used, and numerous related algorithms have been developed. These can be broadly categorized into interpolation, reconstruction, and learning-based methods. Interpolation methods estimate unknown pixel values ​​by interpolating known pixels within a designed linear function. Examples include nearest neighbor interpolation, Lanczos filtering, and bicubic filtering. While these methods can quickly process high-resolution images, filling pixels with low-frequency information can lead to a loss of overall semantic features. As a result, super-resolution images may lose some detail or have unclear boundaries. Reconstruction-based methods aim to establish appropriate prior constraints or registration models between low- and high-resolution images. The core goal of these reconstruction-based methods is to establish a reasonable prior model. However, high-resolution image reconstruction is a highly underdetermined problem with infinite solutions in mathematical theory. Even with optimal parameters, these models exhibit weak transform performance across a variety of tasks. This limitation requires significant resources to perform reproducible and complex computations.

[0004] In recent years, with the development of machine learning frameworks, the academic community has become increasingly interested in learning-based methods, especially deep learning driven by big data. Convolutional neural networks are a representative adaptive learning method that is widely used in the reconstruction of low-frequency and high-frequency images, with lower complexity than traditional manual intervention. Therefore, researchers have expanded and applied convolutional neural networks to high-resolution image tasks. However, both the early 3-5 layer shallow networks and deep super-resolution networks tend to produce low-frequency results. For this reason, scientists have proposed a series of GAN methods and diffusion methods based on convolutional neural networks, such as the Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) model, which is a typical representative.

[0005] Research has found that super-resolution models like ESRGAN were developed to process natural RGB images or remote sensing images in the RGB and near-infrared (NIR) bands commonly found in high-resolution remote sensing images. They are unable to simultaneously improve the spatial resolution of images while fully integrating the multiple bands (red edge, shortwave infrared, etc.) of medium-resolution images. These multispectral bands, such as the red edge, are crucial for identifying land features. For example, the identification of crops and woodlands relies heavily on the red edge band, a challenge that existing super-resolution technologies, including ESRGAN, cannot address. Summary of the Invention

[0006] In response to the problems raised in the above background technology, the present invention provides a remote sensing image spatial-spectral fusion method, device, medium and product, which can not only improve the spatial resolution of the image but also fully inherit the multiple bands of the medium-resolution image, thereby improving the recognition accuracy of crops.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] In one aspect, the present invention provides a remote sensing image spatial-spectral fusion method, the method comprising:

[0009] Acquire a training data set; the training data set includes Sentinel-2 images and high-resolution images corresponding to the Sentinel-2 images; the Sentinel-2 images and high-resolution images of the same area correspond to each other; the Sentinel-2 images are medium-resolution remote sensing images; and the high-resolution images are high-resolution remote sensing images;

[0010] Obtain an enhanced super-resolution generative adversarial network model;

[0011] The enhanced super-resolution generative adversarial network model is trained and optimized using the training data set and the resolution loss function to obtain an optimized enhanced super-resolution generative adversarial network model;

[0012] Acquire Sentinel-2 images of the crop area to be identified;

[0013] Inputting the Sentinel-2 image of the crop area to be identified into the optimized enhanced super-resolution generative adversarial network model to obtain a high-resolution image of the crop area to be identified output by the optimized enhanced super-resolution generative adversarial network model;

[0014] A multiple linear regression method is used to inherit the bands of the Sentinel-2 image of the crop area to be identified from the high-resolution image of the crop area to be identified, so as to obtain a high-resolution multi-band image; the bands of the Sentinel-2 image of the crop area to be identified include a red-edge band and a short-wave infrared band; the high-resolution multi-band image is used for crop identification in the crop area to be identified.

[0015] Optionally, the step of obtaining a training data set further includes:

[0016] Acquiring an original Sentinel-2 image and an original high-resolution image corresponding to the original Sentinel-2 image; the original Sentinel-2 image and the original high-resolution image of the same area correspond to each other;

[0017] Preprocessing the original Sentinel-2 image to obtain a Sentinel-2 image with a spatial resolution of 10 meters;

[0018] Preprocessing the original high-resolution image to obtain a high-resolution image with a spatial resolution of 2 meters;

[0019] A training data set is constructed using the Sentinel-2 images and the high-resolution images.

[0020] Optionally, the preprocessing includes image fusion, radiometric calibration, geometric correction, orthorectification, image registration and reflectivity correction of different data sources.

[0021] Optionally, the resolution loss function is

[0022] in, represents the resolution loss, var represents the variance, represents the predicted image, F2 represents the real image, and a represents the empirical coefficient.

[0023] Optionally, the step of using a multiple linear regression method to inherit the bands of the Sentinel-2 image of the crop area to be identified from the high-resolution image of the crop area to be identified to obtain a high-resolution multi-band image specifically includes:

[0024] A multivariate linear regression method is used to calculate coefficients of corresponding bands between the high-resolution image of the crop area to be identified and the Sentinel-2 image of the crop area to be identified; the corresponding bands between the high-resolution image of the crop area to be identified and the Sentinel-2 image of the crop area to be identified include red, green, blue, and near-infrared bands;

[0025] Calculate the coefficients of the red edge band and the shortwave infrared band of the Sentinel-2 image of the crop area to be identified using a distance-weighted method based on the coefficients of the red band, the green band, the blue band, and the near-infrared band of the Sentinel-2 image of the crop area to be identified;

[0026] According to the coefficients of the red edge band and the coefficients of the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, and the red edge band and the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, the red edge band and the shortwave infrared band of the high-resolution image of the crop area to be identified are obtained; the bands of the high-resolution multi-band image include the red, green, blue, near-infrared, red edge and shortwave infrared bands of the high-resolution image of the crop area to be identified.

[0027] Optionally, obtaining the red edge band and shortwave infrared band of the high-resolution image of the crop area to be identified based on the coefficients of the red edge band and the coefficients of the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, and the red edge band and the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, specifically includes:

[0028] multiplying the coefficient of the red edge band of the Sentinel-2 image of the crop area to be identified by the red edge band of the Sentinel-2 image of the crop area to be identified to obtain the red edge band of the high-resolution image of the crop area to be identified;

[0029] The coefficient of the shortwave infrared band of the Sentinel-2 image of the crop area to be identified is multiplied by the shortwave infrared band of the Sentinel-2 image of the crop area to be identified to obtain the shortwave infrared band of the high-resolution image of the crop area to be identified.

[0030] On the other hand, the present invention also provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned remote sensing image spatial-spectral fusion method.

[0031] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned remote sensing image spatial-spectral fusion method when executed by a processor.

[0032] In yet another aspect, the present invention further provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned remote sensing image spatial-spectral fusion method when executed by a processor.

[0033] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0034] The remote sensing image spatial-spectral fusion method, device, medium and product disclosed in the present invention generate a high-resolution remote sensing image corresponding to the Sentinel-2 image of the crop area to be identified based on an enhanced super-resolution generative adversarial network model, and post-process the high-resolution remote sensing image, that is, use a multivariate linear regression method to fully inherit the multiple bands (red edge, short-wave infrared, etc.) of the Sentinel-2 image of the crop area to be identified in the high-resolution remote sensing image to obtain a high-resolution multi-band (spectral) image for crop identification, thereby realizing multispectral super-resolution image reconstruction. The reconstructed image, that is, the high-resolution multi-band image, has the advantages of high resolution and multi-band, and can not only improve the spatial resolution of the image, but also fully inherit the multiple bands (red edge, short-wave infrared, etc.) of the medium-resolution image. When using the high-resolution multi-band image for crop identification, the multi-spectral spectrum of the high-resolution multi-band image, such as the red edge and short-wave infrared, which are crucial for crop identification, can improve the recognition accuracy of the crop. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A flowchart of a remote sensing image spatial-spectral fusion method provided in Example 1 of the present invention;

[0037] Figure 2 This is a flow chart of the remote sensing image spatial-spectral fusion technology based on deep super-resolution network of the present invention;

[0038] Figure 3 This is the ESRGAN network structure diagram of the present invention;

[0039] Figure 4 This is the super-resolution fusion result diagram of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] The purpose of the present invention is to provide a remote sensing image spatial-spectral fusion method, device, medium and product, which can not only improve the spatial resolution of the image but also fully inherit the multiple bands of the medium-resolution image, thereby improving the recognition accuracy of crops.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Example 1

[0044] This embodiment provides a remote sensing image space-spectrum fusion method, such as Figure 1 As shown, the method includes the following steps:

[0045] Step 101: Acquire a training dataset; the training dataset includes Sentinel-2 images and high-resolution images corresponding to the Sentinel-2 images; Sentinel-2 images and high-resolution images of the same area correspond to each other; Sentinel-2 images are medium-resolution remote sensing images; and high-resolution images are high-resolution remote sensing images.

[0046] In step 101, a training dataset is constructed using real Sentinel-2 images and real high-resolution images. Both the real Sentinel-2 images and the real high-resolution images are acquired via satellite.

[0047] Before step 101, the following steps are also included:

[0048] Obtain original Sentinel-2 images and original high-resolution images corresponding to the original Sentinel-2 images; original Sentinel-2 images and original high-resolution images of the same area correspond to each other.

[0049] The original Sentinel-2 images were preprocessed to obtain Sentinel-2 images with a spatial resolution of 10 meters.

[0050] The original high-resolution image is preprocessed to obtain a high-resolution image with a spatial resolution of 2 meters.

[0051] The training dataset is constructed using Sentinel-2 images and high-resolution images.

[0052] Among them, preprocessing includes image fusion, radiometric calibration, geometric correction, orthorectification, image registration and reflectivity correction of different data sources.

[0053] Step 102: Obtain an enhanced super-resolution generative adversarial network model.

[0054] Step 103: Use the training data set and the resolution loss function to train and optimize the enhanced super-resolution generative adversarial network model to obtain an optimized enhanced super-resolution generative adversarial network model.

[0055] In step 103, the resolution loss function is F2); among which, represents the resolution loss, var represents the variance, represents the predicted image, F2 represents the real image, and a represents the empirical coefficient, which is set to 3 after experiments.

[0056] Step 104: Acquire a Sentinel-2 image of the crop area to be identified.

[0057] Step 105: Input the Sentinel-2 image of the crop area to be identified into the optimized enhanced super-resolution generative adversarial network model to obtain a high-resolution image of the crop area to be identified as output by the optimized enhanced super-resolution generative adversarial network model.

[0058] In step 105, the high-resolution image of the crop area to be identified output by the optimized enhanced super-resolution generative adversarial network model is a high-resolution image of the crop area to be identified reconstructed by the optimized enhanced super-resolution generative adversarial network model based on the Sentinel-2 image of the real crop area to be identified input into the model. The reconstructed high-resolution image of the crop area to be identified is also called a super-resolution image (i.e., super-resolution image). The reconstructed high-resolution image of the crop area to be identified is very close to the high-resolution image of the real crop area to be identified obtained by satellite.

[0059] Step 106: Using a multiple linear regression method, the high-resolution image of the crop area to be identified is integrated with the bands of the Sentinel-2 image of the crop area to be identified to obtain a high-resolution multi-band image; the bands of the Sentinel-2 image of the crop area to be identified include a red-edge band and a short-wave infrared band; the high-resolution multi-band image is used for crop identification in the crop area to be identified.

[0060] The step 106 specifically includes:

[0061] The multivariate linear regression method is used to calculate the coefficients of the corresponding bands between the high-resolution images of the crop area to be identified and the Sentinel-2 images of the crop area to be identified; the corresponding bands between the high-resolution images of the crop area to be identified and the Sentinel-2 images of the crop area to be identified include red, green, blue and near-infrared bands.

[0062] According to the coefficients of the red band, green band, blue band and near-infrared band of the Sentinel-2 image of the crop area to be identified, the distance-weighted method is used to calculate the coefficients of the red edge band and shortwave infrared band of the Sentinel-2 image of the crop area to be identified.

[0063] The red edge band and shortwave infrared band of the high-resolution image of the crop area to be identified are obtained based on the coefficients of the red edge band and the coefficients of the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, as well as the red edge band and shortwave infrared band of the Sentinel-2 image of the crop area to be identified; the bands of the high-resolution multi-band image include the red, green, blue, near-infrared, red edge, and shortwave infrared bands of the high-resolution image of the crop area to be identified.

[0064] The red edge band and shortwave infrared band of the high-resolution image of the crop area to be identified are obtained based on the coefficients of the red edge band and the coefficients of the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, and the red edge band and shortwave infrared band of the Sentinel-2 image of the crop area to be identified, specifically including:

[0065] The coefficient of the red edge band of the Sentinel-2 image of the crop area to be identified is multiplied by the red edge band of the Sentinel-2 image of the crop area to be identified to obtain the red edge band of the high-resolution image of the crop area to be identified.

[0066] The coefficient of the shortwave infrared band of the Sentinel-2 image of the crop area to be identified is multiplied by the shortwave infrared band of the Sentinel-2 image of the crop area to be identified to obtain the shortwave infrared band of the high-resolution image of the crop area to be identified.

[0067] The technical solution of the present invention is described below with a specific embodiment:

[0068] Affected by the balance between the scanning width and pixel width of satellite sensors and observation conditions such as clouds and rain, it is difficult for a single sensor to provide full coverage of high-resolution remote sensing images at a large scale. The remote sensing image spatial-spectral fusion method of the present invention provides a remote sensing image spatial-spectral fusion technology based on a deep super-resolution network. It is a technology that uses coarse-resolution (medium-resolution) remote sensing images based on a deep learning super-resolution network to generate high-resolution images, achieving full coverage of high-resolution images at a large scale. The remote sensing image spatial-spectral fusion method of the present invention mainly provides data support for crop identification. The remote sensing image spatial-spectral fusion method process of the present invention includes: the input coarse-resolution image comes from the Sentinel-2 optical satellite, and the output high-resolution image comes from the Gaofen-1 / 6 satellite. It mainly includes several key stages: image preprocessing, image super-resolution and image post-processing. After experiments, the remote sensing image spatial-spectral fusion method of the present invention completed the high-resolution multi-band image coverage of the crop area in the East China Plain with high precision and high efficiency, which is of great significance to the classification, phenological observation and yield estimation of crops.

[0069] The input of the remote sensing image space-spectrum fusion method of the present invention is a medium-resolution remote sensing image, taking Sentinel-2 (10 meters) as an example, and the output is a high-resolution remote sensing image (high-resolution remote sensing image), taking GF-1 / 6 (2 meters) as an example. Before the formal input model, the image must be preprocessed to make the input image meet the input standard of the model. Ultimately, the output image of the model will be mainly used in the field of remote sensing fine identification of crops and the extraction of key crops. The model used is a deep learning super-resolution model (ESRGAN), which includes two processes: training and prediction. Finally, data post-processing is required to allow the generated image to inherit the band of Sentinel-2. The remote sensing image space-spectrum fusion method of the present invention is as follows Figure 2 As shown, it can be divided into five steps: (1) data collection and preprocessing (2) model training (3) model prediction (4) image post-processing.

[0070] Step 1: Data collection and preprocessing

[0071] Sentinel-2 images have a spatial resolution of 10 meters and a temporal resolution of approximately 5 to 7 days, offering high-frequency access and wide global coverage. The GF-1 / 6 satellites have a spatial resolution of 2 meters, providing global coverage for approximately 15 to 20 days and capable of HR observations. The experiment acquired Sentinel images of East China from GEE, and downloaded GF-1 / 6 satellite data from the Land Observation website (https: / / data.cresda.cn / # / 2dMap). Because the acquired remote sensing images are DN values, not reflectance values ​​required for research, and suffer from geometric bias and paired image misregistration, data preprocessing primarily addresses geographic bias in the acquired data, image registration, and reflectance correction to produce high-quality reflectance remote sensing data products.

[0072] Data preprocessing mainly includes image fusion, geometric correction, radiometric calibration, atmospheric correction, image registration, and reflectivity calibration of different data sources for Sentinel-2 and GF images:

[0073] Image fusion: GF panchromatic images and GF multispectral images are fused to generate multispectral HR images. For Sentinel-2, image fusion is a combination of data from different bands.

[0074] Radiometric calibration: Convert the recorded raw DN values ​​into apparent reflectance; convert the image's brightness grayscale values ​​into absolute radiometric brightness. Atmospheric correction: Use the FLAASH algorithm to perform atmospheric correction on the image to obtain surface reflectance image data.

[0075] Geometric correction and orthorectification: Geometric correction and orthorectification are performed based on the original optical satellite multispectral remote sensing images. After correction, the geographic coordinate error in flat areas and low hilly areas should be less than or equal to 1 pixel, and the geographic coordinate error in mountainous areas and areas with complex terrain should be less than or equal to 2 pixels.

[0076] Image registration: To geo-register the multispectral data with the Tiantu map, at least 20 control points should be selected. The control points should be evenly distributed within the image range, and the control points should be selected at the intersection of fixed terrain and objects as much as possible.

[0077] Reflectivity Calibration (Reflectivity Correction): Although Sentinel, GF-1, and GF-6 remote sensing images capture the same ground feature at the same time, they still produce different reflectivity differences due to the different remote sensing satellite sources. This method uses a multivariate linear equation to fit the reflectivity relationship between GF and Sentinel, thereby correcting the GF reflectivity.

[0078] Step 2: Model training

[0079] Model training includes three main parts: data input, weight training and network optimization. The main framework of the model is as follows Figure 3 The training of the model mainly involves establishing the mapping relationship between Sentinel-2 and GF images:

[0080] Data input: The complete Sentinel-2 image is cropped to a 64*64 image, and the corresponding GF image is cropped to a 256*256 image. To improve the model's generalization ability for images, this paper uses a series of commonly used data augmentation techniques, such as image rotation.

[0081] Weight training and network optimization: During the network weight training process, the batch size is set to 64, the epoch is 100, the convolution feature is 64 layers, and the convolution kernel size is 3. The experiment uses the "Adam" optimization and the weight decay is 10 -3 , momentum is 0.9, and learning rate is 10 -4 .

[0082] Optimize the model's loss function: The previous L1 loss function and GAN loss function are not sensitive to image resolution, which causes the generated image to be blurred and reduces the image resolution. This paper proposes a resolution loss function to constrain the model's prediction:

[0083]

[0084] When the trained ESRGAN model is used for prediction, the input of the model is Sentinel-2, and the output is GF (the GF here is the GF reconstructed by the trained ESRGAN model based on the real Sentinel-2 image input to the model. The reconstructed GF is also called a super-resolution image. The reconstructed GF is very close to the real GF obtained by satellite). Figure 3 The structure of the ESRGAN model shown is a well-known structure. The present invention combines this model to complete the super-resolution workflow of Sentinel2, especially inheriting the red edge band of Sentinel-2 (because Gaofen does not have a red edge band).

[0085] Step 3: Model prediction

[0086] The present invention performs image super-resolution on all the selected sentinel data (covering the entire East China Plain) by region. Due to the large area, the present invention adopts regional execution and divides the entire study area into 8 blocks according to the terrain, geographical distribution and other conditions. The final prediction result is shown in the figure Figure 4 shown. Figure 4 Parts (a), (b), and (c) are Sentinel-2, super-resolution images, and GF, respectively (GF here refers to the real high-resolution image, not the reconstructed image).

[0087] Step 4: Image post-processing

[0088] Sentinel-2 is a multi-band image, including red, green, blue, red edge, near infrared, short-wave infrared, etc., while the super-resolution image obtained according to step 3 is limited by the number of bands in the training data GF, and only has four bands: red, green, blue, and near infrared. To this end, by exploring the intrinsic relationship between multi-source remote sensing images and between bands of remote sensing images of the same source, the present invention found that the correlation between the same bands of different sensors and the adjacent bands of the same sensor is very strong, so the multiple linear regression (MLR) method (such as Figure 2 As shown in Figure 2), the super-resolved image inherits the band of Sentinel-2, so that the reconstructed image is Figure 4 The super-resolution image shown in part (b) has the advantages of high resolution and multi-band, providing a data basis for crop classification and yield estimation in the study area. The multivariate linear regression method is a simple and easy-to-use multi-source regression method. The details are as follows:

[0089] A linear relationship is established between the super-resolution image and the original Sentinel-2 red, green, blue, and near-infrared bands through a regression model. The formula is as follows:

[0090] SR k =W k ×LR k

[0091] The original Sentinel-2 is the Sentinel-2 image of the crop area to be identified that is input into the optimized enhanced super-resolution generative adversarial network model. k refers to the kth band of the image, which here refers to the red band, green band, blue band, and near-infrared band. W k is the weight (coefficient) of the kth band, SR refers to the super-resolution image, LR refers to the Sentinel-2 image, when k is the red band, SR k Refers to the red band of the super-resolution image, W k Refers to the coefficient of the red band, LR k Refers to the red band of the original Sentinel-2 image. When k is the green band, SR k Refers to the green band of the super-resolution image, W k Refers to the coefficient of the green band, LR k Refers to the green band of the original Sentinel-2 image. When k is the blue band, SR k Refers to the blue band of the super-resolution image, W k Refers to the coefficient of the blue band, LR k Refers to the blue band of the original Sentinel-2 image. When k is the near-infrared band, SR k Refers to the near-infrared band of the super-resolution image, W k Refers to the coefficient of the near infrared band, LR k Refers to the near-infrared band of the original Sentinel-2 image.

[0092] Among them, the coefficient W for the red edge and short-wave infrared bands is x , calculated using a distance-weighted approach, the formula is as follows:

[0093]

[0094] Among them, k refers to the kth band, including red, green, blue, and near infrared; d k Refers to the distance between the kth band and the shortwave infrared band or red edge band, W k Refers to the weight between the super-resolved image and the original Sentinel-2 image in the kth band; x refers to the red edge band or short-wave infrared band, W x Refers to the weight of the short-wave infrared or red edge band. When k = 1, k is the red band. At this time, d k Refers to the distance between the red band and the shortwave infrared band or red edge band, W k Refers to the coefficient of the red band. When k=2, k is the green band. At this time, d k Refers to the distance between the green band and the shortwave infrared band or the red edge band, W kRefers to the coefficient of the green band. When k=3, k is the blue band. At this time, d k Refers to the distance between the blue band and the shortwave infrared band or red edge band, W k Refers to the coefficient of the blue band. When k=4, k is the near-infrared band. At this time, d k Refers to the distance between the near-infrared band and the short-wave infrared band or red edge band, W k Refers to the coefficient of the near-infrared band. When x is the red edge band, W x Refers to the coefficient of the red edge band. When x is the short-wave infrared band, W x Refers to the coefficient of the shortwave infrared band.

[0095] Finally, the red edge and short-wave infrared band of the super-resolution image can be obtained through the formula as follows:

[0096] SR x =W x ×LR x

[0097] When x is the red edge band, SR x Refers to the red edge band of the super-resolution image, W x Refers to the coefficient of the red edge band, LR x Refers to the red edge band of the original Sentinel-2 image. When x is the short-wave infrared band, SR x Refers to the short-wave infrared band of the super-resolution image, W x Refers to the coefficient of the shortwave infrared band, LR x Refers to the shortwave infrared band of the original Sentinel-2 image.

[0098] To address the challenges that existing super-resolution technologies, including ESRGAN, cannot handle, this paper proposes a multispectral super-resolution image reconstruction technology based on the deep learning method ESRGAN. It achieves two key breakthroughs: 1) Based on a GAN network technology, this paper optimizes the loss function, making the network more suitable for remote sensing images; and 2) proposes a post-processing technology for remote sensing images, which inherits multiple low-resolution spectra into the reconstructed high-resolution image containing only red, green, blue, and near-infrared, to obtain a high-resolution multispectral image.

[0099] Since its introduction, the super-resolution algorithm has attracted extensive research attention and many related algorithms have been designed. Compared with other methods, the method proposed in this paper has the following advantages:

[0100] (1) The method proposed in the present invention can inherit multi-band information of medium-resolution images. Medium-resolution spectral information is usually richer, but existing methods cannot inherit multi-band information of medium-resolution images during the super-resolution process. The present invention studies the correlation between the same spectral bands and between different spectral bands of multi-source remote sensing satellites and proposes a remote sensing data post-processing method in step 4, which achieves the purpose of inheriting multi-band information of medium-resolution images.

[0101] (2) The method proposed in this paper generates more realistic details and higher resolution. Compared with the original ESRGAN method, the method proposed in this paper inherits its generative adversarial technology and can generate relevant details. In addition, the method proposed in this paper optimizes the loss function in step 2, reduces the fuzzy prediction of the model, alleviates the underdetermination problem of the interpolation process, and improves the true resolution of the image.

[0102] (3) The method proposed in the present invention is more efficient. The present invention integrates the spatial super-resolution of the image and the inheritance of the bands in the same network framework. It is an end-to-end technology that does not need to consider issues such as feature screening and finding band relationships. The overall operating efficiency is high and the accuracy is good.

[0103] In summary, other current deep learning super-resolution algorithms lack a comprehensive workflow for simultaneously addressing both spatial and spectral super-resolution of remote sensing data, particularly when considering the differences in image resolution and inter-band differences between multiple satellite sources. The method proposed in this paper fills this gap in the technology.

[0104] Example 2

[0105] This embodiment provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the steps of the remote sensing image spatial-spectral fusion method in Embodiment 1.

[0106] Example 3

[0107] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the remote sensing image spatial-spectral fusion method in Embodiment 1 are implemented.

[0108] Example 4

[0109] This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the remote sensing image spatial-spectral fusion method in Example 1 are implemented.

[0110] Example 5

[0111] As a further supplement to Example 2, this Example 5 provides a computer device, which can be a database. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the remote sensing image spatial-spectral fusion method in Example 1 is implemented.

[0112] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0113] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided by the present invention may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0114] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A remote sensing image spatial-spectral fusion method, characterized in that: The method comprises: Acquire a training data set; the training data set includes Sentinel-2 images and high-resolution images corresponding to the Sentinel-2 images; the Sentinel-2 images and high-resolution images of the same area correspond to each other; the Sentinel-2 images are medium-resolution remote sensing images; and the high-resolution images are high-resolution remote sensing images; Obtain an enhanced super-resolution generative adversarial network model; The enhanced super-resolution generative adversarial network model is trained and optimized using the training data set and the resolution loss function to obtain an optimized enhanced super-resolution generative adversarial network model; Acquire Sentinel-2 images of the crop area to be identified; Inputting the Sentinel-2 image of the crop area to be identified into the optimized enhanced super-resolution generative adversarial network model to obtain a high-resolution image of the crop area to be identified output by the optimized enhanced super-resolution generative adversarial network model; A multivariate linear regression method is used to inherit the bands of the Sentinel-2 image of the crop area to be identified from the high-resolution image of the crop area to be identified, thereby obtaining a high-resolution multi-band image; the bands of the Sentinel-2 image of the crop area to be identified include a red-edge band and a short-wave infrared band; and the high-resolution multi-band image is used for crop identification in the crop area to be identified; The method of using a multiple linear regression method to inherit the bands of the Sentinel-2 image of the crop area to be identified from the high-resolution image of the crop area to be identified to obtain a high-resolution multi-band image specifically includes: The coefficients of the corresponding bands of the high-resolution image of the crop area to be identified and the Sentinel-2 image of the crop area to be identified are calculated by using a multiple linear regression method; the corresponding bands of the high-resolution image of the crop area to be identified and the Sentinel-2 image of the crop area to be identified include red, green, blue, and near-infrared bands; the multiple linear regression method is as follows: a linear relationship between the super-resolution image and the red, green, blue, and near-infrared bands of the original Sentinel-2 is established through a regression model, and the formula is as follows: SR k =W k ×LR k Wherein, the original Sentinel-2 is the Sentinel-2 image of the crop area to be identified that is input into the optimized enhanced super-resolution generative adversarial network model, and k refers to the kth band of the image, which here refers to the red band, green band, blue band, and near-infrared band; W k is the weight of the kth band, that is, the coefficient of the kth band. SR refers to the super-resolution image, and LR refers to the Sentinel-2 image. When k is the red band, SR k Refers to the red band of the super-resolution image, W k Refers to the coefficient of the red band, LR k Refers to the red band of the original Sentinel-2 image. When k is the green band, SR k Refers to the green band of the super-resolution image, W k Refers to the coefficient of the green band, LR k Refers to the green band of the original Sentinel-2 image. When k is the blue band, SR k Refers to the blue band of the super-resolution image, W k Refers to the coefficient of the blue band, LR k Refers to the blue band of the original Sentinel-2 image. When k is the near-infrared band, SR k Refers to the near-infrared band of the super-resolution image, W k Refers to the coefficient of the near infrared band, LR k Refers to the near-infrared band of the original Sentinel-2 image; Based on the coefficients of the red band, green band, blue band, and near-infrared band of the Sentinel-2 image of the crop area to be identified, the coefficients of the red edge band and the shortwave infrared band of the Sentinel-2 image of the crop area to be identified are calculated using a distance-weighted method, as follows: Among them, k refers to the kth band, including red, green, blue, and near infrared; d k Refers to the distance between the kth band and the shortwave infrared band or red edge band, W k Refers to the weight between the super-resolved image and the original Sentinel-2 image in the kth band; x refers to the red edge band or short-wave infrared band, W x Refers to the weight of the short-wave infrared or red edge band. When k = 1, k is the red band. At this time, d k Refers to the distance between the red band and the shortwave infrared band or red edge band, W k Refers to the coefficient of the red band. When k=2, k is the green band. At this time, d k Refers to the distance between the green band and the shortwave infrared band or the red edge band, W k Refers to the coefficient of the green band. When k=3, k is the blue band. At this time, d k Refers to the distance between the blue band and the shortwave infrared band or red edge band, W k Refers to the coefficient of the blue band. When k=4, k is the near-infrared band. At this time, d k Refers to the distance between the near-infrared band and the short-wave infrared band or red edge band, W k Refers to the coefficient of the near-infrared band. When x is the red edge band, W x Refers to the coefficient of the red edge band. When x is the short-wave infrared band, W x Refers to the coefficient of the shortwave infrared band; According to the coefficients of the red edge band and the coefficients of the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, and the red edge band and the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, the red edge band and the shortwave infrared band of the high-resolution image of the crop area to be identified are obtained; the bands of the high-resolution multi-band image include the red, green, blue, near-infrared, red edge and shortwave infrared bands of the high-resolution image of the crop area to be identified.

2. The remote sensing image spatial-spectral fusion method according to claim 1, characterized in that: The obtaining of the training data set also includes: Acquiring an original Sentinel-2 image and an original high-resolution image corresponding to the original Sentinel-2 image; the original Sentinel-2 image and the original high-resolution image of the same area correspond to each other; Preprocessing the original Sentinel-2 image to obtain a Sentinel-2 image with a spatial resolution of 10 meters; Preprocessing the original high-resolution image to obtain a high-resolution image with a spatial resolution of 2 meters; A training data set is constructed using the Sentinel-2 images and the high-resolution images.

3. The remote sensing image spatial-spectral fusion method according to claim 2, characterized in that: The preprocessing includes image fusion, radiometric calibration, geometric correction, orthorectification, image registration and reflectivity correction of different data sources.

4. The remote sensing image spatial-spectral fusion method according to claim 1, characterized in that: The resolution loss function is in, represents the resolution loss, var represents the variance, represents the predicted image, F2 represents the real image, and a represents the empirical coefficient.

5. The remote sensing image spatial-spectral fusion method according to claim 1, characterized in that: Obtaining the red edge band and the shortwave infrared band of the high-resolution image of the crop area to be identified based on the coefficients of the red edge band and the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, and the red edge band and the shortwave infrared band of the Sentinel-2 image of the crop area to be identified, specifically comprising: multiplying the coefficient of the red edge band of the Sentinel-2 image of the crop area to be identified by the red edge band of the Sentinel-2 image of the crop area to be identified to obtain the red edge band of the high-resolution image of the crop area to be identified; The coefficient of the shortwave infrared band of the Sentinel-2 image of the crop area to be identified is multiplied by the shortwave infrared band of the Sentinel-2 image of the crop area to be identified to obtain the shortwave infrared band of the high-resolution image of the crop area to be identified.

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the remote sensing image spatial-spectral fusion method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the remote sensing image spatial-spectral fusion method according to any one of claims 1 to 5 are implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the remote sensing image spatial-spectral fusion method according to any one of claims 1 to 5 are implemented.

Images