A spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model

Abstract

The present invention discloses a spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model, which is characterized by the following steps: S1, collecting multiple groups of matched multispectral remote sensing images, ground object classification maps, and numerical feature product maps and performing preprocessing; S2, determining the input matrix x of the generative adversarial model according to the preprocessed multispectral remote sensing images, ground object classification maps, and numerical feature product maps, and determining the size of the output matrix y of the generative adversarial model; S3, respectively processing each group of matched image data and then sequentially inputting them into the generative adversarial model for training; S4, inputting new matched image data and using the trained generative adversarial model to complete the spectral domain migration of satellite remote sensing multispectral images. The present invention realizes the direct spectral band migration from pixel to pixel of the image, and enhances the certainty of the simulation spectral mapping of the remote sensing spectrum.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image simulation, and particularly relates to a spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model. Background Technique

[0002] In the context of the rapid development of space-based remote sensing detection technology, there is an urgent need to obtain a large amount of global remote sensing measured and simulated data in multiple spectral bands and dimensions. For applications based on measured data, samples are often limited to finite detection bands, detection regions, and resolutions. When synthesizing remote sensing images through physical modeling and simulation, a complete physical initial field needs to be prepared, and at the same time, the diversity of random objects such as the ocean and clouds is lacking in imaging.

[0003] In the process of satellite remote sensing imaging simulation and prediction, the diversity and complexity of time-varying elements restrict the accuracy and authenticity. In the traditional remote sensing image simulation process based on the reliability of physical processes, it is necessary to build a full-link from the sensor, atmospheric model to the surface background radiation transfer model. Even so, it is still difficult to reproduce the fine features of different spectral bands, different regions, and different random events. Therefore, when simulating spectral images, it is necessary to introduce deep learning technology to achieve an end-to-end high-confidence and fast simulation method from spectral images to spectral images. A spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model of the present invention is of great significance. Summary of the Invention

[0004] The purpose of the present invention is to provide a spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model. By combining global earth atmosphere environment data, land cover classification data, and spectral imaging data, a method framework for spectral image mapping and feature transfer is built, providing a feasible technical approach for making greater use of existing on-orbit satellite remote sensing data information and evaluating the imaging efficiency of future new bands of satellite payloads.

[0005] To achieve the above purpose, the present invention proposes a spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model, including the following steps:

[0006] S1. Collect multiple groups of matched multispectral remote sensing images, land cover classification maps, and numerical feature product maps and perform preprocessing;

[0007] S2. According to the preprocessed multispectral remote sensing images, land cover classification maps, and numerical feature product maps, determine the input matrix x of the generative adversarial model and determine the size of the output matrix y of the generative adversarial model;

[0008] S3. After processing each group of matched image data respectively, input them into the generative adversarial model in sequence to train the generative adversarial model;

[0009] S4. Input the newly matched image data and complete the spectral domain migration of satellite remote sensing multispectral images using the trained generative adversarial model.

[0010] Among them, step S2 further includes the following steps:

[0011] S21. Process one set of preprocessed multispectral remote sensing images, land cover classification maps, and numerical feature product maps to form a source spectral matrix S in , a target spectral matrix S out , a classification feature matrix C1, and a numerical feature matrix C2;

[0012] S22. Form the input matrix x of the generative adversarial model and determine the size of the output matrix y of the generative adversarial model.

[0013] Among them, step S21 further includes the following steps:

[0014] S211. Form an N c ×W×H classification feature matrix C1 by one-hot encoding the preprocessed land cover classification map, where N c is the classification feature dimension, W is the number of image columns, and H is the number of image rows;

[0015] S212. Normalize all numerical feature products in the preprocessed numerical feature product map respectively so that their value ranges are mapped to 0 to 1, and form an N f ×W×H numerical feature matrix C2, where N f is the number of numerical feature product types;

[0016] S213. Normalize the spectral data of each band in the preprocessed multispectral remote sensing image respectively so that their value ranges are mapped to 0 to 1, and merge the spectral data of the source band and the target band respectively to form an N si ×W×H source spectral matrix S in and an N sout ×W×H target spectral matrix S out , where N si is the dimension of the source spectrum, and N sout is the dimension of the target spectrum.

[0017] Among them, step S22 is specifically: merge the classification feature matrix C1, the numerical feature matrix C2, and the source spectral matrix S in , form an input matrix x of the generative adversarial model with a size of N×W×H, and define the size of the output matrix y of the generative adversarial model to be the same as that of the target spectral matrix S out , which is N sout ×W×H, where N = N c +Nf +N si , where N is the dimension of the input data features.

[0018] Among them, the generative adversarial model consists of a multi-scale generator G and a corresponding multi-scale discriminator D; the multi-scale generator G includes n sub-generators G i , and the multi-scale discriminator D includes n corresponding sub-discriminators D i , where i is an integer from 1 to n.

[0019] Among them, the step S3 further includes the following steps:

[0020] S31. The generative adversarial model includes a multi-scale generator G and a corresponding multi-scale discriminator D. Use the multi-scale generator G to process the input matrix x to generate output matrices y of each scale i ;

[0021] S32. Calculate the loss magnitude L between the output matrices y of each scale i and the target spectral matrix S out ; s ;

[0022] S33. After processing the multiple sets of matched multi-spectral remote sensing images, land cover classification maps, and numerical feature product maps described in S1, input them into the generative adversarial model for training in sequence, and continuously reduce the loss magnitude L s until the model converges.

[0023] Among them, the step S4 further includes the following steps:

[0024] S41. Adopt new satellite remote sensing multi-spectral image data that is consistent with the source bands of the multi-spectral remote sensing image data during training, and adopt a land cover classification map and a numerical feature product map that match the new satellite remote sensing multi-spectral image. Then, use the same steps and parameters as in steps S1 and S2 to process this set of new data to form an input matrix x' of the generative adversarial model;

[0025] S42. Input the input matrix x' into the trained multi-scale generator, simulate and generate an output spectral matrix y' with a size of N sout ×W×H, and then perform an inverse normalization operation on the output spectral matrix y' to obtain the spectral data of the target band, thereby completing the spectral domain migration of the satellite remote sensing multi-spectral image.

[0026] Among them, each set of matched image data is a multi-spectral remote sensing image, a land cover classification map, and a numerical feature product map at the same time and in the same area.

[0027] Among them, preprocessing the multiple groups of matched multispectral remote sensing images, land cover classification maps, and numerical feature product maps specifically includes: making the surface positions pointed by the same coordinate pixels of each image within each group of matched data consistent.

[0028] Among them, the loss magnitude L s is the calculation result of the loss function, and the loss function includes an adversarial loss function L, a cycle consistency loss function L f and a feature matching loss function.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] (1) The prior art usually adopts the method of first inverting the land cover type and then simulating the spectral image. However, the present invention uses the compressed and extracted feature data matrix and spectral matrix as the matching data set to train and generate an adversarial model, realizing the implicit modeling of the spectral responses of elements such as land cover types and clouds, avoiding the construction of a huge land cover spectral response library and the complex radiative transfer simulation process, and achieving direct spectral band migration from image pixels to pixels through deep learning methods;

[0031] (2) The prior art of generative adversarial model technology usually based on image-to-image matching data. However, the present invention adds auxiliary feature dimensions such as land cover classification maps and feature data, and ensures that the input feature data and spectral data are at the same time and region, thereby enhancing the certainty of remote sensing spectral simulation spectral mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a simulation flow chart of a satellite remote sensing multispectral image spectral domain mapping method based on a generative adversarial model of the present invention;

[0033] Figure 2 is a land cover classification map of an embodiment of the present invention;

[0034] Figure 3 is a multispectral remote sensing image of a certain area in an embodiment of the present invention;

[0035] Figure 4 is a true image of the target band in an embodiment of the present invention;

[0036] Figure 5 is the output simulation image of the target band in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The following will combine the Figures 1 to 5 in the embodiments of the present invention to elaborate in detail on the technical solutions, structural features, achieved objectives, and effects in the embodiments of the present invention.

[0038] It should be noted that the attached drawings are in a very simplified form and all use non-precise scales, only for conveniently and clearly assisting in explaining the purpose of the embodiments of the present invention, not for limiting the limiting conditions for the implementation of the present invention. Therefore, they do not have technical substantial significance. Any modification of the structure, change of the proportional relationship or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed by the present invention.

[0039] It should be noted that in the present invention, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements clearly listed, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or device.

[0040] A method for spectral domain mapping of satellite remote sensing multispectral images based on a generative adversarial model, as Figure 1 shown, includes the following steps:

[0041] S1. Collect multiple groups of matching multispectral remote sensing images, land cover classification maps, and numerical feature product maps and perform preprocessing;

[0042] Collect multiple groups of matching multispectral remote sensing images, land cover classification maps, and numerical feature product maps at the same time and in the same area for preprocessing, so that the surface positions pointed to by the same coordinate pixels of each image in each group of matching image data are consistent.

[0043] The numerical feature product includes characteristic information required for simulating the optical radiation transmission process such as cloud data, temperature data, terrain data, wind field data, etc. In specific implementation, all feature data or individual combinations can be used as the numerical feature product map.

[0044] S2. Determine the input matrix x of the generative adversarial model and determine the size of the output matrix y of the generative adversarial model;

[0045] S21. Process the preprocessed multispectral remote sensing image, land cover classification map, and numerical feature product map of one of the groups respectively to form a source spectral matrix S in 、target spectral matrix S out 、classification feature matrix C1 and numerical feature matrix C2;

[0046] S211. Form N through one-hot encoding of the preprocessed land cover classification map cClassification feature matrix C1 of ×W×H, where N c is the classification feature dimension, W is the number of columns of the image, and H is the number of rows of the image;

[0047] S212. Normalize all the numerical feature products in the preprocessed numerical feature product map respectively, so that their value range is mapped to 0 to 1, and form an N f ×W×H numerical feature matrix C2, where N f is the number of numerical feature product types, W is the number of columns of the image, and H is the number of rows of the image.

[0048] In this embodiment, the cloud cover rate is used as the numerical feature product map, then N f is 1.

[0049] S213. Normalize the spectral data of each band in the preprocessed multi-spectral remote sensing image respectively, so that their value range is mapped to 0 to 1, and merge the spectral data of the source band and the spectral data of the target band respectively to form an N si ×W×H source spectral matrix S in and an N sout ×W×H target spectral matrix S out , where N si is the dimension of the source spectrum, and N sout is the dimension of the target spectrum.

[0050] As Figure 2 shown is the land cover map of a certain area adopted in this embodiment. The color grayscale in the figure is the type label, which includes a total of 25 surface types. After preprocessing and one-hot encoding, a classification feature matrix with 25-dimensional features can be output, that is, N c is 25; as Figure 3 shown is the multi-spectral remote sensing image of a certain area adopted in this embodiment.

[0051] S22. Form the input matrix x of the generative adversarial model and determine the size of the output matrix y of the generative adversarial model;

[0052] Merge the classification feature matrix C1, the numerical feature matrix C2 and the source spectral matrix S in , and form an input matrix x of the generative adversarial model with a size of N×W×H, where N = N c +N f +N si , N is the input data feature dimension, W is the number of columns of the image, and H is the number of rows of the image; define the size of the output matrix y of the generative adversarial model to be the same as that of the target spectral matrix S out , which is N sout ×W×H.

[0053] S3. Train the generative adversarial model;

[0054] S31. Process the input matrix x using the multi-scale generator G to generate output matrices y at various scales i ;

[0055] The generative adversarial model consists of a multi-scale generator G and a corresponding multi-scale discriminator D; the multi-scale generator G is composed of n sub-generators, each sub-generator has a symmetric structure, and is sequentially connected by a downsampling network, multiple residual networks, and an upsampling network

[0056] The first-level sub-generator is G0, the input matrix x is input into the upsampling network of G0, processed by multiple residual networks, and output by the downsampling network of G0 as a matrix y0 with dimensions N sout ×W×H; this matrix y0 is the input matrix of the second-level sub-generator G1, and the output matrix of G1 is y1; and so on, the input matrix of sub-generator G i is x i and the output matrix is y i and this output matrix y i is the input matrix of G i+1 sub-generator; where the output matrix y i of each level of sub-generator has dimensions N sout ×W×H

[0057] The multi-scale discriminator D corresponds to the multi-scale generator G and is composed of n sub-discriminators D i and this sub-discriminator D i corresponds to the sub-generator G i in the multi-scale generator G

[0058] S32. Calculate the loss magnitude L i between the output matrix y out at each scale and the target spectral matrix S s ;

[0059] The loss magnitude L i between the output matrix y out at each scale and the target spectral matrix S s is calculated according to the loss function and is a dimensionless value of the calculation result of the loss function; the loss function includes an adversarial loss function L, a cycle consistency loss function L f and a feature matching loss function, and the loss function in this embodiment is the weighted sum of the adversarial loss function L, the cycle consistency loss function L f and the feature matching loss function

[0060] In this embodiment, the multi-scale generator G and the multi-scale discriminator D adopt two scales, that is, n = 2, which are the large-scale model and the small-scale model respectively. The large-scale model can obtain a wider model perception field of view, and the small-scale model can enhance image details. The residual network in the 2-level generator is composed of 6 stacked residual networks.

[0061] Therefore, in this embodiment, the input and output matrices of the multi-scale generator are specifically as follows: The input matrix x is input by the upsampling network of G0, processed by 6 residual networks, and output as matrix y0 by the downsampling network of G0. This matrix y0 is used as the input matrix x1 of the next level and input into G1, and the output matrix of G1 is y1. This matrix y1 is used as the input matrix x2 of the next level and input into G2, and the output matrix of G2 is y2.

[0062] The calculation formula of the adversarial loss function L is where n is the number of scales of the multi-scale generator G. In this embodiment, n = 2; the calculation formula of the cycle consistency loss function L f is L f = ||G(G(x)) - x||1, where G(G(x)) is the final output matrix of the multi-scale generator G in this embodiment; the feature matching loss function calculates the feature matching loss between the real image and the training image using the VGG (super-resolution test sequence) pre-trained model.

[0063] S33. Input each group of matched image data obtained in S1 into the generative adversarial model for training until the generative adversarial model converges.

[0064] Repeat the process of S2 to S32 above. Input multiple groups of matched multi-spectral remote sensing images, land cover classification maps, and numerical feature product maps into the generative adversarial model respectively, and use the adaptive moment estimation Adam algorithm to train the multi-scale generator G of the generative adversarial model. During the training process, continuously reduce the loss size L s , until the loss size L s no longer decreases, so as to achieve model convergence and obtain a trained generative adversarial model.

[0065] S4. Use the trained generative adversarial model to complete the spectral domain migration of satellite remote sensing multi-spectral images.

[0066] S41. Adopt new satellite remote sensing multi-spectral image data that is consistent with the source band of the multi-spectral remote sensing image data during training, and adopt the land cover classification map and numerical feature product map that match the new satellite remote sensing multi-spectral image. Then, use the same steps and parameters as in steps S1 and S2 to form the input matrix x' of the generative adversarial model.

[0067] S42. Simulate and generate an output spectral matrix y' of size N sout ×W×H using the multi-scale generator in the trained generative adversarial model in S3, and then perform an inverse normalization operation on the output spectral matrix y' to obtain the spectral data of the target band, thereby completing the spectral domain migration of the satellite remote sensing multi-spectral image.

[0068] As Figure 4 shown is the real image of the target band in this embodiment; Figure 5 is the output simulation image of the target band using the method of the present invention in this embodiment.

[0069] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be recognized that the above description should not be considered as a limitation of the present invention. After those skilled in the art have read the above content, various modifications and substitutions to the present invention will be obvious. Therefore, the protection scope of the present invention should be defined by the appended claims.

Claims

1. A spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model, characterized in that, It includes the following steps: S1. Collect multiple sets of matched multispectral remote sensing images, land cover classification maps, and numerical feature product maps and perform preprocessing; S2. Determine the input matrix x of the generative adversarial model based on the preprocessed multispectral remote sensing images, land cover classification maps, and numerical feature product maps, and determine the size of the output matrix y of the generative adversarial model; S3. After processing each set of matched image data respectively, input them into the generative adversarial model in sequence to train the generative adversarial model; S4. Input new matched image data and use the trained generative adversarial model to complete the spectral domain migration of satellite remote sensing multispectral images; The step S2 further includes the following steps: S21. Process one set of preprocessed multispectral remote sensing images, land cover classification maps, and numerical feature product maps to form a source spectral matrix Sin, a target spectral matrix Sout, a classification feature matrix C1, and a numerical feature matrix C2; S22. Form the input matrix x of the generative adversarial model and determine the size of the output matrix y of the generative adversarial model; The step S21 further includes the following steps: S211. Form an N c ×W×H classification feature matrix C1 from the preprocessed ground object classification map through one-hot encoding, where N c is the classification feature dimension, W is the number of columns of the image, and H is the number of rows of the image; S212. Normalize all the numerical feature products in the preprocessed numerical feature product map respectively, so that their value ranges are mapped to 0 to 1, forming a numerical feature matrix C2 of N f ×W×H, where N f is the number of numerical feature product types; S213. Normalize the spectral data of each band in the preprocessed multi-spectral remote sensing image respectively, so that its value range is mapped to 0 to 1, and combine the spectral data of the source band and the target band respectively to form an N si ×W×H source spectral matrix S in and an N sout ×W×H target spectral matrix S out , where N si is the dimension of the source spectrum, and N sout is the dimension of the target spectrum; The specific steps of S22 are as follows: combining the classification feature matrix C1, the numerical feature matrix C2 and the source spectral matrix S in , to form the input matrix x of the generative adversarial model with a size of N×W×H, where N = N c + N f + N si , N is the input data feature dimension; defining that the size of the output matrix y of the generative adversarial model is the same as that of the target spectral matrix S out , which is N sout ×W×H.

2. The satellite remote sensing multispectral image spectral domain mapping method based on a generative adversarial model according to claim 1, wherein The generative adversarial model consists of a multi-scale generator G and a corresponding multi-scale discriminator D; the multi-scale generator G includes n sub-generators G i , and the multi-scale discriminator D includes n corresponding sub-discriminators D i , where i is an integer from 1 to n.

3. A spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model according to claim 2, characterized in that, The step S3 further includes the following steps: S31. Process the input matrix x using the multi-scale generator G to generate the output matrix y of each scale generator i ; S32. Calculate the output matrix y of each scale generator through the corresponding multi-scale discriminator D i and the target spectral matrix S out to obtain the loss magnitude L s ; S33. After processing the multiple groups of matched hyperspectral remote sensing images, land cover classification maps, and numerical feature product maps described in S1, input them into the generative adversarial model for training respectively, and continuously reduce the loss magnitude L during the training process s until the generative adversarial model converges.

4. The spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model according to claim 1, characterized in that The step S4 further includes the following steps: S41. Adopt new satellite remote sensing multispectral image data that is consistent with the source band of the multispectral remote sensing image data during training, and adopt a land cover classification map and a numerical feature product map that match the new satellite remote sensing multispectral image. Then, use the same steps and parameters as in steps S1 and S2 to process this set of new data to form the input matrix x′ of the generative adversarial model; S42. Input the input matrix x′ into the trained multi-scale generator G to generate an output spectral matrix y′ with dimensions N sout ×W×H. Then perform an inverse normalization operation on the output spectral matrix y′ to obtain the spectral data of the target band, thereby completing the spectral domain migration of the satellite remote sensing multi-spectral image.

5. The spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model according to claim 1, characterized in that Each set of matched image data is a multispectral remote sensing image, a land cover classification map, and a numerical feature product map at the same time and in the same area.

6. The spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model according to claim 4, characterized in that The preprocessing of the multiple sets of matched multispectral remote sensing images, land cover classification maps, and numerical feature product maps is specifically: making the surface positions pointed to by the same coordinate pixels of each image within each set of matched data consistent.

7. A spectral domain mapping method for satellite remote sensing multispectral images based on a generative adversarial model according to claim 3, characterized in that The loss magnitude L s is the calculation result of a loss function, and the loss function includes an adversarial loss function L, a cycle consistency loss function L f and a feature matching loss function.

Images