A sample optimized passive microwave data seamless reconstruction deep learning method

Abstract

The present application belongs to the technical field of remote sensing image intelligent processing, and particularly relates to a sample-optimized passive microwave data seamless reconstruction deep learning method. The present application acquires passive microwave data with track gap loss, fully utilizes the periodicity of the track gap and the spatial complementarity of adjacent time phases, optimizes samples through a small-range loss sample interpolation generation method and a single-cycle mask strategy, fuses U-net and Transformer to construct a multi-time phase feature connection reconstruction network model, realizes seamless reconstruction, and obtains remote sensing data with continuous distribution and seamless connection between the reconstructed area and the original area. The present application effectively solves the limitation that the existing deep learning reconstruction method has obvious "seam" marks in the reconstruction result when the sample quantity is small and the sample features are not rich enough, and is suitable for the reconstruction of all passive microwave image data or quantitative parameter data with track gaps, which will play an important role in the spatial complete monitoring of global or regional scale passive microwave data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent processing of remote sensing images, and particularly relates to a sample-optimized passive microwave data seamless reconstruction deep learning method. BACKGROUND

[0002] Passive microwave remote sensing has all-weather monitoring capability and strong penetration capability for clouds and rainfall, and is an important data source for obtaining all-weather geophysical parameters and plays an important role in the field of climate change monitoring. However, the specific scanning mode of passive microwave sensors (such as AMSR-E, AMSR2, MWRI) causes scanning gaps between adjacent orbits, resulting in missing passive microwave data, which limits the practical application of the data at the global or regional scale. Therefore, it is an important topic to use technical means to repair the missing parts in the existing passive microwave data as necessary.

[0003] The conventional method adopts inverse distance weighting, singular spectrum decomposition, empirical orthogonal function, and geographic and time weighted regression method to fill the inter-orbit gaps in passive microwave data. However, the performance of the conventional method is unstable when reconstructing large inter-orbit gaps in passive microwave data. The machine learning method has strong non-linear feature mining capability, and some scholars consider the potential non-linear relationship between passive microwave products and environmental variables, but face the problem of selecting which auxiliary data to participate in the repair, and different auxiliary data are affected by factors including surface feature spectrum or radiation characteristics, scale difference, observation time, and atmosphere, thereby greatly interfering with product repair. To more efficiently and accurately repair passive microwave data, some scholars consider the spatial complementarity of adjacent time phases and the periodicity of gap position changes, fully utilize effective spatial complementary information, do not need auxiliary data, construct a multi-time phase deep learning model to obtain the non-linear relationship between multi-time phase data, and then repair the missing data, but the position change of different time orbit gaps leads to insufficient training samples in some scenarios, and the repeated mask operation in the multi-time phase deep learning model causes the reconstructed results to be heavily dependent on multi-time phase auxiliary data, and the reconstructed results still have "seam" traces. SUMMARY

[0004] The present application aims to solve the problem of obvious "seam" traces in the reconstructed results of the existing passive microwave inter-orbit gap data reconstruction deep learning method when the number of samples is small and the sample features are not rich enough, and provides a sample-optimized passive microwave data seamless reconstruction deep learning method.

[0005] To achieve the above technical purposes and effects, the present application is implemented by the following technical solutions:

[0006] A sample optimized passive microwave data seamless reconstruction deep learning method, comprising the following steps:

[0007] Step one, obtain multi-temporal passive microwave data or products with track gaps, and perform necessary preprocessing to ensure comparability of multi-temporal data;

[0008] Step two, construct and optimize a multi-temporal passive microwave data sample set using sample nearest neighbor interpolation method and single mask strategy;

[0009] Step three, fuse U-net and Transformer to construct a multi-temporal seamless reconstruction network framework;

[0010] Step four, model training strategy;

[0011] Step five, reconstruct the track gap area in passive microwave data using batch processing method.

[0012] Further, in step one, the preprocessing method includes cropping and normalization.

[0013] Further, in step two, the specific steps for constructing and optimizing the multi-temporal passive microwave data sample set are:

[0014] 1) Analyze the track gap change characteristics of passive microwave data, consider its periodicity and complementarity, and use continuous 3-day passive microwave sample data as appropriate adjacent phase combination;

[0015] 2) Perform data enhancement operations on the original passive microwave data, including rotation according to a certain angle, horizontal flipping and vertical flipping;

[0016] 3) Traverse the cropped continuous 3-day images, if the continuous 3-day image blocks are complete, store them directly into the sample library as sample pairs; the continuous 3-day images are denoted as T1, T2 and T3, where T2 is the target data to be reconstructed, T1 and T3 are multi-temporal auxiliary data; if there is a situation that the proportion of original pixels in the continuous 3-day image blocks is greater than the threshold m, and the proportion of pixels outside the test area is less than the threshold n, then perform nearest neighbor interpolation processing on the missing image blocks; if the interpolation result is between the maximum value max and the minimum value min of the image block, it is considered that the interpolation result is reasonable, and the sample pair is also stored in the sample library;

[0017] 4) Using the image blocks not stored in the sample library to make a track mask library, selecting representative image blocks to make a track gap mask file, ensuring that the number of mask files is greater than the number of target data to be reconstructed; performing single-cycle masking on the target data to be reconstructed, i.e. traversing the continuous 3-day data pairs in the sample set, sequentially selecting a single mask in the mask set to mask the target data to be reconstructed, and combining the output mask result T2 with the corresponding original auxiliary data T1 and T3 to form a new training sample pair, and the original T2 is the label data, thereby obtaining the training sample set.

[0018] Further, the final network input is: the spatially complete target image block T2 as the label, the auxiliary phase complete image blocks T1 and T3, and the target phase simulation mask image block T2.

[0019] Further, in step three, the seamless reconstruction network model includes three parts: downsampling, upsampling, and jump connection module.

[0020] Further, the specific steps of constructing the multi-temporal seamless reconstruction network framework are:

[0021] 1) In the downsampling part, a three-layer convolutional network is used as a feature extractor to extract feature maps with semantic information from the original passive microwave data, and the feature maps are linearly projected, serialized, and position-encoded before being input into the Transformer encoder;

[0022] 2) In the upsampling part, transposed convolution is used in combination with the recovered feature maps to realize the mapping of passive microwave data from low resolution to high resolution;

[0023] 3) Between the downsampling and upsampling, a jump connection structure is used to fuse the feature maps generated at each level in the downsampling stage with the corresponding feature maps of the same level in the upsampling stage, so as to retain the feature information at different levels and improve the accuracy of the reconstruction result.

[0024] Further, in step four, the specific steps of the model training strategy are:

[0025] 1) In the training sample set, the target phase mask result T2 and the corresponding two adjacent phase auxiliary data are connected in the channel as the input of the convolutional neural network;

[0026] 2) Introducing mean square error as an adaptive loss function, to avoid introducing errors generated by nearest neighbor interpolation, only using pixels not processed by nearest neighbor interpolation to adaptively participate in loss function calculation, and its expression is:

[0027]

[0028] Wherein, Xi, Yi respectively represent the original value and the reconstructed value of the i-th pixel in the image without interpolation, and m represents the number of pixels participating in the calculation.

[0029] Further, the specific steps of reconstructing the orbital gap region in the passive microwave data in a batch processing manner are as follows:

[0030] 1) By means of conditional clipping, using a moving window to set a sliding step, continuous multi-day passive microwave data required by the model input layer is generated, and the reconstruction result of the to-be-reconstructed time phase data is obtained after model reconstruction;

[0031] 2) Data splicing is performed by using an average weight splicing method, in the repeated area of adjacent image blocks, the average value of the pixels in the repeated area is taken as the reconstruction result;

[0032] 3) The conditional clipping window moves according to the date, and the result of the previous round of reconstruction is taken as the auxiliary time phase data input into the model in the next round of reconstruction.

[0033] The beneficial effects of the present application are:

[0034] The present application combines the periodicity of the orbital gap of the passive microwave data and the adjacent space-time complementary distribution characteristics, optimizes the samples through the sample nearest neighbor interpolation method and the single mask strategy, fully utilizes the data in the non-missing area, and effectively solves the limitation that the reconstruction result has obvious "seam" marks when the sample number is small and the sample characteristics are not rich enough in the traditional method. First, multi-time phase passive microwave data with orbital gaps are obtained and preprocessed; second, data enhancement and clipping are performed to obtain data image blocks, nearest neighbor interpolation processing is performed on the missing image blocks that meet the conditions, a simulated mask image block is obtained by using a single cycle mask, and an optimized sample data set is obtained; finally, a reconstruction model with multi-time phase feature connection is constructed by fusing U-net and Transformer, and the training and testing of the model are carried out, so as to obtain seamless passive microwave data. The present application has strong universality and high practical value, is suitable for reconstruction of all passive microwave image data or quantitative parameter data with orbital gaps, and will play an important role in spatial complete monitoring of passive microwave data at global or regional scale.

[0035] Of course, implementing any product of the present application does not necessarily need to achieve all the advantages above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed for the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0037] Figure 1 The flowchart of the present application is shown. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0039] Embodiment:

[0040] In this embodiment, the AMSR-E land surface temperature product in the earth observation system is taken as an example, and the seamless reconstruction of land surface temperature is performed by using the sample-optimized passive microwave data seamless reconstruction deep learning method. As shown in Figure 1 The embodiment flow of the present embodiment can be divided into the following five stages:

[0041] First stage: AMSR-E land surface temperature data acquisition and preprocessing

[0042] Step 1-1, obtaining original AMSR-E land surface temperature data, the pixel value unit of which is Kelvin (K), the size of the to-be-measured region is 248 pixels*143 pixels, the spatial resolution is 25km, and each image has 1-3 track gaps.

[0043] Second stage: multi-temporal land surface temperature sample data set construction and optimization

[0044] Step 2-1, the track gap divides the to-be-measured region into at least three parts, causing the discontinuity of AMSR-E land surface temperature in the to-be-measured region. Since the track gap of T2 (the second day) AMSR-E land surface temperature data is complementary to the track gap of T1 (the first day) and T3 (the third day), the multi-temporal land surface temperature data sample set needs to be constructed by using the land surface temperature sample data of three consecutive days.

[0045] Step 2-2, in order to improve the training performance, data enhancement operation is performed on the original AMSR-E land surface temperature image, that is, rotation is performed at a certain angle, and horizontal flipping, vertical flipping and normalization are performed. At the same time, the images of three consecutive days are cut according to the size of 24*24 and the sliding step of 4.

[0046] Step 2-3, sample optimization is performed using nearest neighbor interpolation. For three consecutive days (T1, T2, T3) of images, if the three image blocks are complete, they are directly stored in the sample library as a sample pair. T2 is the target data to be reconstructed, and T1 and T3 are auxiliary data. If the proportion of missing pixels in the three consecutive image blocks is greater than the threshold p, and the proportion of pixels outside the test area is less than the threshold q, nearest neighbor interpolation is performed on the missing image block. If the interpolation result is between the maximum value max and the minimum value min of the image block, the interpolation result is considered reasonable, and the sample pair is also stored in the sample library. The threshold p used in this embodiment is 90%, and the threshold q is 0.9%.

[0047] Step 2-4, for the obtained continuous 3-day (T1, T2, T3) complete cutting image block, in order to better fit the original reconstruction situation, it is necessary to perform a simulated mask operation on the complete image block of T2. Use the image blocks not stored in the sample library to make a track mask library, select representative image blocks to make track gap mask files, and ensure that the number of mask files is greater than the number of target data to be reconstructed. Select a single mask from the mask set to mask the target data to be reconstructed, and output the mask result T2 and the corresponding original auxiliary data T1 and T3 to form a new training sample pair. The original T2 is the label data, and the T1, T2, T3, T2_mask data pair is reconstituted to obtain a training sample set. The final network input data is 3-time 4-class data: T1 complete image block, T2 complete image block (label), T3 complete image block, and T2 simulated mask image block (to be reconstructed data, obtained by simulating a mask from the T2 complete image block).

[0048] Third stage: multi-time seamless reconstruction network model building and training

[0049] The seamless reconstruction network model includes three parts: downsampling, upsampling, and skip connection structure.

[0050] Step 3-1, in the downsampling part, a three-layer convolutional network is used as a feature extractor to extract feature maps with semantic information from the original passive microwave data. After linear projection, the feature maps are serialized and position encoded, and input into the Transformer encoder.

[0051] Step 3-2, in the upsampling part, transposed convolution is used to combine the recovered feature maps to realize the mapping of passive microwave data from low resolution to high resolution.

[0052] Step 3-3, between downsampling and upsampling, a skip connection structure is used to fuse the feature maps generated at each level in the downsampling stage with the corresponding feature maps of the same level in the upsampling stage, so as to retain the feature information of different levels and improve the accuracy of the reconstruction result.

[0053] Fourth stage: model training strategy

[0054] Step 4-1, in the training sample set, in order to effectively utilize the feature information and change information of the surface temperature of each time phase, the multi-time phase data needs to be combined, and the specific combination method is: adding T1 and T2, and adding T3 and T2; connecting T1, T2, T3 and the two addition results in the channel to obtain the input layer (including 5 layers, T1, T2, T3, T1+T2, T2+T3 layers).

[0055] Step 4-2, introducing mean square error as an adaptive loss function, in order to avoid introducing errors generated by the nearest neighbor interpolation, only the pixels not subjected to nearest neighbor interpolation are used to participate in the loss function calculation, and the expression is:

[0056]

[0057] Wherein, Xi, Yi respectively represent the original value and the reconstructed value of the i-th pixel in the image not subjected to interpolation, and m represents the number of pixels participating in the calculation.

[0058] Fifth stage: batch processing method to reconstruct the orbit gap area in AMSR-E land surface temperature data

[0059] Step 5-1, using conditional clipping, a moving window clipping is used in the clipping process, a sliding step is set, and the data required by the model input layer is generated, and the reconstructed result of the T2 time phase data is obtained after the model reconstruction.

[0060] Step 5-2, in the splicing process, an average weight splicing method is adopted. In the repeated area of adjacent image blocks, the average value of the pixels in the repeated area is taken as the pixel value, so as to improve the accuracy of the reconstructed result.

[0061] Step 5-3, the conditional clipping window moves according to the date, and the reconstructed result of T2 in the previous round can be used as the auxiliary time phase T1 input data in the next round of reconstruction.

[0062] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details, and the application is not limited to the specific embodiments. Obviously, according to the content of the specification, many modifications and changes can be made. The embodiments are selected and described in the specification in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application. The application is limited by the claims and their entire scope and equivalents.

Claims

1. A sample-optimized passive microwave data seamless reconstruction deep learning method, characterized in that, The method comprises the following steps: Step one, obtaining multi-temporal passive microwave data or products with track gaps and performing necessary preprocessing to ensure the comparability of multi-temporal data; Step two, constructing and optimizing a multi-temporal passive microwave data sample set by using a sample nearest neighbor interpolation method and a single mask strategy; the specific steps are as follows: 1) analyzing the track gap change characteristics of passive microwave data, considering the periodicity and complementarity thereof, and using continuous 3-day passive microwave sample data as a suitable adjacent phase combination; 2) performing data enhancement operations on the original passive microwave data, including rotation according to a certain angle, horizontal flipping and vertical flipping; 3) traversing the cropped continuous 3-day images, if each image block of the continuous 3-day images is complete, the image block is directly stored in the sample library as a sample pair; the continuous 3-day images are respectively denoted as T1, T2 and T3, wherein T2 is the target data to be reconstructed, and T1 and T3 are multi-temporal auxiliary data; if there is an image block with an original pixel ratio greater than a threshold m and a continental region pixel ratio less than a threshold n, the missing image block is processed by nearest neighbor interpolation, if the interpolation result is between the maximum value max and the minimum value min of the image block, the interpolation result is considered reasonable, and the sample pair is also stored in the sample library; 4) using the image blocks not stored in the sample library to make a track mask library, selecting representative image blocks in the track mask library as track gap mask files, ensuring that the number of mask files is greater than the number of target data to be reconstructed; performing single cycle masking on the target data to be reconstructed, that is, traversing the continuous 3-day data pairs in the sample set, sequentially selecting a single mask in the mask set to mask the target data to be reconstructed, and combining the output mask result T2 with the corresponding original auxiliary data T1 and T3 to form a new training sample pair, wherein the original T2 is the label data, and then a training sample set is obtained; Step three, fusing U-net and Transformer to construct a multi-temporal seamless reconstruction network framework; Step four, model training strategy; Step five, reconstructing the track gap region in the passive microwave data in a batch processing manner.

2. The sample-optimized passive microwave data seamless reconstruction deep learning method of claim 1, wherein, In step one, the preprocessing methods include cropping and normalization.

3. The sample-optimized passive microwave data seamless reconstruction deep learning method according to claim 2, characterized in that, The final network input is: a spatially complete target image block T2 as a label, auxiliary phase complete image blocks T1 and T3, and a simulated mask image block T2 to be reconstructed.

4. The sample-optimized passive microwave data seamless reconstruction deep learning method according to claim 3, characterized in that, In step three, the seamless reconstruction network model comprises three parts of downsampling, upsampling and jump connection modules.

5. The sample-optimized passive microwave data seamless reconstruction deep learning method according to claim 4, characterized in that, The specific steps of constructing the multi-temporal seamless reconstruction network framework are as follows: 1) in the downsampling part, a three-layer convolutional network is used as a feature extractor to extract a feature map with semantic information from the original passive microwave data, and the feature map is linearly projected, serialized and position-encoded and input into a Transformer encoder; 2) in the upsampling part, a transposed convolution is used in combination with a restored feature map to realize the mapping of passive microwave data from low resolution to high resolution; 3) A skip connection structure is used between down-sampling and up-sampling to fuse the feature maps of each level generated in the down-sampling stage with the corresponding up-sampled feature maps, so as to retain the feature information of different levels and improve the accuracy of the reconstruction result.

6. The sample-optimized passive microwave data seamless reconstruction deep learning method according to claim 5, characterized in that, In step four, the specific steps of the model training strategy are as follows: 1) In the training sample set, the to-be-reconstructed time phase mask result T2 and the corresponding two adjacent time phase auxiliary data are connected in the channel as the input of the convolutional neural network; 2) The mean square error is introduced as the adaptive loss function, in order to avoid introducing errors generated by the nearest neighbor interpolation, only the pixels not subjected to the nearest neighbor interpolation are used to adaptively participate in the loss function calculation, and the expression is as follows: ； Wherein, Xi and Yi represent the original value and the reconstructed value of the i th pixel in the image which is not subjected to the interpolation, and m represents the number of pixels participating in the calculation.

7. The sample-optimized, passive-microwave-data seamless-reconstruction deep learning method of claim 6, wherein, In step five, the specific steps of reconstructing the orbital gap region in passive microwave data by batch processing are as follows: 1) By means of conditional clipping, a moving window is used to set a sliding step to generate continuous multi-day passive microwave data required by the input layer of the model, and the reconstruction result of the to-be-reconstructed time phase data is obtained after the model reconstruction; 2) The data splicing is performed by using the average weight splicing method, in the repeated area of adjacent image blocks, the average value of the pixels in the repeated area is taken as the reconstruction result; 3) The conditional clipping window moves according to the date, and the result of the previous round of reconstruction is used as the auxiliary time phase data input into the model in the next round of reconstruction.

Images