A method, device, equipment, medium and product for filling missing pixels of a cloud-polluted remote sensing image

Abstract

The application discloses a cloud-polluted remote sensing image missing pixel filling method, device, equipment, medium and product, and relates to the technical field of remote sensing image processing. The method comprises the following steps: acquiring a plurality of remote sensing images to be filled and acquiring original pixel matrices thereof, preliminarily filling part of the missing pixels through spatial information, marking the missing pixels to be filled, and forming a missing pixel matrix to be filled; extracting the pixels at each pixel position in all the missing pixel matrices to be filled, constructing a plurality of one-dimensional original time sequence arrays, inputting the one-dimensional original time sequence arrays into a SAITS model to predict and fill the missing pixels, and generating one-dimensional filled time sequence arrays. According to the plurality of one-dimensional filled time sequence arrays, a remote sensing filled image is reconstructed. The application considers the spatial neighborhood information of the remote sensing image and fully utilizes the time sequence information in the time sequence, effectively improves the data quality of the cloud-polluted remote sensing image, and thus generates a remote sensing image with spatial and temporal continuity and high precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, in particular to a method and device for filling missing pixels of a cloud-contaminated remote sensing image, equipment, medium and product. BACKGROUND

[0002] With the rapid development of satellite sensors and remote sensing inversion technology, remote sensing data has become an indispensable information base in many research fields. Its characteristics of wide range, multi-temporal and all-weather provide rich information sources for agricultural monitoring, urban planning, earth science and climate change research. However, the problem of cloud contamination has always been an important factor restricting the application of remote sensing data. More than half of the optical remote sensing images are affected, and the data availability is greatly limited. In recent years, the demand for remote sensing images has increased dramatically, and the problem of cloud contamination has become increasingly prominent, and effective solutions are urgently needed.

[0003] Although some techniques for filling missing pixels and benchmark data sets have been developed, the problem of non-uniform data quality caused by multi-source data fusion, and the limitations of single-sensor methods in temporal and spatial filling, still limit the improvement of filling performance. Especially the differences in spatial, temporal and spectral resolution of multi-source data, and the lack of single-sensor images in time and space, limit the filling effect and bring many challenges to the filling work. SUMMARY

[0004] The purpose of the present application is to provide a method and device for filling missing pixels of a cloud-contaminated remote sensing image, which can fill the missing pixels of a cloud-contaminated remote sensing image, thereby generating a high-precision image with temporal and spatial continuity.

[0005] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a method for filling missing pixels of a cloud-contaminated remote sensing image, comprising:

[0007] acquiring a plurality of remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained by any two remote sensing images to be filled are different;

[0008] obtaining the original pixel matrix of each remote sensing image to be filled respectively;

[0009] traversing each original pixel matrix respectively, if the i-th pixel in the original pixel matrix is a cloud contaminated pixel, it is judged whether there is a non-cloud contaminated pixel in a preset window centered on the i-th pixel to obtain a first judgment result; if the first judgment result is yes, the mean value of all non-cloud contaminated pixels in the preset window is calculated, and the mean value is filled into the position of the i-th pixel; if the first judgment result is no, the i-th pixel is marked as a missing pixel to be filled; until the traversal of all original pixel matrices is completed, a plurality of missing pixel matrices to be filled of the remote sensing images to be filled are obtained;

[0010] extracting the pixel at the i-th pixel position in the missing pixel matrix to be filled of each remote sensing image to be filled, and sorting in chronological order to obtain a plurality of one-dimensional original time series arrays;

[0011] inputting the plurality of one-dimensional original time series arrays into a SAITS (Self-Attention-based Imputation for Time Series) model, and the SAITS model respectively predicts and fills the missing pixel value at the t-th moment according to the non-cloud contaminated pixels corresponding to all moments except the t-th moment in each one-dimensional original time series array to obtain a plurality of one-dimensional filled time series arrays;

[0012] reconstructing a remote sensing filled image according to the plurality of one-dimensional filled time series arrays.

[0013] Optionally, the original pixel matrix of each remote sensing image to be filled is obtained respectively, and the previous further comprises:

[0014] preprocessing the plurality of remote sensing images to be filled; the preprocessing specifically comprises: setting the image range, spatial resolution and spatial reference coordinate system of the plurality of remote sensing images to be filled as a preset image range, a preset spatial resolution and a preset spatial reference coordinate system respectively, and labeling the cloud contaminated pixels in each remote sensing image to be filled as "nan".

[0015] Optionally, the mean value of all non-cloud contaminated pixels in the preset window is calculated according to the following formula:

[0016]

[0017] wherein B is an interpolation image pixel matrix, A is an original pixel matrix, K represents a weight matrix of a first preset size sliding window, i=(0, 1, 2,..., i max ) represents the row number of the original pixel matrix, j=(0, 1, 2,..., j max) represents the column number of the original pixel matrix, m and n respectively represent the row offset and column offset of the pixel in the sliding window of the first preset size relative to the current interpolation position (i, j) in the original pixel matrix, and Num is the total number of non-cloud-contaminated pixels in the sliding window of the first preset size.

[0018] Optionally, the SAITS model specifically includes: a first diagonal masked self-attention block, a second diagonal masked self-attention block and a weighted combination module;

[0019] The first diagonal masked self-attention block is connected to the second diagonal masked self-attention block and the weighted combination module respectively, and the second diagonal masked self-attention block is connected to the weighted combination module;

[0020] The first diagonal masked self-attention block is used to take the one-dimensional original time series array as input, learn the relationship between the pixel at time t and the pixels before and after time t in the time dimension, preliminarily predict the filling value of the missing pixel, and output a preliminary one-dimensional filled time series array;

[0021] The second diagonal masked self-attention block is used to re-learn the relationship between the pixel at time t and the pixels before and after time t in the time dimension, re-predict the filling value of the missing pixel, and output the second-filled one-dimensional filled time series array;

[0022] The weighted combination module is used to weightedly fuse the preliminary one-dimensional padded time series array output by the first diagonally masked self-attention block and the secondary padded one-dimensional padded time series array output by the second diagonally masked self-attention block, and output the final one-dimensional padded time series array.

[0023] Optionally, the calculation formula of the first diagonal masked self-attention block is as follows:

[0024]

[0025]

[0026] z = {FFN(DiagMaskedMHA(e))} N ;

[0027]

[0028] in, is the padding value of the first diagonal masked self-attention block, is a one-dimensional original time series array, p is the position code, Concat is the array concatenation function, FFN is the feedforward network, DiagMaskedMHA is the multi-head diagonal mask attention mechanism, is the initial one-dimensional padded time series array, {}N is the execution of N times of nested attention function and feedforward network, We , b e , and W z are a first analog coefficient, a second analog coefficient, and a third analog coefficient, respectively, e and z are a first intermediate variable and a second intermediate variable, respectively, and is an element product, a mask vector of the one-dimensional original time series array;

[0029] The specific implementation formula of the second diagonal masked self-attention block is as follows:

[0030]

[0031] β={FFN(DiagMaskedMHA(α))} N ;

[0032] wherein, is a padding value of the second diagonal masked self-attention block, W α , W β , W γ , b α , b β , and b γ are a fourth analog coefficient, a fifth analog coefficient, a sixth analog coefficient, a seventh analog coefficient, an eighth analog coefficient, a ninth analog coefficient, and a tenth analog coefficient, respectively, and α and β are a third intermediate variable and a fourth intermediate variable, respectively;

[0033] The specific implementation formula of the weighted combination module is as follows:

[0034]

[0035]

[0036] wherein, is a missing pixel padding value, η is a combination weight generated by taking the average attention weight and the missing mask as references, is a weight result based on η.

[0037] Optionally, the remote sensing padding image is reconstructed according to the plurality of one-dimensional padding time series arrays, and then the method further comprises:

[0038] The noise of the remote sensing padding image is eliminated by a median filtering method.

[0039] The formula of the median filtering method is as follows:

[0040] h(i,j)=med{f(i-k,j-l)},k,l∈W;

[0041] Wherein, h is the remote sensing filled image after noise elimination, f(i-k, j-l) is the pixel of the remote sensing filled image at the position (i-k, j-l), W={-1, 0, 1}, k and l are respectively the row offset and column offset relative to the current position (i, j).

[0042] In a second aspect, the present application provides a device for filling missing pixels of a cloud-polluted remote sensing image, comprising:

[0043] A remote sensing image acquisition module is configured to acquire a plurality of to-be-filled remote sensing images according to a preset time interval; wherein each to-be-filled remote sensing image contains image data of only one band, and the bands contained by any two to-be-filled remote sensing images are different;

[0044] A pixel matrix acquisition module is configured to acquire an original pixel matrix of each to-be-filled remote sensing image respectively;

[0045] A spatial filling module is configured to traverse each original pixel matrix respectively, if the i-th pixel in the original pixel matrix is a cloud-polluted pixel, determine whether there is a non-cloud-polluted pixel in a preset window centered on the i-th pixel to obtain a first determination result; if the first determination result is yes, calculate the mean value of all non-cloud-polluted pixels in the preset window, and fill the mean value to the position of the i-th pixel; if the first determination result is no, mark the i-th pixel as a missing pixel to be filled; until the traversal of all original pixel matrices is completed, to obtain a missing pixel matrix to be filled of the plurality of to-be-filled remote sensing images;

[0046] A one-dimensional array acquisition module is configured to extract the pixel at the i-th pixel position in the missing pixel matrix to be filled of all to-be-filled remote sensing images, and sort them in chronological order to obtain a plurality of one-dimensional original time series arrays;

[0047] A temporal filling module is configured to input the plurality of one-dimensional original time series arrays into a SAITS model, and the SAITS model predicts and fills the missing pixel value at the t-th time according to the non-cloud-polluted pixels corresponding to all times except the t-th time in each one-dimensional original time series array, to obtain a plurality of one-dimensional filled time series arrays;

[0048] A filled image acquisition module is configured to reconstruct a remote sensing filled image according to the plurality of one-dimensional filled time series arrays.

[0049] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the missing pixel filling method of the cloud-polluted remote sensing image according to any one of the above.

[0050] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program which, when executed by a processor, implements the steps of the method for filling in missing pixels of a cloud-contaminated remote sensing image according to any one of the preceding aspects.

[0051] In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method for filling in missing pixels of a cloud-contaminated remote sensing image according to any one of the preceding aspects.

[0052] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0053] The present application provides a method, device, equipment, medium and product for filling in missing pixels of a cloud-contaminated remote sensing image. By obtaining a plurality of remote sensing images to be filled in and obtaining the original pixel matrix of each image, the cloud-contaminated pixels are preliminarily judged and locally filled in, and a plurality of missing pixel matrices to be filled in are obtained. The present application solves the problem that the traditional method does not fully utilize the neighborhood information when processing the cloud-contaminated remote sensing image, realizes the preliminary and effective filling in of the local cloud-contaminated pixels, and improves the local integrity of the image. The pixels at each pixel position in all the missing pixel matrices to be filled in are extracted, a plurality of one-dimensional original time sequence arrays are constructed, and the plurality of one-dimensional original time sequence arrays are input into the SAITS model to predict the filling value of the missing pixels. The present application solves the problem that a single image is difficult to accurately fill in a large area of missing pixels under complex cloud contamination, realizes high-precision filling based on time sequence analysis and prediction, and improves the quality and usability of the overall image. The present application utilizes the advantages of the first law of geology "similar to similar" and the characteristics of the SAITS model that can clearly capture the time dependence and feature correlation between time sequence pixels, so as to obtain a high-precision filled image. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0055] Figure 1 A flowchart of a method for filling in missing pixels of a cloud-contaminated remote sensing image according to an embodiment of the present application is shown in the figure.

[0056] Figure 2 A detailed flowchart of the step of filling in missing pixels according to spatial information in the embodiment is shown in the figure. Figure 1

[0057] ​Figure 3 For Figure 1 A detailed flowchart of the step of filling in missing pixels according to the timing information;

[0058] Figure 4 A comparison chart of a remote sensing image to be filled in and a remote sensing image after spatial filling provided by an embodiment of the present application; Figure 4 (a) is a schematic diagram of a remote sensing image to be filled in, Figure 4 (b) is a schematic diagram of a remote sensing image after spatial filling;

[0059] Figure 5 A remote sensing image data processing chart after spatial filling provided by an embodiment of the present application;

[0060] Figure 6 A functional module schematic diagram of a missing pixel filling device for a cloud-polluted remote sensing image provided by an embodiment of the present application;

[0061] Figure 7 A pixel filling precision chart of a missing pixel filling method for a cloud-polluted remote sensing image provided by an embodiment of the present application; wherein, Figure 7 (a) is a root mean square error chart, Figure 7 (b) is an absolute average error chart;

[0062] Figure 8 A comparison chart of two remote sensing images to be filled in and remote sensing filling images provided by an embodiment of the present application; wherein, Figure 8 (a) is a schematic diagram of a remote sensing image to be filled in, Figure 8 (b) is a remote sensing filling image schematic diagram of Figure 8 (a), Figure 8 (c) is a schematic diagram of another remote sensing image to be filled in, Figure 8 (d) is a remote sensing filling image schematic diagram of Figure 8 (c),

[0063] Figure 9 A structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] The related art generally uses multi-sensor images to estimate the pixel values of the cloud-polluted area, and accordingly introduces the reference of multi-source data. However, it does not explicitly solve the problem of different resolutions in multi-sensor images. Only the high-resolution image is artificially down-sampled to match the lowest resolution image, and the spatial and spectral information contained in the down-sampled image cannot be fully utilized. Therefore, the performance of filling in missing pixels may be affected.

[0065] In these methods of filling in the missing cloud-polluted pixels, most of them are based on single-sensor images, because the data from a single source can maintain consistency in quality. However, the application and performance of such single-sensor methods are also limited to a certain extent, because the cloud-free images from a single sensor may be far apart in time or lack too much in space. This brings some problems to the spatio-temporal imputation algorithm. From the spatial aspect, some scholars use harmonic algorithms and various spatial interpolation methods to fill in the missing pixels in space, and these algorithms will decrease in accuracy as the number of missing pixels in space increases. From the temporal aspect, most of the current methods are based on recurrent neural networks (RNN), and later scholars use LSTM (Long Short-Term Memory) models and subsequent LSTM derivative models, but these models have the disadvantages of slow speed and memory limitation, and there are compound errors. The self-attention mechanism has been widely used, but it is rarely used in filling in the missing pixels of remote sensing images. The self-attention mechanism is non-autoregressive, which can overcome the disadvantages of slow speed and memory limitation of RNN, can avoid compound errors, and is helpful to achieve better filling quality and faster speed.

[0066] Therefore, the embodiments of the present application propose a method of filling in the missing cloud-polluted pixels, which considers both spatial information and temporal information. First, the spatial interpolation method is used to fill in the pixels in a small area to ensure that the filled-in pixels have high accuracy, and to provide more effective pixels for the subsequent model training. Then, the self-attention mechanism model is used to obtain the missing pixels, and finally the missing pixels of the remote sensing image are filled in. The self-attention mechanism model of the present application is based on the SAITS (Self-Attention-based Imputation for Time Series) model, which is trained by a joint optimization method to learn the missing values from the weighted combination of two diagonal masking self-attention blocks, and can explicitly capture the temporal dependence and feature correlation between time steps. Compared with previous methods, in terms of data sources, the present application is based on single-source remote sensing data, which ensures the consistency of data quality; in terms of methods, it contains both the spatial information of the first law of thermodynamics "similar to similar" and the dependence and feature correlation between time steps of the SAITS model. From these two aspects, the present application can well fill in the missing cloud-polluted pixels of the remote sensing image.

[0067] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0068] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0069] In one exemplary embodiment, as shown in Figure 1 a cloud-polluted remote sensing image missing pixel filling method is provided, including the following steps 101 to 106. Among them:

[0070] Step 101, obtaining a plurality of remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains only one band of image data, and the bands contained by any two remote sensing images to be filled are different.

[0071] In some embodiments, step 101 specifically includes: pre-processing the plurality of remote sensing images to be filled; the pre-processing specifically includes: setting the image range, spatial resolution and spatial reference coordinate system of the plurality of remote sensing images to be filled as a preset image range, a preset spatial resolution and a preset spatial reference coordinate system respectively, and labeling the cloud-polluted pixels in each remote sensing image to be filled as "nan".

[0072] Among them, the range, spatial resolution and spatial reference of each remote sensing image must be consistent. The range is consistent in order to ensure that only cloud-polluted invalid values exist in the remote sensing image, and there are no other types of invalid values. The spatial resolution and spatial reference are consistent in order to have the same number of time series data for each pixel position.

[0073] For example, the test data is land surface temperature data, the range is set to a certain area, the spatial resolution is set to 1 kilometer, and the spatial reference coordinate system is set to WGS 84.

[0074] Although the remote sensing image product data has a problem of being affected by clouds, there is also a quality evaluation band, through which it can be identified which pixel is a cloud-polluted pixel. For example, the pixel value of the cloud-polluted pixel is set to "nan", not like a certain value as usual and then set the value to null. The purpose of such setting is to automatically skip non-numeric data when generating the training set.

[0075] Step 102, obtaining the original pixel matrix of each of the remote sensing images to be filled.

[0076] Step 103, respectively traversing each original pixel matrix, if the i-th pixel in the original pixel matrix is a cloud pollution pixel, judging whether there is a non-cloud pollution pixel in the preset window centered on the i-th pixel, obtaining a first judgment result; if the first judgment result is yes, calculating the mean value of all non-cloud pollution pixels in the preset window, and filling the mean value to the position of the i-th pixel; if the first judgment result is no, marking the i-th pixel as a missing pixel to be filled; until the traversal of all original pixel matrices is completed, obtaining a plurality of missing pixel matrices to be filled of the missing remote sensing images to be filled.

[0077] In another exemplary embodiment of the present application, the preset window size used when acquiring the spatial information filling pixel is 5x5, which can be adjusted according to the specific product. The principle of adjustment is that the values within how many pixel ranges are similar, and the corresponding size is set. The specific calculation formula is as follows:

[0078] The mean value calculation formula of all non-cloud pollution pixels in the preset window is:

[0079]

[0080] Wherein, B is the interpolation image pixel matrix, A is the original pixel matrix, K represents the weight matrix of the first preset size sliding window, i=(0, 1, 2,..., i max represents the row number of the original pixel matrix, i max represents the maximum value of the row number of the original pixel matrix, j=(0, 1, 2,..., j max represents the column number of the original pixel matrix, j max represents the maximum value of the column number of the original pixel matrix, m and n respectively represent the row offset and column offset of the pixel in the first preset size sliding window relative to the current interpolation position (i, j) in the original pixel matrix, and Num is the total number of non-cloud pollution pixels in the first preset size sliding window.

[0081] The spatial information filling pixel has two purposes, one is to use the first law of geography "similar to similar", this method has high filling accuracy for the area with less missing pixels, and can fully utilize the surrounding pixel information to obtain the value of the center pixel. The second is to expand the number of clear sky pixels (pixels not polluted by clouds), reduce the probability of too many continuous missing pixels in time series, otherwise it is not conducive to the training of the subsequent SAITS model. For example Figure 2As shown, the spatial information filling pixels are mainly obtained by interpolation. First, the original image is converted into an interpolated image by spatial interpolation method. Finally, the interpolated image and the original image are merged to generate a result image. The merging method is as follows: if the original image has a clear sky pixel, the clear sky pixel is output; if the original image has no clear sky pixel but the interpolated image has a clear sky pixel, the clear sky pixel of the interpolated image is output; otherwise, a cloud-contaminated pixel is output. Such a merging method is selected to retain the original clear sky pixel and expand the clear sky pixel. Figure 4 (a) and Figure 4 As shown in the comparison chart of the original image and the spatial information filling image, it can be seen that the small area of missing pixels is preliminarily filled, and only the large area of missing pixels remains.

[0082] In step 104, the pixel at the i-th pixel position in the matrix of missing pixels to be filled in the remote sensing image to be filled is extracted and sorted in chronological order to obtain a plurality of one-dimensional original time series arrays.

[0083] In some embodiments, after the remote sensing image is processed into the same range, spatial resolution and spatial reference, and spatial filling is performed, all the pixels corresponding to the bands are first extracted according to the position to form a one-dimensional array of Sx1, as shown in Figure 5 Because the SAITS model is a time series model, both the independent variable and the dependent variable are time series data. In order to generate the dependent variable data, 10% of the numerical data is randomly selected as the dependent variable, and these data are set to "nan". In addition, the mask data (the value of the numerical type is set to 1, and the value of the non-numerical type is set to 0) is generated to distinguish the dependent variable and the independent variable. Finally, a plurality of time series segment data is obtained by window sliding. Because the pixel array is one-dimensional, the window size is set to Txl. Because the surface temperature changes little within a week, the step of the sliding window is set to 7 days, and the length T of the window is set to 4 weeks, i.e., 28 days. These values can be set according to the specific remote sensing product data, and the principle is the same as above.

[0084] In step 105, the plurality of one-dimensional original time series arrays are input into the SAITS model. The SAITS model predicts and fills the missing pixel values at the t-th time according to the non-cloud-contaminated pixels at all times except the t-th time in each one-dimensional original time series array to obtain a plurality of one-dimensional filling time series arrays.

[0085] In another exemplary embodiment of the present application, the above step 105 is replaced by steps 201 to 204:

[0086] In step 201, the plurality of one-dimensional original time series arrays are input into the SAITS model.

[0087] Step 202, the SAITS model predicts the filling value of the missing pixel at the t time according to the non-cloud pollution pixels corresponding to all time points except the t time in the jth one-dimensional original time sequence array, and fills it into the missing pixel at the t time of the jth one-dimensional original time sequence array.

[0088] Step 203, let t=t+1, return to step 202, until t=T1; wherein T1 is the total time of each one-dimensional original time sequence array.

[0089] Step 204, let j=j+1, return to step 202, until j=J, obtain a plurality of one-dimensional filling time sequence arrays; wherein J is the total number of one-dimensional original time sequence arrays.

[0090] The SAITS model specifically comprises a first diagonal mask self-attention block, a second diagonal mask self-attention block, and a weighted combination module.

[0091] The first diagonal mask self-attention block is connected with the second diagonal mask self-attention block and the weighted combination module respectively, and the second diagonal mask self-attention block is connected with the weighted combination module.

[0092] The first diagonal mask self-attention block is used to learn the relationship between the pixel at t time and the pixels before and after t time in the time dimension with the one-dimensional original time sequence array as input, to preliminarily predict the filling value of the missing pixel, and to output a preliminary one-dimensional filling time sequence array.

[0093] The second diagonal mask self-attention block is used to learn the relationship between the pixel at t time and the pixels before and after t time in the time dimension again, to predict the filling value of the missing pixel again, and to output a secondary one-dimensional filling time sequence array.

[0094] The weighted combination module is used to weight and fuse the preliminary one-dimensional filling time sequence array output by the first diagonal mask self-attention block and the secondary one-dimensional filling time sequence array output by the second diagonal mask self-attention block, and to output a final one-dimensional filling time sequence array.

[0095] In another exemplary embodiment of the present application, the SAITS model is composed of a weighted combination of two diagonal mask self-attention (DMSA) blocks, which can explicitly capture the time dependence and feature correlation between the time sequence pixels, as shown in the following formula: Figure 3

[0096]

[0097]

[0098] ​z = {FFN(DiagMaskedMHA(e))} N .

[0099]

[0100] wherein, is the padding value of the first diagonal masked self-attention block, is a one-dimensional original time series array, p is a position encoding, Concat is an array concatenation function, FFN is a feedforward network, DiagMaskedMHA is a multi-head diagonal masked self-attention mechanism, is a preliminary one-dimensional padded time series array, N is to perform N times of nested attention function and feedforward network, W e , b e and W z are a first simulation coefficient, a second simulation coefficient and a third simulation coefficient respectively, e and z are a first intermediate variable and a second intermediate variable respectively, and is an element product, is a mask vector of the one-dimensional original time series array.

[0101] The second diagonal masked self-attention block inputs the preliminary one-dimensional padded time series array generated by the first diagonal masked self-attention block, and the specific implementation formula is as follows:

[0102]

[0103]

[0104] β = {FFN(DiagMaskedMHA(α))} N .

[0105] wherein, is the padding value of the second diagonal masked self-attention block, W α , W β , W γ , b α , b β and b γ are a fourth simulation coefficient, a fifth simulation coefficient, a sixth simulation coefficient, a seventh simulation coefficient, an eighth simulation coefficient, a ninth simulation coefficient and a tenth simulation coefficient respectively, and α and β are a third intermediate variable and a fourth intermediate variable respectively.

[0106] The final one-dimensional padded time series array is obtained by weighted combination, and the specific implementation formula is as follows:

[0107]

[0108]

[0109] wherein, η is a combined weight generated by taking the average attention weight and the missing mask as reference, and is a weight result based on η.

[0110] In another exemplary embodiment of the present application, the surface temperature image of the regional range set in 2020 is taken as test data, and the method of filling the missing pixels combined with the temporal information based on the spatial information filling proposed in the present application better fills the missing pixels polluted by clouds. In order to analyze the accuracy of the final filled image, the accuracy corresponding to each pixel position is counted, as shown in Figure 7 (a) and Figure 7 (b). The accuracy index used in the present application is the root mean square error and the absolute average error, wherein K is the temperature unit. From the accuracy result, it can be seen that the root mean square error is mainly concentrated at about 4K, and the absolute average error is mainly concentrated at about 2K. Because the error of satellite sensor observing the surface temperature is about 2K, if the filling accuracy is lower than 2K, it is not too meaningful. Therefore, the method of the present application has high accuracy and can completely fill the missing pixels of the remote sensing image polluted by clouds.

[0111] Step 106, reconstructing the remote sensing filled image according to the plurality of one-dimensional filling time sequence arrays.

[0112] In an exemplary embodiment, after the SAITS model generates the pixel filling to the position corresponding to the image, some noise points appear in the result image, which are actually outliers. In order to solve this problem, the present application uses median filtering to eliminate these abnormal pixels, so that the result image becomes smoother. The specific implementation formula is as follows:

[0113] h(i,j)=med{f(i-k,j-l)},k,l∈W;

[0114] wherein h is the remote sensing filled image after eliminating noise, f(i-k,j-l) is the pixel at position (i-k,j-l) of the remote sensing filled image, W={-1,0,1}, k and l are the row offset and column offset relative to the current position (i,j), respectively.

[0115] As shown in Figure 8 (a) to Figure 8 (d), in the temperature range of 24K-30K, by comparing the original cloud-polluted image with the final filled result image, it can be seen that the method proposed in the present application well fills the missing pixels, not only has high accuracy, but also ensures the spatial continuity.

[0116] The application also provides an application scenario of the method for filling missing pixels of a cloud-polluted remote sensing image. Specifically, the method for filling missing pixels provided in this embodiment can be applied in a remote sensing image processing scenario. The remote sensing image processing scenario generally includes image acquisition, preprocessing, feature extraction, and application analysis, etc. The remote sensing image enters the preprocessing link from the acquisition link, and after a series of processing steps, a clear image is obtained and enters the downstream feature extraction and application analysis link. The method for filling missing pixels provided in this embodiment belongs to the image repair step in the preprocessing link. Specifically, in the preprocessing process of the remote sensing image, a plurality of remote sensing images to be filled can be used to fill the missing pixels by judging the cloud-polluted pixels and the surrounding pixels, combining the time sequence prediction capability of the SAITS model, so as to obtain a clear remote sensing image after filling.

[0117] Based on the same inventive concept, the application also provides a device for filling missing pixels of a cloud-polluted remote sensing image. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more device embodiments for filling missing pixels of a cloud-polluted remote sensing image provided below can refer to the limitations of the method for filling missing pixels of a cloud-polluted remote sensing image provided above, and will not be repeated here.

[0118] In one exemplary embodiment, as shown in Figure 6 A device for filling missing pixels of a cloud-polluted remote sensing image is provided, including:

[0119] The remote sensing image acquisition module 301 is configured to acquire a plurality of remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained by any two remote sensing images to be filled are different.

[0120] The pixel matrix acquisition module 302 is configured to acquire the original pixel matrix of each remote sensing image to be filled respectively.

[0121] The spatial filling module 303 is configured to traverse each original pixel matrix respectively, if the i th pixel in the original pixel matrix is a cloud-polluted pixel, determine whether there is a non-cloud-polluted pixel in the preset window centered on the i th pixel to obtain a first determination result; if the first determination result is yes, calculate the mean value of all non-cloud-polluted pixels in the preset window, and fill the mean value to the position of the i th pixel; if the first determination result is no, mark the i th pixel as a missing pixel to be filled; until the traversal of all original pixel matrices is completed, the missing pixel matrix to be filled of the plurality of remote sensing images to be filled is obtained.

[0122] The one-dimensional array acquisition module 304 is configured to extract a pixel at an i-th pixel position in a to-be-filled missing pixel matrix of all to-be-filled remote sensing images, and sort the pixel in a time sequence to obtain a plurality of one-dimensional original time sequence arrays.

[0123] The time filling module 305 is configured to input the plurality of one-dimensional original time sequence arrays into a SAITS model, and use non-cloud-polluted pixels corresponding to all time points except a t-th time point in each one-dimensional original time sequence array to predict and fill a missing pixel value at the t-th time point, to obtain a plurality of one-dimensional filled time sequence arrays.

[0124] The filled image acquisition module 306 is configured to reconstruct a remote sensing filled image according to the plurality of one-dimensional filled time sequence arrays.

[0125] In an example embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram of the computer device can be as shown in Figure 9 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store cloud-polluted remote sensing image data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a missing pixel filling method for cloud-polluted remote sensing images.

[0126] Those skilled in the art can understand that Figure 9 The structure shown in the above figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an example embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in each of the above method embodiments.

[0127] In an exemplary embodiment, a computer readable storage medium storing a computer program is provided, the computer program, when executed by a processor, implements the steps of any of the above method embodiments.

[0128] In an exemplary embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any of the above method embodiments.

[0129] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0130] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0131] The database involved in each of the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, and the like, without being limited thereto. The processor involved in each of the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, and the like, without being limited thereto.

[0132] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but it should be considered that any combination of the technical features is within the scope of the present disclosure, as long as there is no contradiction.

[0133] The principles and implementation manners of the present application are described by using specific examples herein, and the above embodiments are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A method for filling missing pixels in cloud-contaminated remote sensing images, characterized in that: The method for filling missing pixels of cloud-contaminated remote sensing images includes: Acquire multiple remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained in any two remote sensing images to be filled are different; respectively obtaining the original pixel matrix of each remote sensing image to be filled; Traverse each original pixel matrix separately. If the i-th pixel in the original pixel matrix is ​​a cloud-contaminated pixel, determine whether there is a non-cloud-contaminated pixel in the preset window centered on the i-th pixel, and obtain a first judgment result. If the first judgment result is yes, calculate the mean of all non-cloud-contaminated pixels in the preset window and fill the position of the i-th pixel with the mean. If the first judgment result is no, mark the i-th pixel as a missing pixel to be filled. This continues until all original pixel matrices are traversed, and multiple missing pixel matrices to be filled of remote sensing images to be filled are obtained. Extract the pixel at the i-th pixel position in the matrix of missing pixels to be filled in all remote sensing images, and sort them in chronological order to obtain multiple one-dimensional original time series arrays; Multiple one-dimensional original time series arrays are input into the SAITS model. The SAITS model predicts and fills the missing pixel values ​​at time t based on the non-cloud-polluted pixels corresponding to all moments except time t in each one-dimensional original time series array, and obtains multiple one-dimensional filled time series arrays. A remote sensing filled image is reconstructed based on the multiple one-dimensional filled time series arrays.

2. The method for filling missing pixels in cloud-contaminated remote sensing images according to claim 1, characterized in that: The original pixel matrix of each remote sensing image to be filled is obtained respectively, and the following steps are also included before: Preprocessing is performed on multiple remote sensing images to be filled; the preprocessing specifically includes: setting the image range, spatial resolution, and spatial reference coordinate system of the multiple remote sensing images to be filled to a preset image range, preset spatial resolution, and preset spatial reference coordinate system, respectively, and marking cloud-contaminated pixels in each remote sensing image to be filled as "nan".

3. The method for filling missing pixels in cloud-contaminated remote sensing images according to claim 1, wherein: The calculation formula for the mean value of all non-cloud-polluted pixels in the preset window is: Wherein, B is the interpolated image pixel matrix, A is the original pixel matrix, K represents the weight matrix of the sliding window of the first preset size, i=(0, 1, 2, ..., i max ) represents the row number of the original pixel matrix, j = (0, 1, 2, ..., j max ) represents the column number of the original pixel matrix, m and n respectively represent the row offset and column offset of the pixel in the sliding window of the first preset size relative to the current interpolation position (i, j) in the original pixel matrix, and Num is the total number of non-cloud-contaminated pixels in the sliding window of the first preset size.

4. The method for filling missing pixels in cloud-contaminated remote sensing images according to claim 1, wherein: The SAITS model specifically includes: a first diagonal shielded self-attention block, a second diagonal shielded self-attention block and a weighted combination module; The first diagonal masked self-attention block is connected to the second diagonal masked self-attention block and the weighted combination module respectively, and the second diagonal masked self-attention block is connected to the weighted combination module; The first diagonal masked self-attention block is used to take the one-dimensional original time series array as input, learn the relationship between the pixel at time t and the pixels before and after time t in the time dimension, preliminarily predict the filling value of the missing pixel, and output a preliminary one-dimensional filled time series array; The second diagonal masked self-attention block is used to re-learn the relationship between the pixel at time t and the pixels before and after time t in the time dimension, re-predict the filling value of the missing pixel, and output the second-filled one-dimensional filled time series array; The weighted combination module is used to weightedly fuse the preliminary one-dimensional padded time series array output by the first diagonally masked self-attention block and the secondary padded one-dimensional padded time series array output by the second diagonally masked self-attention block, and output the final one-dimensional padded time series array.

5. The method for filling missing pixels in cloud-contaminated remote sensing images according to claim 4, characterized in that: The calculation formula of the first diagonal masked self-attention block is as follows: z={FFN(DiagMaskedMHA(e))} N ; in, is the padding value of the first diagonal masked self-attention block, is a one-dimensional original time series array, p is the position code, Concat is the array concatenation function, FFN is the feedforward network, DiagMaskedMHA is the multi-head diagonal mask attention mechanism, For the initial one-dimensional padded time series array, {} N To perform N nested attention functions and feedforward networks, W e 、b e and W z are the first simulation coefficient, the second simulation coefficient and the third simulation coefficient respectively, e and z are the first intermediate variable and the second intermediate variable respectively, ⊙ is the element product, is the mask vector of the one-dimensional original time series array; The specific implementation formula of the second diagonal masked self-attention block is as follows: β={FFN(DiagMaskedMHA(α))} N : in, is the padding value of the second diagonal masked self-attention block, W α 、W β 、W γ 、b α 、b β and b γ are the fourth simulation coefficient, the fifth simulation coefficient, the sixth simulation coefficient, the seventh simulation coefficient, the eighth simulation coefficient, the ninth simulation coefficient and the tenth simulation coefficient respectively, and α and β are the third intermediate variable and the fourth intermediate variable respectively; The specific implementation formula of the weighted combination module is as follows: in, is the missing pixel filling value, η is the combined weight generated with the average attention weight and the missing mask as reference, is the weight result based on η.

6. The method for filling missing pixels in cloud-contaminated remote sensing images according to claim 1, wherein: Reconstructing a remote sensing filled image according to the plurality of one-dimensional filled time series arrays, and then further comprising: The noise of remote sensing infill images is eliminated by median filtering method; The formula of the median filtering method is: h(i,j)=med{f(ik,jl)},k,l∈W; Where h is the remote sensing infill image after noise removal, f(ik, jl) is the pixel of the remote sensing infill image at position (ik, jl), W = {-1, 0, 1}, k and l are the row offset and column offset relative to the current position (i, j), respectively.

7. A device for filling missing pixels in cloud-contaminated remote sensing images, characterized in that: The missing pixel filling device for cloud-contaminated remote sensing images comprises: A remote sensing image acquisition module is used to acquire multiple remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained in any two remote sensing images to be filled are different; A pixel matrix acquisition module is used to respectively acquire the original pixel matrix of each remote sensing image to be filled; The spatial filling module is used to traverse each original pixel matrix separately. If the i-th pixel in the original pixel matrix is ​​a cloud-contaminated pixel, it is determined whether there is a non-cloud-contaminated pixel in the preset window centered on the i-th pixel to obtain a first judgment result; if the first judgment result is yes, the mean of all non-cloud-contaminated pixels in the preset window is calculated and the mean is filled to the position of the i-th pixel; if the first judgment result is no, the i-th pixel is marked as a missing pixel to be filled; until all original pixel matrices are traversed, multiple missing pixel matrices to be filled of remote sensing images to be filled are obtained; A one-dimensional array acquisition module is used to extract the pixel at the i-th pixel position in the matrix of missing pixels to be filled in all remote sensing images to be filled, and sort them in chronological order to obtain multiple one-dimensional original time series arrays; The time filling module is used to input multiple one-dimensional original time series arrays into the SAITS model. The SAITS model predicts and fills the missing pixel values ​​at time t based on the non-cloud pollution pixels corresponding to all moments except time t in each one-dimensional original time series array, and obtains multiple one-dimensional filled time series arrays. The padded image acquisition module is used to reconstruct and obtain a remote sensing padded image based on the multiple one-dimensional padded time series arrays.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for filling missing pixels in cloud-contaminated remote sensing images according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for filling missing pixels in a cloud-contaminated remote sensing image according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for filling missing pixels in a cloud-contaminated remote sensing image according to any one of claims 1 to 6 is implemented.

Images