Method, device and equipment for filling missing pixels of cloud-polluted remote sensing image, medium and product

Abstract

The invention discloses a missing pixel filling method, device, equipment, medium and product for a cloud-polluted remote sensing image, and relates to the technical field of remote sensing image processing.The method comprises the steps that multiple to-be-filled remote sensing images are obtained, original pixel matrixes of the to-be-filled remote sensing images are obtained, part of missing pixels are preliminarily filled through space information, and the original pixel matrixes of the to-be-filled remote sensing images are obtained; missing pixels to be filled are marked, and a matrix of the missing pixels to be filled is formed; and respectively extracting pixels at each pixel position in all the missing pixel matrixes to be filled, constructing a plurality of one-dimensional original time sequence arrays, inputting the one-dimensional original time sequence arrays into the SAITS model to predict and fill the missing pixels, and generating a one-dimensional filling time sequence array. And reconstructing a remote sensing filling image according to the plurality of one-dimensional filling time sequence arrays. According to the method, the spatial neighborhood information of the remote sensing image is considered, and the time sequence information in the time sequence is fully utilized, so that the data quality of the cloud-polluted remote sensing image is effectively improved, and the remote sensing image with space-time continuity and high precision is generated.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method, device, equipment, medium and product for filling missing pixels of remote sensing images contaminated by clouds. Background Art

[0002] With the rapid development of satellite sensors and remote sensing inversion technology, remote sensing data has become an indispensable information foundation for many research fields. Its large-scale, multi-temporal and all-weather characteristics provide a rich source of information for agricultural monitoring, urban planning, earth science and climate change research. However, cloud pollution has always been an important factor restricting the application of remote sensing data. More than half of optical remote sensing images are affected and data availability is greatly limited. In recent years, the social and economic demand for remote sensing images has increased dramatically, and the cloud pollution problem has become increasingly prominent, and an effective solution is urgently needed.

[0003] Although some techniques and benchmark datasets for filling missing pixels have been developed, the inconsistent data quality caused by multi-source data fusion and the limitations of single-sensor methods in spatiotemporal filling still limit the improvement of filling performance. In particular, the differences in spatial, temporal and spectral resolution of multi-source data and the insufficiency of single-sensor images in spatiotemporal filling limit the filling effect and bring many challenges to the filling work. Summary of the invention

[0004] The purpose of this application is to provide a method, device, equipment, medium and product for filling missing pixels of remote sensing images contaminated by clouds, which can fill the missing pixels of remote sensing images contaminated by clouds, thereby generating high-precision images with spatiotemporal continuity.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for filling missing pixels of a remote sensing image contaminated by clouds, comprising:

[0007] Acquire multiple remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained in any two remote sensing images to be filled are different;

[0008] Respectively obtain the original pixel matrix of each remote sensing image to be filled;

[0009] Traverse each original pixel matrix respectively. If the i-th pixel in the original pixel matrix is ​​a cloud-polluted pixel, determine whether there is a non-cloud-polluted pixel in the preset window centered on the i-th pixel, and obtain a first judgment result; if the first judgment result is yes, calculate the mean of all non-cloud-polluted pixels in the preset window, and fill the mean to the position of the i-th pixel; if the first judgment result is no, mark the i-th pixel as a missing pixel to be filled; until all original pixel matrices are traversed, multiple missing pixel matrices to be filled of remote sensing images to be filled are obtained;

[0010] Extract the pixel at the i-th pixel position in the matrix of missing pixels to be filled in all remote sensing images to be filled, and sort them in chronological order to obtain multiple one-dimensional original time series arrays;

[0011] Input multiple one-dimensional original time series arrays into the SAITS (Self-Attention-based Imputation for Time Series) model. The SAITS model predicts and fills in the missing pixel values ​​at the tth moment according to the non-cloud-polluted pixels corresponding to all moments except the tth moment in each one-dimensional original time series array, and obtains multiple one-dimensional filled time series arrays.

[0012] A remote sensing filled image is reconstructed based on the multiple one-dimensional filled time series arrays.

[0013] Optionally, respectively obtaining the original pixel matrix of each remote sensing image to be filled, before that also includes:

[0014] Preprocessing is performed on multiple remote sensing images to be filled; the preprocessing specifically includes: setting the image range, spatial resolution and spatial reference coordinate system of the multiple remote sensing images to be filled to a preset image range, preset spatial resolution and preset spatial reference coordinate system, respectively, and marking the cloud-contaminated pixels in each remote sensing image to be filled as "nan".

[0015] Optionally, the mean value calculation formula of all non-cloud-contaminated pixels in the preset window is:

[0016]

[0017] Wherein, B is the interpolated image pixel matrix, A is the original pixel matrix, K represents the weight matrix of the sliding window of the first preset size, i=(0, 1, 2, ..., i max ) represents the row number of the original pixel matrix, j = (0, 1, 2, ..., j max) represents the column number of the original pixel matrix, m and n represent the row offset and column offset of the pixel in the sliding window of the first preset size relative to the current interpolation position (i, j) in the original pixel matrix, and Num is the total number of non-cloud-contaminated pixels in the sliding window of the first preset size.

[0018] Optionally, the SAITS model specifically includes: a first diagonal shielded self-attention block, a second diagonal shielded self-attention block and a weighted combination module;

[0019] The first diagonal masked self-attention block is respectively connected to the second diagonal masked self-attention block and the weighted combination module, and the second diagonal masked self-attention block is connected to the weighted combination module;

[0020] The first diagonal masked self-attention block is used to take the one-dimensional original time series array as input, learn the relationship between the pixel at time t and the pixels before and after time t in the time dimension, preliminarily predict the filling value of the missing pixel, and output a preliminary one-dimensional filled time series array;

[0021] The second diagonal masked self-attention block is used to learn again the relationship between the pixel at time t and the pixels before and after time t in the time dimension, predict the filling value of the missing pixel again, and output the one-dimensional filling time series array of the second filling;

[0022] The weighted combination module is used to weightedly fuse the preliminary one-dimensional padded time series array output by the first diagonal masked self-attention block and the secondary padded one-dimensional padded time series array output by the second diagonal masked self-attention block, and output the final one-dimensional padded time series array.

[0023] Optionally, the calculation formula of the first diagonal masked self-attention block is as follows:

[0024]

[0025]

[0026] z={FFN(DiagMaskedMHA(e))} N ;

[0027]

[0028] in, is the padding value of the first diagonal masked self-attention block, is a one-dimensional original time series array, p is the position code, Concat is the array concatenation function, FFN is the feedforward network, DiagMaskedMHA is the multi-head diagonal mask attention mechanism, is the initial one-dimensional padded time series array, {}N is the execution of N times of nested attention function and feedforward network, We , b e and W z are the first simulation coefficient, the second simulation coefficient and the third simulation coefficient respectively, e and z are the first intermediate variable and the second intermediate variable respectively, ⊙ is the element product, Mask vector of the original 1D time series array;

[0029] The specific implementation formula of the second diagonal masked self-attention block is as follows:

[0030]

[0031] β={FFN(DiagMaskedMHA(α))} N ;

[0032] in, is the padding value of the second diagonal masked self-attention block, W α , W β , W γ , b α , b β and b γ are the fourth simulation coefficient, the fifth simulation coefficient, the sixth simulation coefficient, the seventh simulation coefficient, the eighth simulation coefficient, the ninth simulation coefficient and the tenth simulation coefficient respectively, and α and β are the third intermediate variable and the fourth intermediate variable respectively;

[0033] The specific implementation formula of the weighted combination module is as follows:

[0034]

[0035]

[0036] in, is the missing pixel filling value, η is the combined weight generated with the average attention weight and the missing mask as reference, is the weight result based on η.

[0037] Optionally, reconstructing a remote sensing filled image according to the plurality of one-dimensional filled time series arrays, and then further comprising:

[0038] The noise of remote sensing infill images is eliminated by median filtering method;

[0039] The formula of the median filtering method is:

[0040] h(i,j)=med{f(ik,jl)},k,l∈W;

[0041] Wherein, h is the remote sensing infill image after noise elimination, f(ik, jl) is the pixel of the remote sensing infill image at the position (ik, jl), W = {-1, 0, 1}, k and l are the row offset and column offset relative to the current position (i, j), respectively.

[0042] In a second aspect, the present application provides a device for filling missing pixels of a remote sensing image contaminated by clouds, comprising:

[0043] The remote sensing image acquisition module is used to acquire multiple remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained in any two remote sensing images to be filled are different;

[0044] A pixel matrix acquisition module is used to respectively acquire the original pixel matrix of each remote sensing image to be filled;

[0045] The spatial filling module is used to traverse each original pixel matrix respectively. If the i-th pixel in the original pixel matrix is ​​a cloud-polluted pixel, it is judged whether there is a non-cloud-polluted pixel in the preset window centered on the i-th pixel to obtain a first judgment result; if the first judgment result is yes, the mean of all non-cloud-polluted pixels in the preset window is calculated, and the mean is filled to the position of the i-th pixel; if the first judgment result is no, the i-th pixel is marked as a missing pixel to be filled; until all original pixel matrices are traversed, multiple missing pixel matrices to be filled of remote sensing images to be filled are obtained;

[0046] A one-dimensional array acquisition module is used to extract the pixel at the i-th pixel position in the missing pixel matrix to be filled in all remote sensing images to be filled, and sort them in chronological order to obtain multiple one-dimensional original time series arrays;

[0047] The time filling module is used to input multiple one-dimensional original time series arrays into the SAITS model. The SAITS model predicts and fills the missing pixel values ​​at the tth moment according to the non-cloud pollution pixels corresponding to all moments except the tth moment in each one-dimensional original time series array, and obtains multiple one-dimensional filled time series arrays.

[0048] The padded image acquisition module is used to reconstruct a remote sensing padded image based on the multiple one-dimensional padded time series arrays.

[0049] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described methods for filling missing pixels of remote sensing images contaminated by clouds.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for filling missing pixels of remote sensing images contaminated by clouds.

[0051] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for filling missing pixels of cloud-contaminated remote sensing images.

[0052] According to the specific embodiments provided in this application, this application has the following technical effects:

[0053] The present application provides a method, device, equipment, medium and product for filling missing pixels of remote sensing images polluted by clouds. By obtaining multiple remote sensing images to be filled, and respectively obtaining the original pixel matrix of each image, the cloud polluted pixels are preliminarily judged and filled with local mean values, and multiple missing pixel matrices to be filled are obtained, which solves the problem that the traditional method does not fully utilize the neighborhood information when processing cloud polluted remote sensing images, realizes the preliminary effective filling of local cloud polluted pixels, and improves the local integrity of the image; the pixels of each pixel position in all the missing pixel matrices to be filled are respectively extracted, and multiple one-dimensional original time series arrays are constructed, and the multiple one-dimensional original time series arrays are input into the SAITS model to predict the filling value of the missing pixels, which solves the problem that a single image is difficult to accurately fill large-area missing pixels under complex cloud pollution conditions, realizes high-precision filling based on time series analysis and prediction, and improves the quality and availability of the overall image. In the filling method, the present application not only utilizes the advantages of the first law of geoscience "similar to the near", but also uses the characteristics of the SAITS model that can clearly capture the time dependence and feature correlation between time series pixels, so as to obtain a high-precision filled image. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 A schematic diagram of a process for filling missing pixels of a remote sensing image contaminated by clouds provided in one embodiment of the present application;

[0056] Figure 2 for Figure 1 A detailed flowchart of the steps of filling missing pixels based on spatial information;

[0057] Figure 3 for Figure 1 A detailed flowchart of the steps of filling missing pixels based on time series information;

[0058] Figure 4 A comparison diagram of a remote sensing image to be filled and a remote sensing image after space filling provided in an embodiment of the present application; wherein, Figure 4 (a) is a schematic diagram of the remote sensing image to be filled. Figure 4 (b) is a schematic diagram of the remote sensing image after space filling;

[0059] Figure 5 A diagram of remote sensing image data processing after space filling provided by an embodiment of the present application;

[0060] Figure 6 A schematic diagram of functional modules of a device for filling missing pixels of a remote sensing image contaminated by clouds provided in one embodiment of the present application;

[0061] Figure 7 A pixel filling accuracy diagram of a missing pixel filling method for a remote sensing image contaminated by clouds provided in an embodiment of the present application; wherein, Figure 7 (a) is the root mean square error diagram, Figure 7 (b) is the absolute mean error graph;

[0062] Figure 8 A comparison diagram of two remote sensing images to be filled and a remote sensing filled image provided in an embodiment of the present application; wherein, Figure 8 (a) is a schematic diagram of a remote sensing image to be filled. Figure 8 (b) Figure 8 (a) Schematic diagram of remote sensing infill image, Figure 8 (c) is another schematic diagram of a remote sensing image to be filled. Figure 8 (d) Figure 8 (c) Schematic diagram of remote sensing infill image;

[0063] Fig. 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0064] Related technologies usually use multi-sensor images to estimate the pixel values ​​of cloud-contaminated areas, and accordingly introduce a benchmark for multi-source data. However, the problem of different resolutions in multi-sensor images is not explicitly addressed. The high-resolution images are simply artificially downsampled to match the lowest-resolution images, and the spatial and spectral information contained in the downsampled images cannot be fully utilized. Therefore, the performance of filling missing pixels may be affected.

[0065] Among these methods for filling in missing pixels due to cloud pollution, most are based on images from a single sensor, because the quality of data from a single source can be consistent. However, the application and performance of this single-sensor method will also be limited to a certain extent, because cloud-free images from a single sensor may be far apart in time or too missing in space. This brings some problems to the spatiotemporal filling algorithm. From a spatial perspective, some scholars will use harmonic algorithms and various spatial interpolation methods to fill in the missing pixels in space. The accuracy of these algorithms will decrease as the number of missing pixels in space increases. From a temporal perspective, most of them are currently based on recurrent neural networks (RNN). Later, some scholars used LSTM (Long Short-Term Memory) models and subsequent LSTM derivative models, but these models have the disadvantages of slow speed and memory limitations, and there are compound errors. The self-attention mechanism has been widely used, but it is rarely used to fill in missing pixels in remote sensing images. The self-attention mechanism is non-autoregressive and can overcome the shortcomings of slow speed and memory limitation of RNN, avoid compound errors, and help achieve better quality of filled pixels and faster speed.

[0066] In view of this, the embodiment of the present application proposes a method for filling missing pixels polluted by clouds, which considers both spatial information and temporal information. First, the pixels in a small area are filled by spatial interpolation to ensure that the filled pixels have high precision, and more effective pixels are provided for the training of subsequent models. Then, the self-attention mechanism model is used to obtain the missing pixels, and finally fill the missing pixels of the remote sensing image. The self-attention mechanism model of the present application is based on the self-attention time series interpolation model (Self-Attention-based Imputation for Time Series, SAITS), which is trained by a joint optimization method, learns missing values ​​from a weighted combination of two diagonally shielded self-attention blocks, and can explicitly capture the time dependency and feature correlation between time steps. Compared with previous methods, in terms of data source, the present application is based on remote sensing data from a single source, which ensures the consistency of data quality; in terms of methods, it not only contains the spatial information of "similar to similar" of the first law of geoscience, but also takes into account the dependency and feature correlation between time steps of the SAITS model. Considering these two aspects, the present application can well fill in the missing pixels of remote sensing images polluted by clouds.

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0069] In an exemplary embodiment, Figure 1 As shown, a method for filling missing pixels of a remote sensing image contaminated by clouds is provided, comprising the following steps 101 to 106. Among them:

[0070] Step 101, obtaining a plurality of remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained in any two remote sensing images to be filled are different.

[0071] In some embodiments, step 101 specifically includes: preprocessing multiple remote sensing images to be filled; the preprocessing specifically includes: setting the image range, spatial resolution and spatial reference coordinate system of the multiple remote sensing images to be filled to a preset image range, preset spatial resolution and preset spatial reference coordinate system, respectively, and marking the cloud-contaminated pixels in each remote sensing image to be filled as "nan".

[0072] The range, spatial resolution and spatial reference of each remote sensing image must be consistent. The range consistency is to ensure that there are only invalid values ​​due to cloud contamination in the remote sensing image, and no other types of invalid values. The spatial resolution and spatial reference are consistent so that each pixel location has the same amount of time series data.

[0073] Exemplarily, the test data is surface temperature data, the range is a set area, the spatial resolution is set to 1 km, and the spatial reference coordinate system is set to WGS 84.

[0074] Although remote sensing image product data is affected by clouds, it also has a quality assessment band, which can be used to identify which pixels are cloud-contaminated pixels. For example, the pixel value of cloud-contaminated pixels is set to "nan", not to a certain value and then set the value to null as usual. The purpose of this setting is to automatically skip non-numeric data when generating training sets.

[0075] Step 102, respectively obtain the original pixel matrix of each remote sensing image to be filled.

[0076] Step 103, traverse each original pixel matrix respectively, if the i-th pixel in the original pixel matrix is ​​a cloud-polluted pixel, determine whether there is a non-cloud-polluted pixel in the preset window centered on the i-th pixel, and obtain a first determination result; if the first determination result is yes, calculate the mean of all non-cloud-polluted pixels in the preset window, and fill the mean to the position of the i-th pixel; if the first determination result is no, mark the i-th pixel as a missing pixel to be filled; until all original pixel matrices are traversed, multiple missing pixel matrices to be filled of remote sensing images to be filled are obtained.

[0077] In another exemplary embodiment of the present application, the preset window size used when obtaining spatial information to fill in pixels is 5×5. This window size can be adjusted according to the specific product. The principle of adjustment is to set the corresponding size according to the number of similar values ​​within the pixel range. The specific calculation formula is as follows:

[0078] The mean calculation formula for all non-cloud-contaminated pixels in the preset window is:

[0079]

[0080] Wherein, B is the interpolated image pixel matrix, A is the original pixel matrix, K represents the weight matrix of the sliding window of the first preset size, i=(0, 1, 2, ..., i max ) represents the row number of the original pixel matrix, i max Indicates the maximum row number of the original pixel matrix, j = (0, 1, 2, ..., j max ) represents the column number of the original pixel matrix, j max represents the maximum column number of the original pixel matrix, m and n represent the row offset and column offset of the pixel in the sliding window of the first preset size relative to the current interpolation position (i, j) in the original pixel matrix, and Num is the total number of non-cloud-contaminated pixels in the sliding window of the first preset size.

[0081] There are two purposes for filling pixels with spatial information. One is to use the first law of geography, "similar to similar". This method has a higher filling accuracy for areas with few missing pixels, and can make full use of the surrounding pixel information to obtain the value of the central pixel. The second is to expand the number of clear sky pixels (pixels not contaminated by clouds) and reduce the probability of too many consecutive missing pixels in the time series, otherwise it is not conducive to the subsequent training of the SAITS model. Figure 2As shown in the figure, the spatial information filling pixel is mainly obtained by interpolation. First, the original image is converted into an interpolated image through the spatial interpolation method, and finally the interpolated image is merged with the original image to generate the result image. The merging method is: if there is a clear sky pixel in the original image, the clear sky pixel is output; if there is no clear sky pixel in the original image but there is a clear sky pixel in the interpolated image, the clear sky pixel of the interpolated image is output, otherwise the cloud pollution pixel is output. This merging method is selected to retain the original clear sky pixel and expand the clear sky pixel. Figure 4 (a) and Figure 4 As shown in (b), by comparing the original image and the image filled with spatial information, it can be seen that the small areas of missing pixels have been preliminarily filled, leaving only large areas of missing pixels.

[0082] Step 104 , extracting the pixel at the i-th pixel position in the matrix of missing pixels to be filled of all remote sensing images to be filled, and sorting them in chronological order to obtain a plurality of one-dimensional original time series arrays.

[0083] In some embodiments, the remote sensing images are processed into the same range, spatial resolution and spatial reference, and after spatial filling, all pixels of the corresponding band are first extracted according to the position to form an S×1 one-dimensional array, such as Figure 5 As shown. Because the SAITS model is a time series model, the independent variable and the dependent variable are the same time series data. In order to generate the dependent variable data, this application uses 10% as the threshold, that is, randomly selects 10% of the numerical data as the dependent variable, and these data are set to "nan". In addition, mask data is generated (the value of the numerical type is set to 1, and the value of the non-numeric type is set to 0) to distinguish the dependent variable from the independent variable. Finally, multiple time series segment data are intercepted by window sliding. Because the pixel array is one-dimensional, the window size is set to T×1. Because the surface temperature does not change much within a week, the step size of the sliding window is set to 7 days, and the length of the window T is set to 4 weeks, which is 28 days. These values ​​can be set according to the specific remote sensing product data, and the principles are the same as above.

[0084] Step 105, input multiple one-dimensional original time series arrays into the SAITS model, and the SAITS model predicts and fills the missing pixel values ​​at the tth moment based on the non-cloud pollution pixels corresponding to all moments except the tth moment in each one-dimensional original time series array, to obtain multiple one-dimensional filled time series arrays.

[0085] In another exemplary embodiment of the present application, the above step 105 is replaced by the following steps 201 to 204:

[0086] Step 201: input multiple one-dimensional original time series arrays into the SAITS model.

[0087] Step 202, the SAITS model predicts the filling value of the missing pixel at the tth moment based on the non-cloud-contaminated pixels corresponding to all moments except the tth moment in the jth one-dimensional original time series array, and fills the missing pixel at the tth moment in the jth one-dimensional original time series array.

[0088] Step 203, let t=t+1, and return to step 202 until t=T1; wherein T1 is the total number of moments in each one-dimensional original time series array.

[0089] Step 204, let j=j+1, return to step 202, until j=J, to obtain multiple one-dimensional padded time series arrays; wherein J is the total number of one-dimensional original time series arrays.

[0090] Among them, the SAITS model specifically includes: a first diagonal shielded self-attention block, a second diagonal shielded self-attention block and a weighted combination module.

[0091] The first diagonal masked self-attention block is connected to the second diagonal masked self-attention block and the weighted combination module respectively, and the second diagonal masked self-attention block is connected to the weighted combination module.

[0092] The first diagonal masked self-attention block is used to take the one-dimensional original time series array as input, learn the relationship between the pixel at time t and the pixels before and after time t in the time dimension, preliminarily predict the filling value of the missing pixels, and output a preliminary one-dimensional filled time series array.

[0093] The second diagonal masked self-attention block is used to re-learn the relationship between the pixel at time t and the pixels before and after time t in the time dimension, re-predict the filling value of the missing pixel, and output the one-dimensional filling time series array of the second filling.

[0094] The weighted combination module is used to weightedly fuse the preliminary one-dimensional padded time series array output by the first diagonal masked self-attention block and the secondary padded one-dimensional padded time series array output by the second diagonal masked self-attention block, and output the final one-dimensional padded time series array.

[0095] In another exemplary embodiment of the present application, the SAITS model consists of a weighted combination of two diagonal masked self-attention (DMSA) blocks, which can clearly capture the temporal dependency and feature correlation between temporal pixels, such as Figure 3 The calculation formula of the first diagonal masked self-attention block is as follows:

[0096]

[0097]

[0098] z={FFN(DiagMaskedMHA(e))} N .

[0099]

[0100] in, is the padding value of the first diagonal masked self-attention block, is a one-dimensional original time series array, p is the position code, Concat is the array concatenation function, FFN is the feedforward network, DiagMaskedMHA is the multi-head diagonal mask attention mechanism, For the initial one-dimensional padded time series array, {} N To perform N times of nested attention functions and feed-forward networks, W e , b e and W z are the first simulation coefficient, the second simulation coefficient and the third simulation coefficient respectively, e and z are the first intermediate variable and the second intermediate variable respectively, ⊙ is the element product, The mask vector for the original one-dimensional time series array.

[0101] The second diagonal masked self-attention block will input the preliminary one-dimensional padded time series array generated by the first diagonal masked self-attention block. The specific implementation formula is as follows:

[0102]

[0103]

[0104] β={FFN(DiagMaskedMHA(α))} N .

[0105] in, is the padding value of the second diagonal masked self-attention block, W α , W β , W γ , b α , b β and b γ are the fourth simulation coefficient, the fifth simulation coefficient, the sixth simulation coefficient, the seventh simulation coefficient, the eighth simulation coefficient, the ninth simulation coefficient and the tenth simulation coefficient respectively, and α and β are the third intermediate variable and the fourth intermediate variable respectively.

[0106] The final one-dimensional filled time series array is obtained through weighted combination. The specific implementation formula is as follows:

[0107]

[0108]

[0109] in, is the missing pixel filling value, η is the combined weight generated with the average attention weight and the missing mask as reference, is the weight result based on η.

[0110] In another exemplary embodiment of the present application, the surface temperature image of the set area in 2020 is used as the test data. Based on the method of filling in spatial information and filling in pixels with temporal information proposed in the present application, the missing pixels due to cloud pollution are well filled. In order to analyze the accuracy of the final filled image, the accuracy corresponding to each pixel position is statistically calculated, such as Figure 7 (a) and Figure 7 (b) is shown. The accuracy indicators used in this application are root mean square error and absolute average error, where K is the temperature unit. From the accuracy results, it can be seen that the root mean square error is mainly concentrated around 4K, and the absolute average error is mainly concentrated around 2K. Because the error of satellite sensors in observing surface temperature is about 2K, if the filling accuracy is lower than 2K, it is not very meaningful. Therefore, the method of this application has a higher accuracy and can completely fill in the pixels missing from remote sensing images due to cloud pollution.

[0111] Step 106, reconstructing a remote sensing filled image according to the multiple one-dimensional filled time series arrays.

[0112] In an exemplary embodiment, after the pixels generated by the SAITS model are filled into the corresponding positions of the image, some noise points will appear in the resulting image, which are actually outliers. In order to solve this problem, the application uses median filtering to eliminate these abnormal image points, making the resulting image smoother. The specific implementation formula is as follows:

[0113] h(i,j)=med{f(ik,jl)},k,l∈W;

[0114] Wherein, h is the remote sensing infill image after noise elimination, f(ik, jl) is the pixel of the remote sensing infill image at the position (ik, jl), W = {-1, 0, 1}, k and l are the row offset and column offset relative to the current position (i, j), respectively.

[0115] like Figure 8 (a) to Figure 8 As shown in (d), between the temperature range of 24K-30K, by comparing the original cloud-contaminated image with the final filled image, it can be seen that the method proposed in this application fills the missing pixels very well, not only with high accuracy, but also ensuring spatial continuity.

[0116] The present application also provides an application scenario, which applies the above-mentioned method for filling missing pixels of remote sensing images contaminated by clouds. Specifically: the method for filling missing pixels provided in this embodiment can be applied in remote sensing image processing scenarios. Remote sensing image processing scenarios usually include image acquisition, preprocessing, feature extraction and application analysis. The remote sensing image enters the preprocessing link from the acquisition link, and obtains a clear image after a series of processing steps, and enters the downstream feature extraction and application analysis link. The method for filling missing pixels provided in this embodiment belongs to the image restoration step in the preprocessing link. Specifically, in the preprocessing process of remote sensing images, multiple remote sensing images to be filled can be used to judge the situation of cloud-contaminated pixels and their surrounding pixels, combined with the time series prediction ability of the SAITS model, to fill the missing pixels, thereby obtaining a clear remote sensing image after filling.

[0117] Based on the same inventive concept, the embodiment of the present application also provides a device for implementing the missing pixel filling of the cloud-contaminated remote sensing image mentioned above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more missing pixel filling devices for cloud-contaminated remote sensing images provided below can refer to the limitations of the missing pixel filling method for cloud-contaminated remote sensing images above, and will not be repeated here.

[0118] In an exemplary embodiment, Figure 6 As shown, a missing pixel filling device for a remote sensing image contaminated by clouds is provided, comprising:

[0119] The remote sensing image acquisition module 301 is used to acquire multiple remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained in any two remote sensing images to be filled are different.

[0120] The pixel matrix acquisition module 302 is used to respectively acquire the original pixel matrix of each remote sensing image to be filled.

[0121] The spatial filling module 303 is used to traverse each original pixel matrix respectively. If the i-th pixel in the original pixel matrix is ​​a cloud-contaminated pixel, it is determined whether there is a non-cloud-contaminated pixel in the preset window centered on the i-th pixel to obtain a first judgment result; if the first judgment result is yes, the mean of all non-cloud-contaminated pixels in the preset window is calculated, and the mean is filled to the position of the i-th pixel; if the first judgment result is no, the i-th pixel is marked as a missing pixel to be filled; until all original pixel matrices are traversed, multiple missing pixel matrices to be filled of remote sensing images to be filled are obtained.

[0122] The one-dimensional array acquisition module 304 is used to extract the pixel at the i-th pixel position in the missing pixel matrix to be filled in all remote sensing images to be filled, and sort them in chronological order to obtain multiple one-dimensional original time series arrays.

[0123] The time filling module 305 is used to input multiple one-dimensional original time series arrays into the SAITS model. The SAITS model predicts and fills the missing pixel values ​​at the tth moment based on the non-cloud pollution pixels corresponding to all moments except the tth moment in each one-dimensional original time series array, thereby obtaining multiple one-dimensional filled time series arrays.

[0124] The padded image acquisition module 306 is used to reconstruct a remote sensing padded image according to the multiple one-dimensional padded time series arrays.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store cloud-contaminated remote sensing image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for filling missing pixels of a cloud-contaminated remote sensing image is implemented.

[0126] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0127] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0128] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0130] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0131] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for filling missing pixels in remote sensing images contaminated by clouds, characterized in that: The missing pixel filling method of the cloud-contaminated remote sensing image comprises: Acquire multiple remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained in any two remote sensing images to be filled are different; Respectively obtain the original pixel matrix of each remote sensing image to be filled; Traverse each original pixel matrix respectively. If the i-th pixel in the original pixel matrix is ​​a cloud-polluted pixel, determine whether there is a non-cloud-polluted pixel in the preset window centered on the i-th pixel, and obtain a first judgment result; if the first judgment result is yes, calculate the mean of all non-cloud-polluted pixels in the preset window, and fill the mean to the position of the i-th pixel; if the first judgment result is no, mark the i-th pixel as a missing pixel to be filled; until all original pixel matrices are traversed, multiple missing pixel matrices to be filled of remote sensing images to be filled are obtained; Extract the pixel at the i-th pixel position in the matrix of missing pixels to be filled in all remote sensing images to be filled, and sort them in chronological order to obtain multiple one-dimensional original time series arrays; Input multiple one-dimensional original time series arrays into the SAITS model. The SAITS model predicts and fills in the missing pixel values ​​at the tth moment according to the non-cloud-polluted pixels corresponding to all moments except the tth moment in each one-dimensional original time series array, and obtains multiple one-dimensional filled time series arrays. A remote sensing filled image is reconstructed based on the multiple one-dimensional filled time series arrays.

2. The method for filling missing pixels of cloud-contaminated remote sensing images according to claim 1, characterized in that: The original pixel matrix of each remote sensing image to be filled is obtained respectively, and the previous steps also include: Preprocessing is performed on multiple remote sensing images to be filled; the preprocessing specifically includes: setting the image range, spatial resolution and spatial reference coordinate system of the multiple remote sensing images to be filled to a preset image range, preset spatial resolution and preset spatial reference coordinate system, and marking the cloud-contaminated pixels in each remote sensing image to be filled as "nan".

3. The method for filling missing pixels of cloud-contaminated remote sensing images according to claim 1, characterized in that: The mean calculation formula for all non-cloud-contaminated pixels in the preset window is: Wherein, B is the interpolated image pixel matrix, A is the original pixel matrix, K represents the weight matrix of the sliding window of the first preset size, i=(0, 1, 2, ..., i max ) represents the row number of the original pixel matrix, j = (0, 1, 2, ..., j max ) represents the column number of the original pixel matrix, m and n represent the row offset and column offset of the pixel in the sliding window of the first preset size relative to the current interpolation position (i, j) in the original pixel matrix, and Num is the total number of non-cloud-contaminated pixels in the sliding window of the first preset size.

4. The method for filling missing pixels of cloud-contaminated remote sensing images according to claim 1, characterized in that: The SAITS model specifically includes: a first diagonal shielded self-attention block, a second diagonal shielded self-attention block and a weighted combination module; The first diagonal masked self-attention block is respectively connected to the second diagonal masked self-attention block and the weighted combination module, and the second diagonal masked self-attention block is connected to the weighted combination module; The first diagonal masked self-attention block is used to take the one-dimensional original time series array as input, learn the relationship between the pixel at time t and the pixels before and after time t in the time dimension, preliminarily predict the filling value of the missing pixel, and output a preliminary one-dimensional filled time series array; The second diagonal masked self-attention block is used to learn again the relationship between the pixel at time t and the pixels before and after time t in the time dimension, predict the filling value of the missing pixel again, and output the one-dimensional filling time series array of the second filling; The weighted combination module is used to weightedly fuse the preliminary one-dimensional padded time series array output by the first diagonal masked self-attention block and the secondary padded one-dimensional padded time series array output by the second diagonal masked self-attention block, and output the final one-dimensional padded time series array.

5. The method for filling missing pixels of cloud-contaminated remote sensing images according to claim 4, characterized in that: The calculation formula of the first diagonal masked self-attention block is as follows: z={FFN(DiagMaskedMHA(e))} N ; in, is the padding value of the first diagonal masked self-attention block, is a one-dimensional original time series array, p is the position code, Concat is the array concatenation function, FFN is the feedforward network, DiagMaskedMHA is the multi-head diagonal mask attention mechanism, For the initial one-dimensional padded time series array, {} N To perform N times of nested attention functions and feed-forward networks, W e 、b e and W z are the first simulation coefficient, the second simulation coefficient and the third simulation coefficient respectively, e and z are the first intermediate variable and the second intermediate variable respectively, ⊙ is the element product, is the mask vector of the one-dimensional original time series array; The specific implementation formula of the second diagonal masked self-attention block is as follows: β={FFN(DiagMaskedMHA(α))} N : in, is the padding value of the second diagonal masked self-attention block, W α , W β , W γ 、b α 、b β and b γ are the fourth simulation coefficient, the fifth simulation coefficient, the sixth simulation coefficient, the seventh simulation coefficient, the eighth simulation coefficient, the ninth simulation coefficient and the tenth simulation coefficient respectively, and α and β are the third intermediate variable and the fourth intermediate variable respectively; The specific implementation formula of the weighted combination module is as follows: in, is the missing pixel filling value, η is the combined weight generated with the average attention weight and the missing mask as reference, is the weight result based on η.

6. The method for filling missing pixels of cloud-contaminated remote sensing images according to claim 1, characterized in that: Reconstructing a remote sensing filled image according to the plurality of one-dimensional filled time series arrays, and then further comprising: The noise of remote sensing infill images is eliminated by median filtering method; The formula of the median filtering method is: h(i,j)=med{f(ik,jl)},k,l∈W; Wherein, h is the remote sensing infill image after noise elimination, f(ik, jl) is the pixel of the remote sensing infill image at the position (ik, jl), W = {-1, 0, 1}, k and l are the row offset and column offset relative to the current position (i, j), respectively.

7. A device for filling missing pixels of remote sensing images contaminated by clouds, characterized in that: The missing pixel filling device for the cloud-contaminated remote sensing image comprises: The remote sensing image acquisition module is used to acquire multiple remote sensing images to be filled according to a preset time interval; wherein each remote sensing image to be filled contains image data of only one band, and the bands contained in any two remote sensing images to be filled are different; A pixel matrix acquisition module is used to respectively acquire the original pixel matrix of each remote sensing image to be filled; The spatial filling module is used to traverse each original pixel matrix respectively. If the i-th pixel in the original pixel matrix is ​​a cloud-polluted pixel, it is judged whether there is a non-cloud-polluted pixel in the preset window centered on the i-th pixel to obtain a first judgment result; if the first judgment result is yes, the mean of all non-cloud-polluted pixels in the preset window is calculated, and the mean is filled to the position of the i-th pixel; if the first judgment result is no, the i-th pixel is marked as a missing pixel to be filled; until all original pixel matrices are traversed, multiple missing pixel matrices to be filled of remote sensing images to be filled are obtained; A one-dimensional array acquisition module is used to extract the pixel at the i-th pixel position in the missing pixel matrix to be filled in all remote sensing images to be filled, and sort them in chronological order to obtain multiple one-dimensional original time series arrays; The time filling module is used to input multiple one-dimensional original time series arrays into the SAITS model. The SAITS model predicts and fills the missing pixel values ​​at the tth moment according to the non-cloud pollution pixels corresponding to all moments except the tth moment in each one-dimensional original time series array, and obtains multiple one-dimensional filled time series arrays. The padded image acquisition module is used to reconstruct a remote sensing padded image based on the multiple one-dimensional padded time series arrays.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for filling missing pixels of a remote sensing image contaminated by clouds as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for filling missing pixels of a cloud-contaminated remote sensing image described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for filling missing pixels of a cloud-contaminated remote sensing image described in any one of claims 1 to 6 is implemented.

Images