A remote sensing data processing method and device based on time-space fusion

Abstract

The present application discloses a remote sensing data processing method and device based on time-space fusion. The method includes: obtaining a low-resolution image sequence and a high-resolution image sequence, wherein the high-resolution image includes M first pixels located at M pixel positions, and the low-resolution image includes N second pixels located at N pixel positions; correcting the first abnormal pixel existing in the m-th pixel position to obtain a first corrected image sequence; setting the second abnormal pixel existing in the n-th pixel position among the N pixel positions to a null value to obtain a second corrected image sequence; setting the first target pixel corresponding to the second abnormal pixel in the first corrected image sequence to a null value to obtain a third corrected image sequence, and setting a number of second target pixels corresponding to the first target pixel in the second corrected image sequence to null values ​​to obtain a fourth corrected image sequence; performing machine learning based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing data processing, and in particular to a remote sensing data processing method and device based on time-space fusion. Background Art

[0002] In remote sensing image processing, image data obtained by different sensors have different disadvantages. For example, the data obtained by Landsat satellites have high resolution but are discontinuous in time, while the data obtained by MODIS has low spatial resolution. Through spatiotemporal fusion methods, data with different characteristics can be fused to achieve complementary advantages between data from different sources and achieve the purpose of remote sensing data enhancement. However, most of the current spatiotemporal fusion methods have weak robustness of fusion results, and due to the high repetitiveness of the fusion process in each period, the fusion process efficiency is low. Summary of the invention

[0003] The present application provides a remote sensing data processing method and device based on time-space fusion.

[0004] In a first aspect, a remote sensing data processing method based on time-space fusion is provided. The method comprises:

[0005] Acquire a low-resolution image sequence and a high-resolution image sequence acquired for the area to be tested, wherein any i-th low-resolution image and any i-th high-resolution image are acquired by the first sensor and the second sensor at the same time, respectively, the low-resolution image includes M first pixels located at M pixel positions, and the high-resolution image includes second pixels located at N pixel positions;

[0006] For any m-th pixel position among the M pixel positions, determine whether there is a first abnormal pixel in the m-th first pixel sequence corresponding to the pixel position, and if yes, determine a correction value corresponding to the first abnormal pixel according to a number of adjacent pixel sequences adjacent to the m-th first pixel sequence in the M first pixel sequences corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a first corrected image sequence corresponding to the low-resolution image sequence;

[0007] For any n-th pixel position among the N pixel positions, determine whether there is a second abnormal pixel in the corresponding n-th second pixel sequence, and if yes, set the second abnormal pixel to a null value to obtain a second corrected image sequence corresponding to the high-resolution image sequence;

[0008] The first target pixel corresponding to the second abnormal pixel in the first corrected image sequence is set to a null value to obtain a third corrected image sequence, and a number of second target pixels corresponding to the first target image in the second corrected image sequence are all set to null values ​​to obtain a fourth corrected image sequence;

[0009] Machine learning is performed based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model, wherein the time series regression model is used to process a target low-resolution image collected at any target moment to obtain a target high-resolution image corresponding to the target moment.

[0010] In a possible implementation, performing machine learning according to the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model further includes:

[0011] Sampling i*M first pixels included in the third corrected image sequence to obtain an amplified third corrected image sequence; the amplified third corrected image sequence has the same resolution as the high-resolution image;

[0012] Machine learning is performed based on the amplified third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model.

[0013] In a possible implementation, the method further includes:

[0014] Calculate i*M difference items according to the i*M first pixels included in the third corrected image sequence and the i*M second pixels included in the fourth corrected image sequence; the difference items are difference values ​​generated when the first sensor and the second sensor acquire images of the target area to be measured;

[0015] A difference term estimation model is constructed based on the difference term.

[0016] In a possible implementation, performing machine learning according to the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model includes:

[0017] Obtaining a third corrected image sequence with differences eliminated according to the i*M first pixels and the i*M difference items included in the third corrected image sequence;

[0018] Machine learning is performed based on the third corrected image sequence with differences eliminated and the fourth corrected image sequence to obtain a time series regression model.

[0019] In a possible implementation manner, performing machine learning based on the third corrected image sequence and the fourth corrected image sequence with differences eliminated to obtain a time series regression model further includes:

[0020] Machine learning is performed based on the third corrected image sequence and the fourth corrected image sequence with differences eliminated, as well as a difference term estimation model, to obtain a time series regression model.

[0021] In a possible implementation, the method further includes:

[0022] Acquire a target low-resolution image collected from the area to be tested, wherein the target low-resolution image is collected by the first sensor at a target time; the target low-resolution image includes M first pixels located at M pixel positions,

[0023] For any m-th pixel position among the M pixel positions, determine whether the m-th pixel corresponding to it is a first abnormal pixel, and if it is, determine a correction value corresponding to the first abnormal pixel according to a number of adjacent pixels of the m-th first pixel sequence in the M first pixels corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a third target corrected image corresponding to the target low-resolution image;

[0024] The third target corrected image is processed according to the time series regression model; and a target high-resolution image corresponding to the target moment is obtained.

[0025] In a possible implementation, performing machine learning according to the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model includes:

[0026] Sampling i*M first pixels included in the third corrected image sequence to obtain an amplified third corrected image sequence; the amplified third corrected image sequence has the same resolution as the high-resolution image;

[0027] The step of processing the third target corrected image according to the time series regression model to obtain a target high-resolution image corresponding to the target moment includes:

[0028] The M first pixels included in the third target correction image are sampled to obtain an amplified third target correction image; the amplified third target correction image has the same resolution as the high-resolution image.

[0029] The amplified third target corrected image is processed according to the time series regression model to obtain a target high-resolution image corresponding to the target moment.

[0030] In a possible implementation, performing machine learning according to the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model includes:

[0031] Calculate i*M difference items according to the i*M first pixels included in the third corrected image sequence and the i*M second pixels included in the fourth corrected image sequence; the difference items are difference values ​​generated when the first sensor and the second sensor acquire images of the target area to be measured;

[0032] According to the difference terms, a difference term estimation model is constructed;

[0033] Obtaining a third corrected image sequence with differences eliminated according to the i*M first pixels and the i*M difference items included in the third corrected image sequence;

[0034] Perform machine learning based on the third corrected image sequence and the fourth corrected image sequence that eliminate differences to obtain a time series regression model;

[0035] The step of processing the third target corrected image according to the time series regression model to obtain a target high-resolution image corresponding to the target moment includes:

[0036] According to the difference term estimation model, M difference terms corresponding to the third target corrected image are calculated,

[0037] Obtaining a third target corrected image with differences eliminated according to the M first pixels included in the third target corrected image and the M difference items;

[0038] The third target corrected image with the difference eliminated is processed according to the time series regression model; and a target high-resolution image corresponding to the target moment is obtained.

[0039] In a second aspect, a remote sensing data processing device based on time-space fusion is provided, the device comprising:

[0040] A data acquisition unit, used to acquire a low-resolution image sequence and a high-resolution image sequence acquired from the area to be tested, wherein any i-th low-resolution image and any i-th high-resolution image are acquired by the first sensor and the second sensor at the same time, respectively, the low-resolution image includes M first pixels located at M pixel positions, and the high-resolution image includes second pixels located at N pixel positions;

[0041] a first correction unit, configured to determine, for any m-th pixel position among the M pixel positions, whether there is a first abnormal pixel in the m-th first pixel sequence corresponding to the pixel position, and if so, determine a correction value corresponding to the first abnormal pixel according to a plurality of adjacent pixel sequences adjacent to the m-th first pixel sequence in the M first pixel sequences corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a first corrected image sequence corresponding to the low-resolution image sequence;

[0042] A second correction unit is used to determine, for any n-th pixel position among the N pixel positions, whether there is a second abnormal pixel in the corresponding n-th second pixel sequence, and if yes, set the second abnormal pixel to a null value to obtain a second corrected image sequence corresponding to the high-resolution image sequence;

[0043] a third correction unit, configured to set the first target pixel corresponding to the second abnormal pixel in the first corrected image sequence to a null value to obtain a third corrected image sequence, and to set a number of second target pixels corresponding to the first target image in the second corrected image sequence to null values ​​to obtain a fourth corrected image sequence;

[0044] A model building unit is used to perform machine learning based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model, wherein the time series regression model is used to process a target low-resolution image collected at any target moment to obtain a target high-resolution image corresponding to the target moment.

[0045] According to a third aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the first aspects is implemented.

[0046] In the technical solution provided by the present application, firstly, a low-resolution image sequence and a high-resolution image sequence collected from the area to be tested are obtained, and according to the correlation between the pixel sequences, the abnormal pixels existing in the low-resolution image sequence are corrected; then, the abnormal pixels existing in the high-resolution image sequence are set to null values ​​corresponding to the first pixel in the low-resolution image sequence, and all the second pixels in the high-resolution image sequence corresponding to the first pixel are set to null values; finally, machine learning is performed based on the low-resolution image sequence after two corrections and the high-resolution image sequence after a single correction to obtain a time series regression model; by combining the characteristics of the traditional spatiotemporal fusion prior principle and the machine learning method, the obtained time series regression model can more efficiently and accurately predict the target high-resolution image based on the target low-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 One of the flow charts of a remote sensing data processing method based on time-space fusion provided in an embodiment of the present application;

[0049] Figure 2 The second flowchart of a remote sensing data processing method based on time-space fusion provided in an embodiment of the present application;

[0050] Figure 3 The third flowchart of a remote sensing data processing method based on spatiotemporal fusion provided in an embodiment of the present application;

[0051] Figure 4 The fourth flowchart of a remote sensing data processing method based on spatiotemporal fusion provided in an embodiment of the present application;

[0052] Figure 5 This is a comparison diagram of model errors provided as an example in the embodiments of the present application;

[0053] Figure 6 A comparison diagram of a target high-resolution image provided exemplarily in an embodiment of the present application;

[0054] Figure 7 This is a schematic diagram of a remote sensing data processing device based on time-space fusion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be described below in conjunction with the accompanying drawings. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0057] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0058] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "plurality" means two or more.

[0059] In the description of the embodiments of the present application, the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or are the positions or positional relationships in which the product of the application is usually placed when in use. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0060] In the description of the embodiments of the present application, the terms "horizontal", "vertical", "overhanging", "vertical" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0061] In addition, in the description of the embodiments of the present application, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0062] Remote sensing satellites can easily obtain surface temperature data, which helps humans to fully perceive the urban thermal environment and provide a quantitative basis for mitigating the urban heat island effect. However, in remote sensing image processing, different sensors have different disadvantages in image processing. For example, the spatial resolution data below 100 meters that can be obtained by Landsat satellites are more suitable for urban thermal environment monitoring and can reflect the thermal structure of urban internal objects such as streets and factories to a certain extent. However, due to its long revisit cycle of 16 days, cloud pollution, and sensor failure, the actual available land surface temperature data is limited in time series. Although MODIS data can be revisited daily and obtain data with high temporal resolution, the spatial resolution is low and cannot reflect the details of the city, making it difficult to meet the increasingly sophisticated monitoring scale requirements. Therefore, the existing single sensor cannot take into account both temporal and spatial resolution, which makes thermal infrared remote sensing have long-term data accuracy problems of "high time, low space" or data continuity problems of "high space, low time".

[0063] The spatiotemporal fusion method achieves complementary advantages between data from different sources by fusing high temporal resolution and high spatial resolution data, generating fusion products with both spatiotemporal resolution, and achieving the purpose of enhancing the continuity of remote sensing data. Generally, the spatiotemporal fusion method can accommodate a small number of input image pairs, and requires manual selection of reference images, resulting in poor robustness of the fusion results. The high-resolution images available for input are expensive to obtain and few in number. Even if enough samples are prepared, the fusion efficiency under multiple image pair input is low due to computational cost and hardware conditions.

[0064] In order to obtain sufficiently continuous data in time, the spatiotemporal fusion model needs to generate multi-period data. However, most of the current spatiotemporal fusion models are based on a single input and single output mode, and need to be run repeatedly to produce multi-period fused images. In the process of repeated fusion, there are multiple identical spatial features, which may result in additional costs to process the same features, resulting in reduced fusion efficiency.

[0065] In response to the above problems, the embodiments of the present application at least provide a remote sensing data processing method and device based on time-space fusion.

[0066] Figure 1 This is a flowchart of a remote sensing data processing method based on time-space fusion provided in an embodiment of the present application. Figure 1 The corresponding method mainly and exemplarily describes the process of obtaining a time series regression model for predicting images; the method can be executed by any device, platform, equipment or equipment cluster with computing / processing capabilities.

[0067] like Figure 1 As shown, the method may include but is not limited to part or all of the following steps S101 to S115.

[0068] Step S101, obtaining a low-resolution image sequence and a high-resolution image sequence captured for the area to be tested, wherein any i-th low-resolution image and any i-th high-resolution image are captured by the first sensor and the second sensor respectively at the same time, wherein the low-resolution image includes M first pixels located at M pixel positions, and the high-resolution image includes second pixels located at N pixel positions.

[0069] In the satellite sensor, multiple low-resolution images and multiple high-resolution images corresponding to multiple consecutive time points in the area to be measured can be arranged in time sequence to obtain a low-resolution image sequence and a high-resolution image sequence. Figure 2 As shown in the space-time cube expression in , the low-resolution image sequence can be regarded as a three-dimensional (longitude, latitude, time) first space-time cube that can represent the data of the area to be tested in a continuous time; wherein each low-resolution image in the first space-time cube can represent the spatial distribution of the area to be tested at a specific time point, and the arrangement of these low-resolution image sequences on the time axis forms a record of the data of the area to be tested during continuous time series changes; each first pixel in a single low-resolution image can represent the observation value at a specific spatial point and time point in the area to be tested. Corresponding to the aforementioned first space-time cube, the high-resolution image sequence can be regarded as a three-dimensional (longitude, latitude, time) second space-time cube that can represent the data of the area to be tested in a continuous time.

[0070] Among them, the number of images in the high-resolution image sequence is the same as the number of images in the low-resolution image sequence, and the images in the high-resolution image sequence and the low-resolution image sequence correspond one-to-one in time; and because the size of the first pixel is larger than the second pixel, a single first pixel in the low-resolution image can correspond to multiple second pixels in the high-resolution image, in other words, M<N.

[0071] For example, you can Figure 3 As shown in the training phase, the t1-t corresponding to the target area to be tested is obtained. n The fine-resolution image and the coarse-resolution image at each moment are obtained and preprocessed.

[0072] Step S103, for any m-th pixel position among the M pixel positions, determine whether there is a first abnormal pixel in the m-th first pixel sequence corresponding to it, and if so, determine a correction value corresponding to the first abnormal pixel based on a number of adjacent pixel sequences adjacent to the m-th first pixel sequence in the M first pixel sequences corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a first corrected image sequence corresponding to the low-resolution image sequence.

[0073] For any m-th pixel position among the M pixel positions, according to the quality band of the low-resolution image sequence, cloud masking, quality control index calculation, neighborhood information utilization, or multi-source data fusion methods can be combined to determine the first abnormal pixel in the m-th first pixel sequence caused by cloud cover, pollution or other reasons.

[0074] For the first abnormal pixel existing in the mth first pixel sequence, the size and shape of the local window used to fill the first abnormal pixel in the low-resolution image where the first abnormal pixel is located can be determined first. The local window can be an area of ​​any shape centered on the missing first abnormal pixel in the low-resolution image corresponding to the first abnormal pixel, such as a square or rectangular area, or a circular or other shaped area centered on the first abnormal pixel; the size of the local window is determined by the spatial resolution of the low-resolution image and the local correlation of the data.

[0075] After the local window is determined, the correlation between the mth first pixel sequence and a plurality of adjacent pixel sequences adjacent to the mth first pixel sequence in the local window is calculated; wherein the correlation can be represented by a correlation coefficient or other appropriate similarity index. For example, the correlation can be calculated by using a Pearson correlation coefficient, a Spearman correlation coefficient, etc.

[0076] Subsequently, the adjacent pixel sequence with the highest correlation with the mth first pixel sequence is selected from the adjacent pixel sequence in the local window as the reference pixel sequence, and the reference pixel sequence is used as the independent variable and the mth first pixel sequence is used as the dependent variable to construct a corrected regression model to fill the first abnormal pixel, thereby obtaining a first corrected image sequence corresponding to the low-resolution image sequence. The corrected regression model can adopt linear regression, polynomial regression or other regression models.

[0077] It should be noted that in order to better evaluate the effect of image spatiotemporal fusion, in some embodiments, the first abnormal pixel filled in the low-resolution image sequence can be recorded; for example, the pixel after the first abnormal pixel is corrected can be recorded as 1, and the first pixel without abnormality can be recorded as 0.

[0078] For example, combining Figure 2 and Figure 3 As shown; for Figure 2 In the mth first pixel sequence of the coarse pixel image sequence represented by a series of rectangular blocks, the large area missing image in the mth first pixel sequence is screened out, that is, Figure 2 The pixel values ​​represented by the dark rectangles in the series of rectangles shown are set to null values; and based on the correlation between the first pixel sequences in the coarse pixel image sequence, a coarse-coarse regression relationship is constructed to correct the large-area missing image, that is, to correct Figure 2 The pixels represented by the light-colored rectangles in the series of rectangles shown are corrected.

[0079] Step S105, for any nth pixel position among the N pixel positions, determine whether there is a second abnormal pixel in the corresponding nth second pixel sequence, and if so, set the second abnormal pixel to a null value to obtain a second corrected image sequence corresponding to the high-resolution image sequence.

[0080] Similar to the above step S103, the process of obtaining the second corrected image sequence corresponding to the high-resolution image sequence is not further described here.

[0081] Step S107, setting the first target pixel corresponding to the second abnormal pixel in the first corrected image sequence to a null value to obtain a third corrected image sequence, and setting a number of second target pixels corresponding to the first target image in the second corrected image sequence to null values ​​to obtain a fourth corrected image sequence.

[0082] Optional, see Figure 3 In the prediction stage shown, the difference term estimation can be implemented through steps S107 to S109.

[0083] Optionally, in step S109, K*M difference items are calculated based on the K*M first pixels included in the third corrected image sequence and the K*M second pixels included in the fourth corrected image sequence; the difference items are difference values ​​generated when the first sensor and the second sensor capture images of the target area to be measured.

[0084] Since the low-resolution image sequence and the high-resolution image sequence are obtained by different sensors and there is a sensor deviation between different sensors, there is a difference between the pixel value of the first pixel and the pixel value of the second pixel at the same position of the target area to be measured at the same time, thereby weakening the correlation between the low-resolution image sequence and the high-resolution image sequence.

[0085] Based on the first law of geography and the heat transfer hypothesis, the difference term can be solved according to the pixel position of the first pixel in the low-resolution image sequence and the pixel position of the first pixel and the corresponding second pixel in the high-resolution image sequence. For example, it can be solved according to the following formula 1:

[0086]

[0087] Among them, ξ(x i ,t) is the first pixel (x i ,t) corresponding to the difference term; c(x i ,t) is the first pixel (x i, t); m is the number of multiple second pixels corresponding to a single first pixel; F(x ik ,t) is the second pixel (x ik ,t) pixel value; In the high-resolution image sequence, the first pixel (x i ,t) corresponds to the pixel mean of the plurality of second pixels. Optionally, in step S109, a difference term estimation model is constructed according to the K*M difference terms.

[0088] For example, the difference term estimation model can be constructed by the following formula 2:

[0089]

[0090] Among them, ξ(x i ,t) is the first pixel (x i ,t) corresponding to the difference term; α ξ , β ξ , γ ξ ,θ ξ is the first pixel (x i ,t), where J(t) is the Julian day at time t.

[0091] Then, the K*M difference terms obtained in step S109 are substituted into formula 2 to obtain the unknown parameter α of the difference term estimation model: ξ , β ξ , γ ξ ,θ ξ , thus obtaining the difference term estimation model.

[0092] Optionally, in step S111, a third corrected image sequence with differences eliminated is obtained according to the K*M first pixels and the K*M difference items included in the third corrected image sequence.

[0093] The corresponding difference term is subtracted from each of the K*M first pixels in the third corrected image sequence to obtain a third corrected image sequence with the difference eliminated.

[0094] By eliminating the difference term, the pixel value of the first pixel in the third corrected image sequence from the first sensor and processed is made closer to the pixel value of the second pixel in the fourth corrected image sequence, thereby reducing the difference between the two. This can improve the difference between the images obtained by different sensors in the target area to be measured, making the subsequent fusion algorithm more accurate and stable.

[0095] Optionally, in step S113, K*M first pixel paths included in the third corrected image sequence with the difference eliminated are sampled to obtain an amplified third corrected image sequence, and the amplified third corrected image sequence has the same resolution as the high-resolution image.

[0096] The first pixel in the third corrected image sequence that eliminates the difference is spatially interpolated and expanded through interpolation or other sampling methods, so that the low-resolution image and the high-resolution image in the enlarged third corrected image sequence have the same number of pixels and spatial resolution, so as to facilitate alignment and fusion processing.

[0097] To avoid the occurrence of artifacts, in some possible embodiments, spatial filtering may be performed on the high-resolution first space-time cube.

[0098] After sampling the first pixel, a spatial filtering method can be used to smooth the amplified third corrected image sequence, so that the amplified third corrected image sequence after sampling is smoother and the jagged artifacts at the boundary are avoided as much as possible. The spatial filtering method includes mean filtering, Gaussian filtering, median filtering, etc.

[0099] It should be noted that the K*M first pixels included in the third corrected image sequence may also be directly sampled to obtain the amplified third corrected image sequence.

[0100] Step S115, machine learning is performed based on the amplified third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model, wherein the time series regression model is used to process the target low-resolution image collected at any target time to obtain the target high-resolution image corresponding to the target time.

[0101] Feature data is extracted from the third corrected image sequence and the fourth corrected image sequence, and a regression model is established according to the extracted feature data.

[0102] For example, the time series regression model can be obtained by the following formula 3:

[0103] F RM (x ij , t) = ∝ F (x ij )[c(x i , t)-ξ(x i ,t)]+β F (x ij )+r CF (3)

[0104] Among them, F RM (x ij , t) represents the second pixel (xij , t) pixel value; c(x i ,t) is the first pixel (x i ,t) pixel value; ξ(x i ,t) is the first pixel (x i ,t) corresponding to the difference term; ∝ F (x ij ), β F (x ij ) is the second pixel (x ij , t) is related to the pixel sequence, r CF is the linear residual between the second pixels.

[0105] r in the above formula 3 CF It can be calculated according to the following formula 4:

[0106]

[0107] Among them, r CF is the linear residual between the second pixels; r(x ik , x ij , t) is the second pixel (x) in the t-th second pixel sequence among the N second pixel sequences of the high-resolution image sequence. ij , t) and the second pixel (x ik , t) linear residual; a(x ij , x ij ) weight coefficient.

[0108] It should be noted that machine learning may also be performed directly based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model.

[0109] After obtaining the time series regression model for predicting images through the aforementioned steps S101 to S115, the time series regression model can be deployed to any device, platform, equipment or equipment cluster with computing / processing capabilities. Correspondingly, the subsequent process can use the deployed time series regression model for predicting images as needed to predict the target high-resolution image corresponding to the target moment.

[0110] See also Figure 4 As shown, the method may include but is not limited to part or all of the following steps S201 and S209.

[0111] Step S201 , obtaining a target low-resolution image captured from the area to be tested, wherein the target low-resolution image is captured by a first sensor at a target time; the target low-resolution image includes M third pixels located at M pixel positions.

[0112] Step S203, for any m-th third pixel position among the M pixel positions, determine whether the corresponding m-th pixel is a third abnormal pixel; if so, determine a correction value corresponding to the third abnormal pixel according to a number of adjacent pixels of the m-th third pixel sequence among the M third pixels corresponding to the M pixel positions, and correct the third abnormal pixel to obtain a third target corrected image corresponding to the target low-resolution image.

[0113] Similar to the aforementioned step S103, cloud masking, quality control index calculation, neighborhood information utilization, or multi-source data fusion methods may be used in combination to determine that the mth pixel has a third abnormal pixel caused by cloud cover, pollution or other reasons.

[0114] Subsequently, for the third abnormal pixel in the target low-resolution image, a local window for correcting the third abnormal pixel can be determined first. After the local window is determined, the correlation between each third pixel in the local window and the third abnormal pixel is calculated; subsequently, the third pixel with the highest correlation with the third abnormal pixel is selected from the third pixels in the local window as a reference pixel, and the pixel value of the reference pixel is used as the pixel value of the third abnormal pixel, thereby forming a complete target low-resolution image.

[0115] It should be noted that, in some embodiments, when filling, the target low-resolution image can also be placed in the low-resolution image sequence proposed in step 101, and the third abnormal pixel in the target low-resolution image can be corrected according to the repair regression model constructed in the aforementioned step S103.

[0116] Optionally, in step S205, M difference terms corresponding to the third target corrected image are calculated according to the difference term estimation model; and the third target corrected image with the difference eliminated is obtained according to the M third pixels and the M difference terms included in the third target corrected image.

[0117] The M difference terms corresponding to the M third pixels included in the third target corrected image are calculated by using the difference term estimation model obtained in the aforementioned step S109.

[0118] After the difference term is calculated, similar to the aforementioned step S111 , the pixel values ​​of the third pixels in the third target corrected image are respectively subtracted from the corresponding difference terms to obtain a third target corrected image with the difference eliminated.

[0119] Optionally, in step S207, the M third pixels included in the third target corrected image with the difference eliminated are sampled to obtain an amplified third target corrected image, and the amplified third target corrected image has the same resolution as the high-resolution image.

[0120] Similar to the aforementioned step S113, the third pixel in the third target correction image with the difference can be eliminated, and the third pixel is spatially interpolated and expanded by interpolation or other sampling methods to obtain an enlarged third target correction image. The enlarged third target correction image has the same number of pixels and spatial resolution as the high-resolution image, so as to facilitate alignment and fusion processing.

[0121] To avoid the occurrence of artifacts, in some possible embodiments, spatial filtering may be performed on the amplified third target corrected image.

[0122] It should be noted that step S207 may also directly obtain the enlarged third target corrected image according to the third target corrected image obtained in step 203 .

[0123] Step S209, processing the amplified third target corrected image according to the time series regression model to obtain a target high-resolution image corresponding to the target moment.

[0124] The amplified third target corrected image is input into the time series regression model obtained in the aforementioned step S115 to calculate the target high-resolution image corresponding to the target moment. It should be noted that the third target corrected image can also be directly input into the time series regression model obtained in the aforementioned step S115 to calculate the target high-resolution image corresponding to the target moment.

[0125] In one embodiment of the remote sensing data processing method according to the application of the spatiotemporal fusion provided by the present application, referring to Figure 5 As shown in the comparison diagram of different model errors, the method provided by the present application can approach the high-precision area more stably.

[0126] In another embodiment of the remote sensing data processing method using the spatiotemporal fusion provided by the present application, referring to Figure 6 As shown, the data produced by the method provided in this application has a significant improvement in spatial resolution compared to MODIS data, is consistent with Landsat, and has a higher temporal resolution.

[0127] Corresponding to the above method provided by the present application, the present application also provides a remote sensing data processing device based on time-space fusion. Figure 7 As shown, the device includes: a data acquisition unit, a first correction unit, a second correction unit, a third correction unit, and a model building unit.

[0128] A data acquisition unit is used to acquire a low-resolution image sequence and a high-resolution image sequence captured on the test area, wherein any i-th low-resolution image and any i-th high-resolution image are captured by the first sensor and the second sensor respectively at the same time, the low-resolution image includes M first pixels located at M pixel positions, and the high-resolution image includes second pixels located at N pixel positions.

[0129] The first correction unit is used to determine, for any m-th pixel position among the M pixel positions, whether there is a first abnormal pixel in the m-th first pixel sequence corresponding to it; if so, determine a correction value corresponding to the first abnormal pixel based on a number of adjacent pixel sequences adjacent to the m-th first pixel sequence in the M first pixel sequences corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a first corrected image sequence corresponding to the low-resolution image sequence.

[0130] The second correction unit is used to determine whether there is a second abnormal pixel in the corresponding nth second pixel sequence for any nth pixel position among the N pixel positions, and if so, set the second abnormal pixel to a null value to obtain a second corrected image sequence corresponding to the high-resolution image sequence.

[0131] The third correction unit is used to set the first target pixel corresponding to the second abnormal pixel in the first corrected image sequence to a null value to obtain a third corrected image sequence, and to set a number of second target pixels corresponding to the first target image in the second corrected image sequence to null values ​​to obtain a fourth corrected image sequence.

[0132] The model building unit is used to perform machine learning based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model, wherein the time series regression model is used to process the target low-resolution image collected at any target moment to obtain the target high-resolution image corresponding to the target moment.

[0133] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 1 The method steps in the described method are performed by the data acquisition unit, the first correction unit, the second correction unit, the third correction unit, and the model building unit.

[0134] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor. The memory stores executable code. When the processor executes the executable code, the Figure 1 The method steps in the described method are performed by the data acquisition unit, the first correction unit, the second correction unit, the third correction unit, and the model building unit.

[0135] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A remote sensing data processing method based on time-space fusion, characterized in that: include: Acquire a low-resolution image sequence and a high-resolution image sequence acquired for the area to be tested, wherein any i-th low-resolution image and any i-th high-resolution image are acquired by the first sensor and the second sensor at the same time, respectively, the low-resolution image includes M first pixels located at M pixel positions, and the high-resolution image includes second pixels located at N pixel positions; For any m-th pixel position among the M pixel positions, determine whether there is a first abnormal pixel in the m-th first pixel sequence corresponding to the pixel position, and if yes, determine a correction value corresponding to the first abnormal pixel according to a number of adjacent pixel sequences adjacent to the m-th first pixel sequence in the M first pixel sequences corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a first corrected image sequence corresponding to the low-resolution image sequence; For any n-th pixel position among the N pixel positions, determine whether there is a second abnormal pixel in the corresponding n-th second pixel sequence, and if yes, set the second abnormal pixel to a null value to obtain a second corrected image sequence corresponding to the high-resolution image sequence; The first target pixel corresponding to the second abnormal pixel in the first corrected image sequence is set to a null value to obtain a third corrected image sequence, and the plurality of second target pixels corresponding to the first target pixel in the second corrected image sequence are all set to null values ​​to obtain a fourth corrected image sequence; Performing machine learning based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model, wherein the time series regression model is used to process a target low-resolution image collected at any target moment to obtain a target high-resolution image corresponding to the target moment; The method further comprises: Acquire a target low-resolution image collected from the area to be tested, wherein the target low-resolution image is collected by the first sensor at a target time; the target low-resolution image includes M third pixels located at M pixel positions, For any m-th pixel position among the M pixel positions, determine whether the m-th pixel corresponding thereto is a third abnormal pixel, and if so, determine a correction value corresponding to the first abnormal pixel according to a number of adjacent pixels adjacent to the m-th first pixel among the M third pixels corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a third target corrected image corresponding to the target low-resolution image; The third target corrected image is processed according to the time series regression model; and a target high-resolution image corresponding to the target moment is obtained.

2. The method according to claim 1, wherein the performing machine learning based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model further comprises: Upsampling each first pixel in the third corrected image sequence to obtain an amplified third corrected image sequence, wherein each image in the amplified third corrected image sequence has the same resolution as the high-resolution image; Machine learning is performed based on the amplified third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model.

3. The method according to claim 1, further comprising: Calculating K*M difference items according to the K*M first pixels included in the third corrected image sequence and the K*M second pixels included in the fourth corrected image sequence; A difference term estimation model is constructed based on the K*M difference terms.

4. The method according to claim 3, wherein the performing machine learning based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model comprises: Obtaining a third corrected image sequence with differences eliminated according to the K*M first pixels and the K*M difference items included in the third corrected image sequence; Machine learning is performed based on the third corrected image sequence with differences eliminated and the fourth corrected image sequence to obtain a time series regression model.

5. The method according to claim 4, wherein the performing machine learning based on the third corrected image sequence and the fourth corrected image sequence with differences eliminated to obtain a time series regression model further comprises: Machine learning is performed based on the third corrected image sequence that eliminates the difference, the fourth corrected image sequence, and the difference term estimation model to obtain a time series regression model.

6. The method according to claim 1, said processing of the third target corrected image according to the temporal regression model; Obtain a high-resolution image of the target corresponding to the target time, including: Sampling the M third pixels included in the third target correction image to obtain an amplified third target correction image; the amplified third target correction image has the same resolution as the high-resolution image; The amplified third target corrected image is processed according to the time series regression model to obtain a target high-resolution image corresponding to the target moment.

7. The method according to claim 1, said processing of the third target corrected image according to the temporal regression model; Obtain a high-resolution image of the target corresponding to the target time, including: According to a predetermined difference term estimation model, M difference terms corresponding to the third target corrected image are calculated, Obtaining a third target corrected image with differences eliminated according to the M third pixels included in the third target corrected image and the M difference items; The third target corrected image with the difference eliminated is processed according to the time series regression model; and a target high-resolution image corresponding to the target moment is obtained.

8. A remote sensing data processing device based on time-space fusion, the device comprising: A data acquisition unit, used to acquire a low-resolution image sequence and a high-resolution image sequence acquired from the area to be tested, wherein any i-th low-resolution image and any i-th high-resolution image are acquired by the first sensor and the second sensor at the same time, respectively, the low-resolution image includes M first pixels located at M pixel positions, and the high-resolution image includes second pixels located at N pixel positions; a first correction unit, configured to determine, for any m-th pixel position among the M pixel positions, whether there is a first abnormal pixel in the m-th first pixel sequence corresponding to the pixel position, and if so, determine a correction value corresponding to the first abnormal pixel according to a plurality of adjacent pixel sequences adjacent to the m-th first pixel sequence in the M first pixel sequences corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a first corrected image sequence corresponding to the low-resolution image sequence; A second correction unit is used to determine, for any n-th pixel position among the N pixel positions, whether there is a second abnormal pixel in the corresponding n-th second pixel sequence, and if yes, set the second abnormal pixel to a null value to obtain a second corrected image sequence corresponding to the high-resolution image sequence; a third correction unit, configured to set the first target pixel corresponding to the second abnormal pixel in the first corrected image sequence to a null value to obtain a third corrected image sequence, and to set a number of second target pixels corresponding to the first target pixel in the second corrected image sequence to null values ​​to obtain a fourth corrected image sequence; A model building unit, configured to perform machine learning based on the third corrected image sequence and the fourth corrected image sequence to obtain a time series regression model, wherein the time series regression model is used to process a target low-resolution image acquired at any target moment to obtain a target high-resolution image corresponding to the target moment; The device also includes: Acquire a target low-resolution image collected from the area to be tested, wherein the target low-resolution image is collected by the first sensor at a target time; the target low-resolution image includes M third pixels located at M pixel positions, For any m-th pixel position among the M pixel positions, determine whether the m-th pixel corresponding thereto is a third abnormal pixel, and if so, determine a correction value corresponding to the first abnormal pixel according to a number of adjacent pixels adjacent to the m-th first pixel among the M third pixels corresponding to the M pixel positions, and correct the first abnormal pixel to obtain a third target corrected image corresponding to the target low-resolution image; The third target corrected image is processed according to the time series regression model; and a target high-resolution image corresponding to the target moment is obtained.

9. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.

Images