Remote sensing image space-time super-resolution method based on earth surface process modeling

Abstract

The invention provides a remote sensing image space-time super-resolution method based on earth surface process modeling, and relates to the technical field of remote sensing image processing, and the method comprises the steps: carrying out the time-based and space-based interpolation processing of a low-resolution earth surface remote sensing image, obtaining an interpolated earth surface remote sensing image, constructing a control equation used for describing the earth surface space-time change, and carrying out the interpolation processing of the interpolated earth surface remote sensing image; the control equation is solved by adopting an LASSO regression mode to obtain a differential equation used for calculating a high-resolution prediction image, finally, the high-resolution prediction image at a specified time and a specified spatial position is calculated according to the differential equation, and spatial detail features of an interpolation earth surface remote sensing image are fused into the high-resolution prediction image, so that the high-resolution prediction image is obtained. And a super-resolution earth surface remote sensing image is obtained. According to the method and the device, the defect that limitation exists when a traditional interpolation method and a learning method are used for solving space and time resolution limitation of the remote sensing image in the prior art is overcome.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image spatiotemporal super-resolution method based on surface process modeling. Background Art

[0002] Image super-resolution is a challenging task in the field of computer vision. Its purpose is to extract feature information from low-resolution images and then reconstruct high-resolution images. With the continuous improvement and development of modern remote sensing science and technology, satellites have acquired a large amount of high spatial resolution, high temporal resolution, high spectral resolution and high radiometric resolution data, providing rich multimodal information for human exploration of the earth. However, the resolution of remote sensing images is usually limited by the hardware of satellite sensors. The spatial resolution, temporal resolution and radiometric resolution are mutually restricted, and the sensor cannot provide images with high temporal resolution and high spatial resolution at the same time. For example, high-resolution sensors usually have high spatial resolution, but the frequency of capturing the same area is low; while low-resolution sensors can frequently obtain data, but their spatial resolution is low. In addition, affected by factors such as atmospheric disturbances, cloud cover and compression during image transmission, the resolution of remote sensing images is usually low, and the direct analysis of the original images may not be ideal.

[0003] In order to solve the spatial and temporal resolution limitations of remote sensing images in the existing technology, there are two types of spatial and temporal super-resolution methods proposed, but both have limitations. One is the traditional super-resolution method based on interpolation, which has difficulty in capturing complex spatiotemporal dynamic features, which will lead to inaccurate characterization of dynamic changes, and cannot handle high-frequency changes and details in the image well, which will lead to poor processing of high-frequency changes and details in the image. The other is the super-resolution method based on learning, but these methods rely on a large amount of high-quality training data, the model training process is complex, and the computational cost is high. When the temporal data is incomplete or missing, the performance of the model will drop significantly. In addition, the learning method may introduce artificial noise in the data generation process, affecting the authenticity of the image. Summary of the invention

[0004] The present invention provides a remote sensing image spatiotemporal super-resolution method based on surface process modeling, which is used to overcome the defects of the prior art in solving the spatial and temporal resolution limitations of remote sensing images using traditional interpolation methods and learning methods.

[0005] The present invention provides a remote sensing image spatiotemporal super-resolution method based on surface process modeling, the method comprising the following steps: Obtain long-term series of low-resolution surface remote sensing images; Performing time-based and space-based interpolation processing on the low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image; Based on the interpolated surface remote sensing image, a control equation for describing the spatiotemporal changes of the surface is constructed, and the control equation is solved by using a LASSO regression method to obtain a differential equation for calculating a high-resolution prediction image; A high-resolution predicted image of a specified time and spatial position is calculated according to the differential equation, and the spatial detail features of the interpolated surface remote sensing image are integrated into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

[0006] In some embodiments, the performing time-based and space-based interpolation processing on the low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image includes: For any two pixels in the low-resolution surface remote sensing image, the spatial covariance corresponding to the two pixels is obtained by fitting the variogram in the Kriging interpolation; For any two moments in the long time series of the low-resolution surface remote sensing image, the time covariance corresponding to the two moments is obtained by fitting an autoregressive model; Taking the product of the spatial covariance and the temporal covariance as a spatiotemporal covariance matrix; Calculate the spatiotemporal interpolation weights of each neighboring point of the point to be interpolated in the low-resolution surface remote sensing image according to the spatiotemporal covariance matrix; The spatiotemporal interpolation of the points to be interpolated is calculated based on the spatiotemporal interpolation weights to obtain an interpolated surface remote sensing image.

[0007] In some embodiments, constructing a control equation for describing the spatiotemporal changes of the surface based on the interpolated surface remote sensing image includes: A candidate function library is constructed according to the longitude direction information, latitude direction information and time value of the pixel observation value in the interpolated surface remote sensing image, wherein the candidate function library is composed of a first function term and a second function term, the first function term includes: a constant 1, the pixel observation value, the square value of the pixel observation value, the sine transform value of the pixel observation value, and the cosine transform value of the pixel observation value, and the second function term includes: the first-order partial derivative and the second-order partial derivative of the pixel observation value with respect to the longitude direction, the first-order partial derivative and the second-order partial derivative of the pixel observation value with respect to the latitude direction, and the mixed partial derivative of the pixel observation value with respect to the longitude direction and the latitude direction; Combining the first function term and the second function term to obtain a plurality of composite function terms; The control equation of the pixel in the interpolated surface remote sensing image is obtained by fitting the composite function term.

[0008] In some embodiments, the control equation is solved by LASSO regression to obtain a differential equation for calculating a high-resolution predicted image, including: Substituting the pixel observation value in the interpolated surface remote sensing image into each function term in the control equation to obtain a regression equation composed of multiple candidate functions; Determine the coefficient vector composed of the weights of each candidate function in the regression equation by LASSO regression method; Substituting the coefficient vector into the regression equation, a differential equation for calculating a high-resolution predicted image is obtained.

[0009] In some embodiments, determining the coefficient vector composed of the weights of each candidate function in the regression equation by LASSO regression method includes: Calculate the residual sum of squares based on the actual observed values ​​of pixels and the predicted observed values ​​of pixels in the interpolated surface remote sensing image; Determine a weighted regularization value of the coefficient vector, and sum the weighted regularization value with the residual sum of squares to obtain a LASSO loss function of the coefficient vector; The sparse coefficient vector obtained when the LASSO loss function achieves the minimum value is used as the coefficient vector composed of the weights of each candidate function in the regression equation.

[0010] In some embodiments, the step of integrating the spatial detail features of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image includes: Correcting the interpolated surface remote sensing image to obtain a high-resolution image to be introduced whose spatial spectral characteristics are consistent with the high-resolution predicted image; Performing color matching between the color histogram of the high-resolution predicted image and the color histogram of the high-resolution image to be introduced; The spectral response information to be introduced into the high-resolution image is migrated to the high-resolution predicted image after color matching to obtain a super-resolution surface remote sensing image.

[0011] The present invention also provides a remote sensing image spatiotemporal super-resolution device based on surface process modeling, the device comprising the following modules: The acquisition module is used to obtain long-term series of low-resolution surface remote sensing images; An interpolation module, used for performing time-based and space-based interpolation processing on the low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image; A calculation module, used for constructing a control equation for describing the spatiotemporal changes of the surface based on the interpolated surface remote sensing image, and solving the control equation by LASSO regression to obtain a differential equation for calculating the high-resolution predicted image; The fusion module is used to calculate a high-resolution predicted image of a specified time and spatial position according to the differential equation, and integrate the spatial detail features of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the remote sensing image spatiotemporal super-resolution method based on surface process modeling as described in any one of the above-mentioned methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described spatiotemporal super-resolution methods for remote sensing images based on surface process modeling.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the remote sensing image spatiotemporal super-resolution method based on surface process modeling as described in any one of the above is implemented.

[0015] The spatiotemporal super-resolution method of remote sensing images based on surface process modeling provided by the present invention is to perform time-based and space-based interpolation processing simultaneously for long-time series of low-resolution surface remote sensing images, then model the surface change process based on the interpolated surface remote sensing images, construct the control equation for describing the spatiotemporal change of the surface, and use the LASSO regression method to solve the differential equation for calculating the high-resolution predicted image, thereby organically combining the surface transformation process with the remote sensing image data features to calculate the high-resolution predicted image of the specified time and space position, which can overcome the limitations of traditional interpolation methods and learning methods. Finally, the spatial detail features of the interpolated surface remote sensing images are integrated to obtain super-resolution surface remote sensing images, which can significantly improve the spatial resolution and generation quality of the remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1It is a flow chart of the spatiotemporal super-resolution method of remote sensing images based on surface process modeling provided by the present invention.

[0018] Figure 2 It is a framework diagram of low-resolution surface remote sensing image interpolation calculation provided by the present invention.

[0019] Figure 3 It is a schematic diagram of constructing a candidate function library provided by the present invention.

[0020] Figure 4 It is a principle framework diagram of constructing control equations for describing the spatiotemporal changes of the earth's surface provided by the present invention.

[0021] Figure 5 It is a visualization diagram of the super-resolution process provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of a remote sensing image spatiotemporal super-resolution device based on surface process modeling provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The following describes the spatiotemporal super-resolution method and device for remote sensing images based on surface process modeling of the present invention in conjunction with the accompanying drawings. Figure 1 is a flow chart of a remote sensing image spatiotemporal super-resolution method based on surface process modeling provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104, which are described in detail below.

[0026] Step 101: Acquire a long-term series of low-resolution surface remote sensing images.

[0027] The first step is to obtain long-term series of surface remote sensing image data. In order to extract the temporal and spatial changes of the surface as much as possible, it is necessary to obtain a sufficiently long series of low-resolution remote sensing image data, which usually comes from satellite remote sensing observations, such as MODIS, Landsat, etc. According to actual needs, surface remote sensing image data of different periods should have sufficient time span and density to ensure that the process and trend of surface changes can be fully reflected. In the process of data collection, the resolution, coverage, imaging frequency and other factors of the sensor should be comprehensively considered, and the appropriate data source and product layer (such as L2, L3, and L4 data) should be selected for collection, without setting higher requirements for the resolution of surface remote sensing images.

[0028] After acquiring a long-time series of low-resolution surface remote sensing images, in order to model the surface change process, the low-resolution surface remote sensing images need to be preprocessed so that the long-time series of remote sensing images have consistent geometric and radiation benchmarks. When the selected data source is different, the specific preprocessing steps are different, usually including atmospheric correction, radiation correction, and geometric correction steps. The specific correction method of each step also has a corresponding processing method. The embodiment of the present invention does not limit this, but it is necessary to ensure that the long-time series images have consistent geometric and radiation benchmarks, which is the basis for correctly modeling the surface change process.

[0029] Specifically, low-resolution surface remote sensing images may be affected by atmospheric conditions, such as clouds, aerosols, water vapor, etc., which in turn affect the radiation characteristics of the image, so atmospheric correction and radiation correction are required. The purpose of atmospheric correction is to eliminate the influence of the atmosphere on the image, so that the data is closer to the actual surface reflectivity, thereby improving the accuracy and consistency of subsequent spatiotemporal analysis. Radiation correction is to correct the radiation error of the image to ensure the consistency of the image radiation response under different periods and different observation conditions. This standardization process can effectively reduce the radiation differences between time series images and improve the comparability and stability of the data. After completing the atmospheric and radiation corrections, the images need to be geometrically aligned to ensure that the images of different periods are consistent in spatial position, so that pixels in the same geographical location can be accurately aligned in the time series, which can effectively avoid the problem of spatial dislocation and ensure the accuracy and reliability of spatiotemporal analysis.

[0030] Through step 101, high-quality time series remote sensing image data with unified geometric and radiation benchmarks can be obtained, providing a reliable basis for subsequent surface spatiotemporal change modeling and super-resolution reconstruction.

[0031] Step 102: Perform time-based and space-based interpolation processing on the low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image.

[0032] After preprocessing the low-resolution surface remote sensing image, it is necessary to perform time-based and space-based interpolation processing on the low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image.

[0033] The purpose of interpolation is to enhance resolution. The embodiments of the present invention do not limit the specific interpolation method. For example, bilinear interpolation, bicubic interpolation or other interpolation methods are used to perform spatial interpolation on data, and polynomial interpolation, spline interpolation and other methods are used to perform temporal interpolation on data. In some embodiments, temporal and spatial interpolation can also be performed simultaneously. For example, a spatiotemporal coupled interpolation method is selected to perform temporal and spatial interpolation processing on low-resolution surface remote sensing images, so as to enhance the integrity of the data and improve the resolution in temporal and spatial dimensions.

[0034] Step 103: construct a control equation for describing the spatiotemporal variation of the surface based on the interpolated surface remote sensing image, and solve the control equation using the LASSO regression method to obtain a differential equation for calculating the high-resolution prediction image.

[0035] Here, the control equation for describing the spatiotemporal variation of the surface is extracted from the interpolated surface remote sensing image, and a candidate function library of pixel observation values ​​varying with the spatiotemporal variation of the surface is established for the pixel observation values ​​in the interpolated surface remote sensing image, and then the control equation is constructed by linearly combining the function items in the candidate function library, and then the control equation is solved by LASSO regression to determine the coefficient vector of each function item in the candidate function library. Of course, the least squares method can also be used to solve the control equation to determine the coefficient of each function item in the candidate function library, which is not limited in the embodiment of the present invention. The solved differential equation can characterize the spatiotemporal dynamic variation of the surface and be used to calculate high-resolution predicted images.

[0036] Step 104: Calculate a high-resolution predicted image at a specified time and spatial position according to the differential equation, and integrate the spatial detail features of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

[0037] After determining the differential equation used to calculate the high-resolution predicted image, the time information and spatial position information of the pixel observation values ​​in the interpolated surface remote sensing image are substituted into the differential equation to calculate the corresponding high-resolution pixel. Thus, by substituting each pixel into the differential equation for calculation, a high-resolution predicted image of the specified time and spatial position can be obtained.

[0038] After calculating the high-resolution predicted image based on the differential equations that characterize the spatiotemporal changes of the surface, the high-resolution image can be introduced to further improve the image resolution. Here, the spatial detail characteristics of the interpolated surface remote sensing image are integrated into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

[0039] Because the interpolated surface remote sensing image obtained after the interpolation processing in step 102 has enhanced resolution and is a high-resolution image, the spatial detail features of the interpolated surface remote sensing image can be extracted. These spatial detail features include color features, spectral response information features, edge features, texture features and other subtle features. By integrating these spatial detail features into the high-resolution predicted image, a higher-resolution super-resolution surface remote sensing image can be obtained.

[0040] When introducing interpolated surface remote sensing images, it is necessary to perform geometric correction and radiation correction on the interpolated surface remote sensing images to ensure their spatial and spectral consistency with the high-resolution predicted images. Then, the spectral response information characteristics and color characteristics in the interpolated surface remote sensing images are migrated to the high-resolution predicted images through color migration and spectral consistency correction methods. Finally, other spatial details in the interpolated surface remote sensing images, such as edge features, texture features and other subtle features, can be extracted using methods such as principal component analysis (PCA), and these detailed features can be integrated into the high-resolution predicted images to generate the final super-resolution surface remote sensing images.

[0041] The embodiment of the present invention, for a long time series of low-resolution surface remote sensing images, first performs time-based and space-based interpolation processing simultaneously, then models the surface change process based on the interpolated surface remote sensing images, constructs a control equation for describing the spatiotemporal change of the surface, and uses the LASSO regression method to solve the differential equation for calculating the high-resolution predicted image, thereby organically combining the surface transformation process with the remote sensing image data features to calculate the high-resolution predicted image of the specified time and spatial position, which can overcome the limitations of traditional interpolation methods and learning methods. Finally, the spatial detail features of the interpolated surface remote sensing images are integrated to obtain super-resolution surface remote sensing images, which can significantly improve the spatial resolution and generation quality of the remote sensing images.

[0042] In some embodiments, time-based and space-based interpolation processing is performed on a low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image. This can be achieved by following steps 201 to 204, which are described in detail below.

[0043] Step 201: for any two pixels in the low-resolution surface remote sensing image, the spatial covariance corresponding to the two pixels is obtained by fitting the variogram in the Kriging interpolation; for any two moments in the long time series of the low-resolution surface remote sensing image, the temporal covariance corresponding to the two moments is obtained by fitting the autoregressive model.

[0044] like Figure 2As shown in the figure, after preprocessing, low-resolution surface remote sensing images need to be interpolated in time and space. For the preprocessed low-resolution surface remote sensing images, the space-time coupling interpolation method is used for interpolation. In the space-time coupling interpolation method, it is necessary to first build a space-time covariance model, specifically defining the spatial covariance and time covariance. Here, for any two pixels in the low-resolution surface remote sensing image, and , the spatial covariance corresponding to the two pixels is obtained by fitting the variogram in Kriging interpolation, denoted as , which is used to describe the correlation between two pixels. For any two moments in a long time series of low-resolution surface remote sensing images and , the time covariance corresponding to the two moments is obtained by fitting the autoregressive model , which is used to describe the time correlation between two moments. Here, the time variation function can also be used to fit the time covariance corresponding to the two moments , the embodiments of the present invention are not limited.

[0045] Step 202: The product of the spatial covariance and the temporal covariance is taken as the spatiotemporal covariance matrix.

[0046] The next step is to construct the space-time covariance matrix. Here, the spatial covariance is directly Covariance with time The product of is taken as the space-time covariance matrix, and the space-time covariance matrix can be written as .

[0047] Step 203: Calculate the spatiotemporal interpolation weights of each neighboring point of the point to be interpolated in the low-resolution surface remote sensing image according to the spatiotemporal covariance matrix.

[0048] According to steps 201 and 202, the spatiotemporal covariance matrix of all pixels in the entire low-resolution surface remote sensing image can be calculated, indicating the degree of dependence between different pixels in time and space. The subsequent step is to calculate the spatiotemporal interpolation weight, which is calculated based on the spatiotemporal covariance matrix. Specifically, the spatiotemporal interpolation weight of each neighboring point of the interpolation point in the low-resolution surface remote sensing image is calculated based on the spatiotemporal covariance matrix.

[0049] According to the spatiotemporal covariance matrix, the weights of the neighboring points of the interpolation point in the spatiotemporal dimension can be calculated as the spatiotemporal interpolation weight, denoted as The neighboring points are known points (known pixels) that do not need to be interpolated. The weight depends on the distance between the interpolation point and the known point in time and space. The closer the distance, the greater the weight. The calculation formula is as follows: (1) In the above formula (1), is the point to be interpolated The kth neighbor of The weight of represents a total of m neighboring points, Represents a total of n moments in a long time series. Indicates the point to be interpolated and the kth neighbor The spatiotemporal covariance matrix of Indicates that within n moments, the interpolation point The sum of the spatiotemporal covariance matrices with m neighboring points.

[0050] Step 204: Calculate the spatiotemporal interpolation of the points to be interpolated based on the spatiotemporal interpolation weights to obtain an interpolated surface remote sensing image.

[0051] Finally, joint interpolation calculation is performed based on the spatiotemporal interpolation weights, and the spatiotemporal interpolation of each point to be interpolated is calculated based on the spatiotemporal interpolation weights of each point to be interpolated to obtain the interpolated surface remote sensing image. Space-time interpolation The calculation process is as follows formula (2): (2) In the above formula (2), is the spatiotemporal interpolation of the points to be interpolated, is the pixel observation value of the neighboring point to be interpolated, is the spatiotemporal interpolation weight of the point to be interpolated calculated by the above formula (1), and k represents the kth neighboring point of the point to be interpolated.

[0052] After calculating the spatiotemporal interpolation of each point to be interpolated, the point to be interpolated is used as the interpolation pixel in the low-resolution surface remote sensing image, and the corresponding spatiotemporal interpolation is used as the observation value of the corresponding interpolation pixel, thus forming the corresponding interpolation surface remote sensing image, which is output as the interpolation result.

[0053] The embodiment of the present invention adopts a spatiotemporal coupled interpolation method to simultaneously perform time-based and space-based interpolation processing on a long-time series of low-resolution surface remote sensing images. Compared with simply performing time interpolation and space interpolation separately, the spatiotemporal coupled interpolation can better capture the joint dynamic characteristics of space and time, making the interpolation results more accurate and consistent, enhancing the integrity of the image data and improving the resolution in the time and space dimensions.

[0054] In order to model the dynamic spatiotemporal changes of the earth's surface, in some embodiments, control equations for describing the spatiotemporal changes of the earth's surface are constructed based on interpolated surface remote sensing images. The specific construction process is described in detail below.

[0055] First, a candidate function library is constructed based on the longitude direction information, latitude direction information and time value of the pixel observation value in the interpolated surface remote sensing image. The pixel observation value in the interpolated surface remote sensing image, that is, the image pixel value in the interpolated surface remote sensing image, is denoted as , x represents longitude, y represents latitude, and t represents time. The three-dimensional representation in time and space is as follows: Figure 3 shown.

[0056] The candidate function library consists of the first function term and the second function term. The first function term includes: constant 1, pixel observation value , the square value of the pixel observation , the sine transformation value of the pixel observation , cosine transform value of pixel observation , written as: The second function term includes: the first-order partial derivative of the pixel observation value z with respect to the longitude direction x and the second-order partial derivatives , the first-order partial derivative of the pixel observation value z with respect to the latitude direction y and the second-order partial derivatives , the mixed partial derivatives of the pixel observation value z with respect to the longitude direction x and the latitude direction y , written as: .

[0057] like Figure 3 As shown, when constructing the second function term, for the interpolated surface remote sensing image, traverse each pixel, according to any pixel observation value , respectively calculate the first and second partial derivatives of x in the longitude direction, the first and second partial derivatives of y in the latitude direction, and the mixed partial derivatives of x in the longitude direction and y in the latitude direction. These partial derivatives represent the pixel observation values Information about changes in spatial position and orientation.

[0058] See also Figure 4 After constructing the candidate function library, the first function term and the second function term are combined to obtain multiple composite function terms. Here, combination means permutation and combination. Take one term from each of the two groups of function terms and multiply them to obtain all function terms. In other words, by taking any term from the first function term and the second function term and combining them, multiple composite function terms can be obtained. These function terms reflect the complex interaction between the observations and their derivatives. The candidate function library finally constructed contains 30 composite functions (obtained from all combinations of the two groups of function terms).

[0059] Finally, the control equation of the pixel in the interpolated surface remote sensing image is obtained by fitting the composite function terms. The candidate function library is constructed by interpolating the surface remote sensing image, and the function terms are combined to obtain multiple composite function terms. The final control equation is fitted to describe the spatiotemporal variation of the surface. The form of the control equation is as follows (3): (3) In the above formula (3), Represents the pixel observation value The rate of change over time, i.e. the time differential. , , , , , , , , , , , , , , , , , , , , , , , , , , , , , is the parameter to be fitted, representing the coefficient of each candidate function term, and Represents the pixel observation value and its square value. yes About longitude direction The first-order partial derivative of yes About latitude direction The first-order partial derivative of and They are The second-order partial derivatives in the longitude direction x and the latitude direction y, express About longitude direction The mixed partial derivatives of y in the latitudinal direction reflect the joint changes in space. The explanations of the remaining parameters can refer to the parameter explanations of the first and second function items in the above candidate function library, which will not be repeated here.

[0060] In an embodiment of the present invention, based on the interpolation of each pixel in the surface remote sensing image, the spatiotemporal change process of the surface is modeled, and the constructed control equation has a certain universality in the spatiotemporal range, that is, the control equation can be reused in adjacent or similar areas for spatiotemporal super-resolution modeling, without the need to relearn the equation for each local area in the image, which makes the spatiotemporal change modeling of the surface highly adaptable in a wide range of scenarios.

[0061] In some embodiments, Figure 4 As shown in the figure, after constructing the control equation, the control equation is solved. For the control equation of the pixels in the interpolated surface remote sensing image, the LASSO regression method is used to solve the control equation to obtain the differential equation used to calculate the high-resolution prediction image, which is explained in detail below.

[0062] First, the regression equation is constructed based on the control equation to interpolate the pixel observation values ​​in the surface remote sensing image. Substituting each function term in the control equation, we obtain a regression equation consisting of multiple candidate functions, expressed as the following formula (4): (4) In the above formula (4), The coordinates are The pixel at the location The differential of the moment, The observed value of the pixel at time The difference in the observed values ​​of the pixels at time instant, The coordinates are The pixel at the location The observed value at time, for The square value of The coordinates are The pixel at the location The square of the observed value at the time multiplied by its relative longitude and latitude direction The mixed partial derivatives of is the coefficient vector of the candidate function to be solved, corresponding to the weight of each candidate function, which is actually the parameter to be fitted in formula (3), obtained by merging similar terms.

[0063] like Figure 4 As shown, when solving the control equation, LASSO regression is first used to solve the control equation, that is, the coefficient vector composed of the weights of each candidate function in the regression equation is determined by LASSO regression. That is, the coefficient vector of the candidate function in the above formula (4) is solved , and when solving the coefficient vector w, the coefficient vector composed of the weights of each candidate function in the regression equation is determined by the LASSO regression method. According to the LASSO regression method, firstly, based on the coefficient vector Construct the loss function value of LASSO, and then determine the corresponding coefficient vector when the loss function value reaches the minimum value Therefore, through the fitting of LASSO regression, the sparse solution of the model can be obtained.

[0064] Determine the coefficient vector After obtaining the sparse solution of , the coefficient vector w is substituted into the regression equation to obtain the differential equation for calculating the high-resolution predicted image.

[0065] Assume that the solution is , substituting this solution into the regression equation of the above formula (4), we get the differential equation for calculating the high-resolution predicted image, which can be expressed as formula (5): (5) The explanation of the parameters in the above formula (5) can refer to the above formula (4), which will not be repeated here. According to formula (5), Only with , , as well as That is to say, the differential result of the observed value of the pixel is only related to the value of these function terms, so the above formula (5) can be used as a differential equation for calculating high-resolution predicted images.

[0066] Finally, the high-resolution predicted image is calculated by directly using differential equations to calculate the high-resolution predicted image at the specified time and spatial position. Here, the observed values ​​of the pixels in the surface remote sensing image are interpolated. Include specified time And the spatial location information of the pixel , so the observed value of each pixel in the interpolated surface remote sensing image can be directly By substituting them into the differential equation one by one, we can calculate the observed value of each pixel as the surface changes, thereby obtaining a high-resolution predicted image.

[0067] The embodiment of the present invention uses LASSO regression to solve the control equation, obtains a differential equation for calculating high-resolution prediction images, and proposes a method for modeling and solving the surface change process to calculate high-resolution prediction images. This image is not only more detailed, but also retains the spatiotemporal continuity of surface changes, avoiding artifacts or over-smoothing that may be generated in the interpolation process of traditional methods. In addition, the differential equation can accurately describe the laws of spatiotemporal changes in the surface, can maintain good computing power with less data support, and has a certain spatiotemporal universality, that is, the differential equation can be directly used to calculate high-resolution images in adjacent or similar spatiotemporal regions without the need to solve the equation again.

[0068] In some embodiments, the coefficient vector composed of the weights of each candidate function in the regression equation is determined by LASSO regression. The main step is to determine the coefficient vector based on the coefficient vector. Construct the loss function value of LASSO, and then determine the corresponding coefficient vector when the loss function value reaches the minimum value , detailed description is given below.

[0069] The first step is to construct the LASSO loss function, based on the pixel observation value z and pixel prediction observation value in the interpolated surface remote sensing image Calculate the residual sum of squares, denoted as RSS(w), expressed as Here the pixel predicted observation value is In the process of LASSO regression, the pixel observation value is calculated by solving the differential equation. In each LASSO regression process, the residual sum of squares between the pixel predicted observation value and the pixel original observation value z needs to be calculated to construct the LASSO loss function in order to continue the regression calculation.

[0070] Then add the regularization term to the loss function, where the weighted regularization value of the coefficient vector is determined, expressed as ,in, , and finally sum the weighted regularization value with the residual sum of squares to obtain the LASSO loss function of the coefficient vector, and find the minimum value, which is expressed as the following formula (6): (6) In the above formula (6), the weighted regularization value In Indicates parameters used to control The sparsity of the solution, The bigger the More coefficient terms are compressed to 0.

[0071] According to formula (6), the sparse coefficient vector w obtained when the LASSO loss function reaches the minimum value is used as the coefficient vector composed of the weights of each candidate function in the regression equation. For example, the solution is , in this sparse coefficient vector, most of the coefficients of w are 0, and only a few of them are significantly greater than 0. These non-zero coefficients are the key terms in the control equation, representing the characteristics that have a major impact on surface changes and can be used to construct the differential equation corresponding to the pixel.

[0072] The embodiment of the present invention uses the LASSO regression method to determine the coefficient vector composed of the weights of each candidate function in the regression equation, which can accurately predict the features that have a major impact on surface changes during the LASSO regression fitting process, so as to facilitate the subsequent direct calculation of high-resolution predicted images at a specified time and spatial position under these features.

[0073] In some embodiments, after the high-resolution predicted image is calculated, in order to further introduce spatial details, the high-resolution predicted image obtained based on the surface process modeling is optimized. Here, the high-resolution image is introduced, that is, the interpolated surface remote sensing image obtained by spatiotemporal coupling interpolation, and then the spatial detail features of the interpolated surface remote sensing image are integrated into the high-resolution predicted image to obtain a super-resolution surface remote sensing image, which is described in detail below.

[0074] First, the interpolated surface remote sensing image is corrected to obtain a high-resolution image to be introduced whose spatial spectral characteristics are consistent with the high-resolution predicted image. The correction process includes geometric correction and radiation correction of the image. Here, the spatial spectral characteristics are adjusted by geometric correction and radiation correction to ensure the consistency of the high-resolution predicted image and the interpolated surface remote sensing image in space and spectrum.

[0075] After geometric and radiometric correction, the high-resolution image needs to be color migrated, that is, the color features of the interpolated surface remote sensing image are integrated into the high-resolution predicted image. The embodiment of the present invention adopts a histogram matching method to perform color matching between the color histogram of the high-resolution predicted image and the color histogram of the high-resolution image to be introduced, thereby ensuring the consistency of the color features.

[0076] Next, the spectral response information of the high-resolution image to be introduced can be migrated to the high-resolution predicted image after color matching to obtain a super-resolution surface remote sensing image. Here, spectral reconstruction or spectral adjustment methods based on multispectral images can be used to perform spectral migration to ensure that the high-resolution image to be introduced and the high-resolution predicted image have similar spectral responses in the same spectral band.

[0077] In some embodiments, when the spectral response information is migrated, the principal component analysis method can also be used to extract other spatial detail features from the high-resolution predicted image. These spatial detail features generally also include edge features, texture features and other subtle features in the image. These spatial detail features are injected into the interpolated surface remote sensing image obtained by spatiotemporal coupling interpolation, thereby generating a higher resolution remote sensing image. This process involves image fusion, because direct injection may cause some inconsistencies in spectrum, color or texture, so it is also necessary to use the fusion step to ensure that the final image is consistent and natural in all aspects (including color, details, spectral response, etc.). Therefore, the embodiment of the present invention adopts a method based on frequency domain fusion, such as wavelet transform or Fourier transform, for high-resolution predicted images and interpolated surface remote sensing images, effectively combining high-frequency detail parts and low-frequency overall structure parts, and outputting the final super-resolution surface remote sensing image.

[0078] A visualization of the super-resolution process of low-resolution surface remote sensing images can be found in Figure 5 As shown in FIG, firstly, spatiotemporal coupled interpolation is performed on the low-resolution surface remote sensing image to obtain the interpolated surface remote sensing image, then the high-resolution predicted image is generated through the control equation, and finally, the spatial detail features are integrated to obtain the super-resolution surface remote sensing image. Figure 5 Shown in , , , , Image comparison effects of surface remote sensing images at low resolution and high resolution at five moments. The images at low resolution are low-resolution surface remote sensing images, and the images at high resolution are super-resolution surface remote sensing images.

[0079] The embodiment of the present invention combines the predicted image obtained by modeling the surface change process with the high-resolution interpolated surface remote sensing image, which can not only accurately represent the spatiotemporal changes of the surface, but also improve the temporal and spatial resolution. By introducing high-resolution images for multi-source data fusion, the spatial details of the predicted image are further improved, and the generation quality of super-resolution surface remote sensing images is improved.

[0080] The following is a description of the remote sensing image spatiotemporal super-resolution device based on surface process modeling provided by the present invention. The remote sensing image spatiotemporal super-resolution device based on surface process modeling described below and the remote sensing image spatiotemporal super-resolution method based on surface process modeling described above can be referenced to each other.

[0081] like Figure 6As shown, the remote sensing image spatiotemporal super-resolution device based on surface process modeling includes: an acquisition module 601, an interpolation module 602, a calculation module 603, and a fusion module 604. Specifically, the acquisition module 601 is used to acquire a long-time series of low-resolution surface remote sensing images; the interpolation module 602 is used to perform time-based and space-based interpolation processing on the low-resolution surface remote sensing images to obtain interpolated surface remote sensing images; the calculation module 603 is used to construct a control equation for describing the spatiotemporal changes of the surface based on the interpolated surface remote sensing images, and solve the control equation using the LASSO regression method to obtain a differential equation for calculating a high-resolution predicted image; the fusion module 604 is used to calculate a high-resolution predicted image of a specified time and spatial position according to the differential equation, and integrate the spatial detail features of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

[0082] It should be noted that the beneficial effects of the remote sensing image spatiotemporal super-resolution device based on surface process modeling here and the remote sensing image spatiotemporal super-resolution method based on surface process modeling mentioned above can correspond to each other, so the beneficial effects of the remote sensing image spatiotemporal super-resolution device based on surface process modeling will not be repeated here.

[0083] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the spatiotemporal super-resolution method of remote sensing images based on surface process modeling, the method comprising: obtaining a long-time series of low-resolution surface remote sensing images; performing time-based and space-based interpolation processing on the low-resolution surface remote sensing images to obtain interpolated surface remote sensing images; constructing a control equation for describing the spatiotemporal changes of the surface based on the interpolated surface remote sensing images, and solving the control equation using the LASSO regression method to obtain a differential equation for calculating a high-resolution predicted image; calculating a high-resolution predicted image of a specified time and spatial position according to the differential equation, and integrating the spatial detail features of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

[0084] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the remote sensing image spatiotemporal super-resolution method based on surface process modeling provided by the above-mentioned methods, and the method includes: obtaining a long-term series of low-resolution surface remote sensing images; performing time-based and space-based interpolation processing on the low-resolution surface remote sensing images to obtain interpolated surface remote sensing images; constructing a control equation for describing the spatiotemporal changes of the surface based on the interpolated surface remote sensing images, and solving the control equation using the LASSO regression method to obtain a differential equation for calculating a high-resolution predicted image; calculating a high-resolution predicted image of a specified time and spatial position according to the differential equation, and incorporating the spatial detail characteristics of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

[0086] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the spatiotemporal super-resolution method of remote sensing images based on surface process modeling provided by the above-mentioned methods, the method comprising: acquiring a long-time series of low-resolution surface remote sensing images; performing time-based and space-based interpolation processing on the low-resolution surface remote sensing images to obtain interpolated surface remote sensing images; constructing a control equation for describing the spatiotemporal changes of the surface based on the interpolated surface remote sensing images, and solving the control equation using the LASSO regression method to obtain a differential equation for calculating a high-resolution predicted image; calculating a high-resolution predicted image of a specified time and spatial position according to the differential equation, and incorporating the spatial detail features of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

[0087] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spatiotemporal super-resolution method for remote sensing images based on surface process modeling, characterized in that: include: Obtain long-term series of low-resolution surface remote sensing images; Performing time-based and space-based interpolation processing on the low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image; Based on the interpolated surface remote sensing image, a control equation for describing the spatiotemporal changes of the surface is constructed, and the control equation is solved by using a LASSO regression method to obtain a differential equation for calculating a high-resolution prediction image; A high-resolution predicted image of a specified time and spatial position is calculated according to the differential equation, and the spatial detail features of the interpolated surface remote sensing image are integrated into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

2. The spatiotemporal super-resolution method for remote sensing images based on surface process modeling according to claim 1, characterized in that: The step of performing time-based and space-based interpolation processing on the low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image includes: For any two pixels in the low-resolution surface remote sensing image, the spatial covariance corresponding to the two pixels is obtained by fitting the variogram in the Kriging interpolation; For any two moments in the long time series of the low-resolution surface remote sensing image, the time covariance corresponding to the two moments is obtained by fitting an autoregressive model; Taking the product of the spatial covariance and the temporal covariance as a spatiotemporal covariance matrix; Calculate the spatiotemporal interpolation weights of each neighboring point of the point to be interpolated in the low-resolution surface remote sensing image according to the spatiotemporal covariance matrix; The spatiotemporal interpolation of the points to be interpolated is calculated based on the spatiotemporal interpolation weights to obtain an interpolated surface remote sensing image.

3. The remote sensing image spatiotemporal super-resolution method based on surface process modeling according to claim 1, characterized in that: The control equations for describing the spatiotemporal changes of the surface are constructed based on the interpolated surface remote sensing images, including: A candidate function library is constructed according to the longitude direction information, latitude direction information and time value of the pixel observation value in the interpolated surface remote sensing image, wherein the candidate function library is composed of a first function term and a second function term, the first function term includes: a constant 1, the pixel observation value, the square value of the pixel observation value, the sine transform value of the pixel observation value, and the cosine transform value of the pixel observation value, and the second function term includes: the first-order partial derivative and the second-order partial derivative of the pixel observation value with respect to the longitude direction, the first-order partial derivative and the second-order partial derivative of the pixel observation value with respect to the latitude direction, and the mixed partial derivative of the pixel observation value with respect to the longitude direction and the latitude direction; Combining the first function term and the second function term to obtain a plurality of composite function terms; The control equation of the pixel in the interpolated surface remote sensing image is obtained by fitting the composite function term.

4. The remote sensing image spatiotemporal super-resolution method based on surface process modeling according to claim 1, characterized in that: The control equation is solved by LASSO regression to obtain a differential equation for calculating a high-resolution predicted image, including: Substituting the pixel observation value in the interpolated surface remote sensing image into each function term in the control equation to obtain a regression equation composed of multiple candidate functions; Determine the coefficient vector composed of the weights of each candidate function in the regression equation by LASSO regression method; Substituting the coefficient vector into the regression equation, a differential equation for calculating a high-resolution predicted image is obtained.

5. The remote sensing image spatiotemporal super-resolution method based on surface process modeling according to claim 4, characterized in that: The method of determining the coefficient vector composed of the weights of each candidate function in the regression equation by LASSO regression includes: Calculate the residual sum of squares based on the actual observed values ​​of pixels and the predicted observed values ​​of pixels in the interpolated surface remote sensing image; Determine a weighted regularization value of the coefficient vector, and sum the weighted regularization value with the residual sum of squares to obtain a LASSO loss function of the coefficient vector; The sparse coefficient vector obtained when the LASSO loss function achieves the minimum value is used as the coefficient vector composed of the weights of each candidate function in the regression equation.

6. The remote sensing image spatiotemporal super-resolution method based on surface process modeling according to claim 1, characterized in that: The step of integrating the spatial detail features of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image comprises: Correcting the interpolated surface remote sensing image to obtain a high-resolution image to be introduced whose spatial spectral characteristics are consistent with the high-resolution predicted image; Performing color matching between the color histogram of the high-resolution predicted image and the color histogram of the high-resolution image to be introduced; The spectral response information to be introduced into the high-resolution image is migrated to the high-resolution predicted image after color matching to obtain a super-resolution surface remote sensing image.

7. A remote sensing image spatiotemporal super-resolution device based on surface process modeling, characterized in that: include: The acquisition module is used to obtain long-term series of low-resolution surface remote sensing images; An interpolation module, used for performing time-based and space-based interpolation processing on the low-resolution surface remote sensing image to obtain an interpolated surface remote sensing image; A calculation module, used for constructing a control equation for describing the spatiotemporal changes of the surface based on the interpolated surface remote sensing image, and solving the control equation by LASSO regression to obtain a differential equation for calculating the high-resolution predicted image; The fusion module is used to calculate a high-resolution predicted image of a specified time and spatial position according to the differential equation, and integrate the spatial detail features of the interpolated surface remote sensing image into the high-resolution predicted image to obtain a super-resolution surface remote sensing image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the remote sensing image spatiotemporal super-resolution method based on surface process modeling as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the remote sensing image spatiotemporal super-resolution method based on surface process modeling as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the remote sensing image spatiotemporal super-resolution method based on surface process modeling as described in any one of claims 1 to 6 is implemented.

Images