Data fusion method and device

Abstract

The data fusion method comprises the following steps: dividing a target region into a plurality of sub-regions, obtaining a remote sensing image of each sub-region, and constructing a data set of each sub-region based on a median synthesis mode; performing primary processing on the data in each data set; the primary processing comprises time interpolation, cloud and abnormal value processing, upscaling and resampling; determining input image data of the model based on the processed data; the input image data comprises image data of two coarse resolutions, image data of the coarse resolution of a prediction date and image data of two high resolutions, and the image data of the coarse resolution has high time resolution and is obtained based on a true value synthesis mode; and based on a data fusion mode, determining high-resolution image data of a prediction date. Through the method, large-scale and large-range data fusion can be realized.

Description

Technical Field

[0001] The present invention relates to the field of spatial information technology, and in particular to a method and device for improving large-area remote sensing image data fusion based on synthesizing input data using a mathematical model. Background Art

[0002] In recent years, spatiotemporal data fusion has attracted widespread attention in the field of remote sensing research at home and abroad. Relevant researchers have been committed to exploring various computational methods to achieve fusion calculation of remote sensing images from different data sources. The current research methods can be mainly summarized into the following categories: methods based on spectral pixel decomposition, methods based on weighted functions, and methods based on deep learning. However, the current research methods are usually not suitable for large-scale and large-scale data fusion research. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, computing device, computer storage medium and product including a computer program for improving large-area data fusion by synthesizing input data based on a mathematical model, which can be applied to large-scale and large-range data fusion.

[0004] In a first aspect, an embodiment of the present application provides a data fusion method, comprising: dividing a target area into multiple sub-areas, acquiring a remote sensing image of each sub-area, and constructing a data set for each sub-area based on a median synthesis method; performing preliminary processing on the data in each data set respectively; the preliminary processing includes time interpolation, cloud and outlier processing, upscaling and resampling; based on the processed data, determining the input image data of the model; the input image data includes two coarse-resolution image data, coarse-resolution image data of a predicted date, and two high-resolution image data, wherein the coarse-resolution image data has a high temporal resolution, and the coarse-resolution image data is obtained based on a true value synthesis method; based on a data fusion method, determining the high-resolution image data of the predicted date.

[0005] In some possible implementations, high-resolution image data for a predicted date is determined based on data fusion, including: processing pixels in a window at different positions on an image by sliding a sliding window on the image at a certain step length; the processing includes: calculating spectral similarity of coarse-resolution image data, performing inverse distance weighting based on the spectral similarity, and determining a first weight; calculating a statistical relationship between coarse-resolution and high-resolution images, and determining a second weight; determining results of forward data prediction and backward data prediction based on the first weight and the second weight; calculating a time weight of the coarse-resolution image based on two input images at the coarse resolution and the coarse-resolution image acquired at the predicted date; and determining high-resolution image data for a predicted date based on the time weight and the results of forward data prediction and backward data prediction.

[0006] In some possible implementations, the input image data of the model is determined based on the processed data, including: using the processed data in a true value synthesis manner to generate an image without clouds and missing pixels for each season; based on a correlation analysis method, two images with the greatest correlation are selected from the images as the input image data of the model.

[0007] In some possible implementations, the spectral similarity of the coarse-resolution image data is calculated according to the following formula:

[0008]

[0009] In the formula, R 10 The spectral correlation coefficient between the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 1 within the sliding window, R 20 It represents the spectral correlation coefficient between the coarse-resolution image of the prediction date and the coarse-resolution image of the input date 2 within the sliding window, C0 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the prediction date within the sliding window, 1 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the input date 1 within the sliding window, C2 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the input date 2 within the sliding window, E represents the expected value, and D represents the variance.

[0010] In some possible implementations, the first weight is determined and calculated according to the following formula:

[0011]

[0012] Where W 10 The weight of the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 1 in the sliding window, W 20 Represents the weight of the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 2 within the sliding window.

[0013] In some possible implementations, the statistical relationship between the coarse resolution image and the high resolution image is calculated to determine the second weight, which is calculated according to the following formula:

[0014]

[0015] In the formula, K 10 K represents the weight of the coarse-resolution image of the predicted date and the input high-resolution image 1 in the sliding window. 20 represents the weight of the predicted date image in the sliding window and the input high-resolution image 2, n represents the total number of pixels in the sliding window, and F 1i Represents the i-th pixel of the high-resolution image 1 in the sliding window, F 2i Represents the i-th pixel of high-resolution image 2 in the sliding window, C 0i The i-th pixel in the sliding window of the coarse-resolution image representing the predicted date, Represents the mean of the high-resolution image input within the sliding window, Represents the mean of the coarse-resolution image of the predicted date within the sliding window.

[0016] In some possible implementations, the results of the forward data prediction and the backward data prediction are determined and calculated according to the following formula:

[0017] F 10 =F1+W 10 *K 10 (C0-C1)

[0018] F 20 =F2+W 20 *K 20 (C0-C2)

[0019] In the formula, F 10 Represents the forward data prediction results, F 20 Characterize the backward data prediction results.

[0020] In some possible implementations, the temporal weight of the coarse resolution image is calculated according to the following formula:

[0021]

[0022] Wherein, T1 represents the temporal weight of coarse resolution image 1, T2 represents the temporal weight of coarse resolution image 2, d1 represents the temporal distance between coarse resolution images and d1 = |C1-C0|, d2 represents the temporal distance between coarse resolution images and d2 = |C2-C0|.

[0023] In some possible implementations, the high-resolution image data for the predicted date is determined according to the following formula:

[0024] F0=T1*F 10 +T2*F 20 .

[0026] In a second aspect, an embodiment of the present application provides a data fusion device, comprising: a construction module, used to divide a target area into multiple sub-areas, obtain a remote sensing image of each sub-area, and construct a data set for each sub-area based on a median synthesis method; a processing module, used to perform preliminary processing on the data in each data set respectively; the processing module is also used to determine the input image data of the model based on the processed data; the input image data includes two coarse-resolution image data, coarse-resolution image data of a predicted date, and two high-resolution image data, the coarse-resolution image data having a high temporal resolution; wherein the coarse-resolution image data is obtained based on a true value synthesis method; the processing module is also used to determine the high-resolution image data of the predicted date based on a data fusion method.

[0027] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method described in any one of the first aspects.

[0028] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method described in any one of the first aspects is executed.

[0029] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program. When the computer program product runs on a processor, the processor executes the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0031] Figure 1 It is a flowchart of a data fusion method provided in an embodiment of the present application;

[0032] Figure 2 It is a structural schematic diagram of a data fusion device provided in an embodiment of the present application;

[0033] Figure 3 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0036] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0037] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0038] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0039] To facilitate the understanding of the embodiments of the present application, the following will further explain and illustrate specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0040] For the fusion of spatiotemporal data, in some existing technologies, such as the spatiotemporal adaptive reflectance fusion model in the method based on spectral pixel decomposition, its implementation scheme usually includes three steps: data preprocessing, fusion prediction and output image. In the data preprocessing stage, the images of different sensors need to be resampled to match the resolution. In the fusion prediction stage, the correlation between bands is considered, multi-band data is used for fusion, and similar pixels are selected according to the moving window. The output image stage is to output the fused data.

[0041] However, existing methods cannot handle large-scale and large-scale data fusion well, cannot make full use of time series data, and cannot exclude or utilize mixed pixels in images.

[0042] In view of this, the embodiment of the present application provides a data fusion method, which synthesizes input data based on a mathematical model, including block data synthesis, median synthesis and true value synthesis, as well as accuracy evaluation and reliability analysis of the fusion results of different input data. This method uses block method, median synthesis and true value synthesis to generate large-scale, large-range continuous surface reflectance data, improves the existing model, takes into account the existence of mixed pixels, and improves precision and accuracy.

[0043] For example, Figure 1 A flow chart of a data fusion method provided in an embodiment of the present application is shown. The method can be used to improve the spatiotemporal adaptive reflectance fusion model in the aforementioned method based on spectral pixel decomposition. The method adopts different mathematical models to synthesize data as input data of the spatiotemporal data fusion model. In order to study the generation algorithm of large-scale, large-range continuous surface reflectance data, mathematical operations are performed on all data in the study area as input data of the spatiotemporal data fusion model, and the spatiotemporal data fusion algorithm is run in multiple threads, thereby improving the production speed and accuracy of large-scale, large-range continuous spatiotemporal surface reflectance data. Figure 1 As shown, the method may include the following steps:

[0044] S11: Divide the target area into multiple sub-areas, obtain the remote sensing image of each sub-area, and construct a data set for each sub-area.

[0045] In this embodiment, the geographical area to be studied is called the target area, and the target area is divided into a plurality of sub-areas. The division of the sub-areas can be based on geographical factors such as its geographical location, topographical features, etc., can also be based on administrative boundaries, or can be based on other relevant geographical and environmental factors, which are not specifically limited here. Remote sensing data of each sub-area is obtained. Remote sensing data can include image data of the same sub-area at different times, different sensors, or different resolutions. Dividing a large area into a plurality of sub-areas can improve the efficiency and quality of remote sensing data processing, and facilitate the synthesis and analysis of data. Since the time difference between the input data and the image to be predicted has a greater impact on the accuracy of the predicted image, a block data synthesis method can be adopted, and the block data can be synthesized and processed respectively according to different time periods. In addition, since the scheme is calculated by a computing device, the configuration level of the computing device also directly affects the efficiency of the scheme calculation. By means of block processing, the configuration dependence on the computing device is also reduced. In some examples, the block can adopt a fishing net (FISHNET) method.

[0046] In addition, a high-resolution image has a small coverage area. When processing large-area data, it is usually necessary to input multiple views. In order to solve the problem of spatial inconsistency of multiple view data acquired at different times, a data processing method based on median synthesis is adopted. Median synthesis is a statistical method that uses the median of the data to generate a data set representing the entire time period. This method does not require cloud cover screening, but uses all available remote sensing image data as input.

[0047] S12: Perform preliminary processing on the data in each data set respectively.

[0048] In this embodiment, after constructing the data set, the data in the data set can be preliminarily processed to improve the accuracy of the surface reflectivity prediction. Specifically, the high temporal resolution data of the prediction date needs to be preprocessed (high temporal resolution data, whose spatial resolution is low). Preprocessing can eliminate the influence of clouds and outliers and ensure that its resolution is consistent with that of the high spatial resolution data.

[0049] In some possible embodiments, preliminary processing is performed on the data in the data set, which may include: time interpolation, cloud and outlier processing, upscaling and resampling, etc.

[0050] Specifically, for time interpolation, the high time resolution reflectivity data within 30 days can be used as auxiliary data to perform time interpolation on the high time resolution reflectivity data of the predicted date. The high time resolution reflectivity data of the two most recent phases are used to perform linear interpolation in proportion to the time distance to obtain complete high time resolution reflectivity data.

[0051] For cloud and outlier processing, interpolated data can be used to replace clouds and outliers in the reflectance data of the predicted date.

[0052] For upscaling and resampling, high temporal resolution reflectivity data can be upscaled and resampled using bilinear interpolation to ensure resolution consistency, smooth the data, and generate better continuity of results. The bilinear interpolation method uses the reflectivity values ​​of the four nearest pixels and performs linear interpolation in proportion to the spatial distance to obtain the reflectivity value of the new pixel.

[0053] S13: Determine input image data of the model based on the processed data.

[0054] In this embodiment, the declouded data is synthesized using the true value data method, and the image with the most effective image elements in each season is selected as the reference image, and the data of other images are used to fill in the missing parts. According to the seasonal changes in vegetation coverage, the original input data of each year is divided into four time periods, namely January-March, April-June, July-September, and October-December. The seasonal data of each sub-region in different seasons is determined by true value synthesis. True value synthesis is a method that uses time series analysis to synthesize data from the same season, taking into account the periodicity and trend of the data, to generate a data set that can represent the corresponding season, namely seasonal data.

[0055] Specifically, determining seasonal data of each sub-region in different seasons may include seasonal data synthesis, reference image selection, and data filling.

[0056] Among them, seasonal data synthesis divides the original input data of each year into four time periods according to the seasonal changes in vegetation cover. Truth synthesis is a method that uses time series analysis to synthesize data from the same season to generate a data set representing that season, taking into account the periodicity and trend of the data.

[0057] The benchmark image selection can screen images with cloud coverage below 20% in each season, count the number of valid pixels, and evaluate the quality of the data. The image with the lowest cloud coverage, the highest data integrity, and the lowest noise is selected as the benchmark image for that season.

[0058] For data filling, the cloud-blocked or data-missing areas in each candidate image of the same season can be filled with the cloud-free and non-missing pixel values ​​of the same position in other candidate images of the same season to obtain a complete and cloud-free candidate image. In this way, a high-quality image with no cloud or missing data can be obtained in each season.

[0059] After the seasonal data were determined, the two most relevant images were selected as part of the input data for the model.

[0060] In some possible embodiments, correlation is determined by spectral similarity.

[0061] Specifically, after obtaining an image data of each season, the overall correlation analysis is used to select two images with the greatest correlation. The overall correlation analysis can be performed using spectral similarity, and the formula is as follows:

[0062]

[0063] In the formula, R i represents the spectral correlation coefficient between the i-th pixel in the predicted date image pixel and the seasonal candidate image pixel, E represents the expected value, and F i Represents the spectral vector containing the reflectance of the image pixel in each band of the i-th prediction date, C i The spectral vector representing the reflectance of the seasonal candidate image pixel in each band, and D represents the variance.

[0064] S14: Determine the high-resolution image data of the predicted date based on the data fusion method.

[0065] In this embodiment, a high-resolution image of a certain date can be predicted based on the input coarse-resolution image and two high-resolution images. The coarse-resolution image includes two coarse-resolution images with the greatest correlation and a coarse-resolution image of the predicted date, and the high-resolution image and the coarse-resolution image have the same date. When performing data fusion, the pixels in the window at different positions on the image can be processed by sliding the sliding window on the image according to a certain step length. The sliding window size can be 20*20. In the process of processing, the spectral similarity of the coarse-resolution image data can be calculated first, and the inverse distance weighting is performed based on the spectral similarity to determine the first weight. Then the statistical relationship between the coarse-resolution and high-resolution images is calculated to determine the second weight. Based on the first weight and the second weight, the results of the forward data prediction and the backward data prediction are determined. Then, based on the two input images at the coarse resolution and the coarse-resolution image obtained at the predicted date, the time weight between the coarse-resolution images is calculated. Based on the time weight and the two high-resolution input images, the high-resolution image of the predicted date can be obtained.

[0066] In some possible embodiments, the spectral similarity of the coarse-resolution image data is calculated according to the following formula:

[0067]

[0068] In the formula, R 10The spectral correlation coefficient between the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 1 within the sliding window, R 20 It represents the spectral correlation coefficient between the coarse-resolution image of the prediction date and the coarse-resolution image of the input date 2 within the sliding window, C0 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the prediction date within the sliding window, C1 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the input date 1 within the sliding window, C2 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the input date 2 within the sliding window, E represents the expected value, and D represents the variance.

[0069] In some possible embodiments, inverse distance weighting is performed based on spectral similarity, and the first weight is determined according to the following formula:

[0070]

[0071] Where W 10 The weight of the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 1 in the sliding window, W 20 Represents the weight of the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 2 within the sliding window.

[0072] In some possible embodiments, the statistical relationship between the coarse resolution image and the high resolution image is calculated, and the second weight is determined according to the following formula:

[0073]

[0074] In the formula, K 10 K represents the weight of the coarse-resolution image of the predicted date and the input high-resolution image 1 in the sliding window. 20 represents the weight of the predicted date image in the sliding window and the input high-resolution image 2, n represents the total number of pixels in the sliding window, and F 1i Represents the i-th pixel of the high-resolution image 1 in the sliding window, F 2i Represents the i-th pixel of high-resolution image 2 in the sliding window, C 0i The i-th pixel in the sliding window of the coarse-resolution image representing the predicted date, Represents the mean of the high-resolution image input within the sliding window, Represents the mean of the coarse-resolution image of the predicted date within the sliding window.

[0075] In some possible embodiments, based on the first weight and the second weight, the results of the forward data prediction and the backward data prediction are determined to be calculated according to the following formula:

[0076] F 10 =F1+W 10 *K10 (C0-C1)

[0077] F 20 =F2+W 20 *K 20 (C0-C2)

[0078] In the formula, F 10 Represents the forward data prediction results, F 20 Characterize the backward data prediction results.

[0079] In some possible embodiments, the time weight between coarse resolution images is calculated according to the following formula:

[0080]

[0081] Wherein, T1 represents the temporal weight of coarse resolution image 1, T2 represents the temporal weight of coarse resolution image 2, d1 represents the temporal distance between coarse resolution images and d1 = |C1-C0|, d2 represents the temporal distance between coarse resolution images and d2 = |C2-C0|.

[0082] In some possible embodiments, the high-resolution image of the predicted date is calculated according to the following formula:

[0083] F0=T1*F 10 +T2*F 20

[0084] In some possible embodiments, in order to verify the high-resolution image data of the predicted date, the method may further include step S15: verifying the high-resolution image data.

[0085] In this embodiment, in order to verify the accuracy and reliability of high-resolution data products predicted by existing basic data, a comprehensive accuracy analysis was conducted on the high-resolution image data of the predicted date and compared with the high-quality surface reflectance data. Two main evaluation indicators were used, namely, Root Mean Square Error (RMSE) and Bias.

[0086] Among them, the root mean square error is used as an evaluation index to measure the degree of deviation between the predicted value and the observed value. The calculation formula is as follows:

[0087]

[0088] In the formula, n represents the number of pixels in the image, y i Represents the true value of the image, Characterize the predicted value of the image.

[0089] Bias is used to evaluate the average difference or degree of deviation between the model's predicted value and the actual true value, and can be used to evaluate the accuracy and degree of deviation of the model. The calculation formula is as follows:

[0090]

[0091] The above is an introduction to the data synthesis method provided in the embodiment of the present application. Through block processing, each block pre-processes the high temporal resolution data, including cloud removal and outlier processing, and then unifies the resolution through bilinear interpolation. Next, the spectral similarity and statistical relationship between the images are calculated using a sliding window, and the high-resolution image is predicted in combination with the time weight. Finally, the accuracy of the predicted data is verified through precision analysis, such as root mean square error and deviation. This method divides the study area into blocks, and performs true value synthesis according to the surface reflectivity and median synthesis according to the time range. It uses mixed pixels in the image and can be applied to large-scale and large-scale data fusion.

[0092] In order to analyze the feasibility of this method, this method was implemented on a computer equipped with an Intel second-generation Core i3-2100@3.10GHz dual-core processor, 4G memory, and Nvidia GeForce GT 630 graphics card. Taking MODIS and Landsat data as examples, data with different temporal and spatial resolutions were processed, and then the fusion effect of synthesizing large-area data based on mathematical models was compared. Among them, MODIS data is image data with low spatial resolution taken once a day, and Landsat data is image data with high resolution taken once every 16 days.

[0093] The Haihe River Basin is divided into 30 sub-areas by the data block method, and the true value synthesis is calculated based on this partition. Each sub-area is divided into 4 pairs of median synthetic remote sensing image data according to the above. Data fusion calculations are performed in each area to obtain the results, and finally spliced ​​to synthesize the continuous surface reflectance results of the entire Haihe River Basin.

[0094] According to the results of median synthesis, there are a total of 4 sets of median data, while the input data of this method has a maximum of two pairs of data. The selection of suitable data pairs as input data is highly correlated with the accuracy and reliability of the data fusion results. According to the MODIS image of the predicted date and the MODIS-M1, MODIS-M2, MODIS-M3, and MODIS-M4 synthesized by the previous median, the correlation coefficients are calculated for the blue, green, red, and near-infrared bands, and then the correlation coefficients of these four bands are summed. The two remote sensing images with the largest sum of correlation coefficients are selected as the input data of this algorithm, namely Landsat-Mi, MODIS-Mi, Landsat-Mj, and MODIS-Mj. The reflectance result of the true value synthesis obtained by the above two pairs of input data and the input data of the predicted date is EM. April 13, 2020 and August 28, 2020 were selected as test results. These two dates correspond to spring and summer, respectively, which can reflect the seasonal changes in the Haihe River Basin. The EM results of these two dates are evaluated and analyzed to verify the effectiveness and applicability of the data fusion method proposed in this patent.

[0095] Data fusion calculation is performed on each block area to obtain a spatiotemporal continuous surface reflectance product with a resolution of 30 meters, which effectively improves the resolution of the predicted data.

[0096] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any feature of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0097] Based on the method in the above embodiment, the present application also provides a data synthesis device

[0098] For example, Figure 2 A data synthesis device is shown, which is deployed in a computing device. The data synthesis device 600 includes: a construction module 201 and a processing module 202 .

[0099] The construction module 201 is used to divide the target area into multiple sub-areas, obtain the remote sensing image of each sub-area, and construct the data set of each sub-area based on the median synthesis method;

[0100] The processing module 202 is used to perform preliminary processing on the data in each data set respectively;

[0101] The processing module 202 is further used to determine the input image data of the model based on the processed data; the input image data includes two coarse-resolution image data, coarse-resolution image data of the predicted date, and two high-resolution image data, the coarse-resolution image data has high temporal resolution; wherein the coarse-resolution image data is obtained based on a true value synthesis method;

[0102] The processing module 202 is further used to determine the high-resolution image data of the predicted date based on data fusion.

[0103] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0104] The present application also provides a computing device 300. Figure 3 As shown, computing device 300 includes: bus 302, processor 304, memory 306 and communication interface 308. Processor 304, memory 306 and communication interface 308 communicate through bus 302. Computing device 300 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 300.

[0105] The bus 302 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus 304 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 304 may include a path for transmitting information between various components of the computing device 300 (eg, the memory 306, the processor 304, and the communication interface 308).

[0106] The processor 304 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0107] The memory 306 may include a volatile memory, such as a random access memory (RAM). The processor 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0108] The memory 306 stores executable program codes, and the processor 304 executes the executable program codes to respectively implement the functions of the aforementioned construction module 201 and the processing module 202, thereby implementing all or part of the steps of the method in the above embodiment. That is, the memory 306 stores instructions for executing all or part of the steps in the method in the above embodiment.

[0109] Alternatively, the memory 306 stores executable codes, and the processor 304 executes the executable codes to respectively implement the functions of the aforementioned data synthesis device 200, thereby implementing all or part of the steps in the above-mentioned embodiment method. That is, the memory 306 stores instructions for executing all or part of the steps in the above-mentioned embodiment method.

[0110] The communication interface 308 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 300 and other devices or a communication network.

[0111] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0112] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0113] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0114] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0115] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0116] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. A data fusion method, characterized in that: The method comprises: Divide the target area into multiple sub-areas, obtain the remote sensing image of each sub-area, and construct the data set of each sub-area based on the median synthesis method; Perform preliminary processing on the data in each dataset respectively; the preliminary processing includes time interpolation, cloud and outlier processing, upscaling and resampling; Based on the processed data, the input image data of the model is determined; the input image data includes two coarse-resolution image data, coarse-resolution image data of the predicted date, and two high-resolution image data, wherein the coarse-resolution image data has high temporal resolution, and the coarse-resolution image data is obtained based on a true value synthesis method; Based on data fusion, high-resolution image data for the predicted date is determined.

2. The method according to claim 1, characterized in that The method of determining the high-resolution image data of the predicted date based on data fusion includes: processing pixels in windows at different positions on the image by sliding the sliding window on the image according to a certain step length; the processing includes: Calculating spectral similarity of the coarse-resolution image data, performing inverse distance weighting based on the spectral similarity, and determining a first weight; Calculate the statistical relationship between the coarse resolution and high resolution images to determine the second weight; Determining results of forward data prediction and backward data prediction based on the first weight and the second weight; Based on the two input images at coarse resolution and the coarse resolution image acquired at the prediction date, the temporal weight of the coarse resolution image is calculated; Based on the time weight and the results of the forward data prediction and the backward data prediction, high-resolution image data of the predicted date is determined.

3. The method according to claim 1, characterized in that The step of determining the input image data of the model based on the processed data comprises: The processed data are synthesized using true value to generate a cloud-free and pixel-missing image for each season; Based on the correlation analysis method, two images with the greatest correlation are selected from the images as input image data of the model.

4. The method according to claim 2, characterized in that: The spectral similarity of the coarse resolution image data is calculated according to the following formula: In the formula, R 10 The spectral correlation coefficient between the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 1 within the sliding window, R 20 It represents the spectral correlation coefficient between the coarse-resolution image of the prediction date and the coarse-resolution image of the input date 2 within the sliding window, C0 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the prediction date within the sliding window, C1 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the input date 1 within the sliding window, C2 represents the spectral vector of the reflectance of each band of the coarse-resolution image of the input date 2 within the sliding window, E represents the expected value, and D represents the variance.

5. The method according to claim 2, characterized in that: The first weight is determined by calculating according to the following formula: Where W 10 The weight of the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 1 in the sliding window, W 20 Represents the weight of the coarse-resolution image of the predicted date and the coarse-resolution image of the input date 2 within the sliding window.

6. The method according to claim 2, characterized in that The statistical relationship between the coarse resolution and the high resolution image is calculated to determine the second weight, which is calculated according to the following formula: In the formula, K 10 K represents the weight of the coarse-resolution image of the predicted date and the input high-resolution image 1 in the sliding window. 20 represents the weight of the predicted date image in the sliding window and the input high-resolution image 2, n represents the total number of pixels in the sliding window, and F 1i Represents the i-th pixel of the high-resolution image 1 in the sliding window, F 2i Represents the i-th pixel of high-resolution image 2 in the sliding window, C 0i The i-th pixel in the sliding window of the coarse-resolution image representing the predicted date, Represents the mean of the high-resolution image input within the sliding window, Represents the mean of the coarse-resolution image of the predicted date within the sliding window.

7. The method according to claim 2, characterized in that The results of determining the forward data prediction and the backward data prediction are calculated according to the following formula: F 10 =F1+W 10 *K 10 (C0-C1) F 20 =F2+W 20 *K 20 (C0-C2) In the formula, F 10 Represents the forward data prediction results, F 20 Characterize the backward data prediction results.

8. The method according to claim 2, characterized in that: The time weight of the coarse resolution image is calculated according to the following formula: Wherein, T1 represents the temporal weight of coarse resolution image 1, T2 represents the temporal weight of coarse resolution image 2, d1 represents the temporal distance between coarse resolution images and d1 = |C1-C0|, d2 represents the temporal distance between coarse resolution images and d2 = |C2-C0|.

9. The method according to claim 2, characterized in that: The high-resolution image data for determining the predicted date is calculated according to the following formula: F0=T1*F 10 +T2*F 20 。 10. A data fusion device, characterized in that: The device comprises: A construction module is used to divide the target area into multiple sub-areas, obtain the remote sensing image of each sub-area, and construct the data set of each sub-area based on the median synthesis method; A processing module, used for performing preliminary processing on the data in each data set; The processing module is further used to determine the input image data of the model based on the processed data; the input image data includes two coarse-resolution image data, coarse-resolution image data of the predicted date, and two high-resolution image data, the coarse-resolution image data has high temporal resolution; wherein the coarse-resolution image data is obtained based on a true value synthesis method; The processing module is also used to determine the high-resolution image data of the predicted date based on data fusion.

Images