A multi-source remote sensing image fusion method, system and device

Abstract

This invention discloses a method, system, and apparatus for multi-source remote sensing image fusion, relating to the field of remote sensing image processing. The scheme includes acquiring multiple original remote sensing images and performing standard preprocessing to obtain standard remote sensing images; processing these images according to a preset downsampling strategy to obtain degraded remote sensing images at a first spatial resolution; performing adaptive robust smoothing processing on these images using a temporal robustness processing strategy to obtain robust remote sensing images corresponding to each time point and at the first spatial resolution; and performing upsampling processing on these images according to a preset fusion strategy to obtain a fused remote sensing image at a second spatial resolution. This scheme ensures geometric stability and spectral physical consistency through standard preprocessing, and reliably achieves robust processing, adaptive smoothing, and information preservation of degraded remote sensing images through a temporal robustness processing strategy, improving the continuity and consistency of robust remote sensing images over long time scales, and ultimately determining the fused remote sensing image, which is beneficial for anomaly monitoring.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method, system and apparatus for multi-source remote sensing image fusion. Background Technology

[0002] With the continuous development of optical remote sensing satellite technology, remote sensing images with different spatial resolutions, temporal resolutions, and spectral configurations are widely used in various dynamic monitoring scenarios such as land use or land cover. However, under the existing remote sensing imaging system, different types of remote sensing sensors generally face the technical challenge of balancing spatial resolution, temporal replay capability, and spectral configuration. On the one hand, high spatial resolution remote sensing images usually rely on imaging systems with small imaging swaths and long orbital revisit periods, making it difficult to achieve high temporal frequency and continuous stable observations at the regional scale. On the other hand, although medium-resolution or medium-to-high spatial resolution remote sensing images have certain advantages in terms of temporal coverage capability, their observation results are easily affected by factors such as cloud cover, shadow occlusion, and changes in atmospheric conditions, resulting in problems such as missing observations, abnormal fluctuations, and irregular sampling in the time series, which reduces the reliability of time series analysis and change monitoring.

[0003] To alleviate the limitations of single remote sensing sensors in terms of spatiotemporal resolution, some related technologies have proposed solutions for fusing multi-source remote sensing data, such as cross-sensor image fusion methods based on single or a small number of temporal phases. These methods achieve cross-domain mapping by performing geometric registration and histogram matching on the images. However, in essence, the fusion results of current technical solutions are still constrained by factors such as discontinuous original observation time, frequent cloud interference, and limited spatial resolution, making it difficult to form stable, continuous remote sensing image sequences with high spatial detail representation capabilities over long time scales.

[0004] Therefore, how to provide a multi-source remote sensing image fusion scheme to effectively integrate remote sensing images with different spatial resolutions and imaging sources, thereby constructing a continuous, stable, and robust remote sensing image sequence against cloud interference over a long period of time, is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a method, system and apparatus for multi-source remote sensing image fusion, which can achieve adaptive smoothing and information preservation of remote sensing images when there are spatial resolution differences, inconsistent imaging conditions and irregular sampling times among the original remote sensing images from multiple sources, so as to suppress the influence of cloud cover and shadow interference, improve the continuity and consistency of remote sensing images over a long time scale, and realize cross-sensor fusion.

[0006] To address the aforementioned technical problems, this application provides a multi-source remote sensing image fusion method, comprising:

[0007] Multiple raw remote sensing images were acquired from the first and second remote sensing devices, respectively.

[0008] The original remote sensing image is subjected to standard preprocessing operations to obtain a standard remote sensing image;

[0009] The standard remote sensing image is processed according to a preset downsampling strategy to obtain a degraded remote sensing image with a spatial resolution of the first spatial resolution.

[0010] A time axis for smoothing observations is pre-established, and the time axis includes various time points;

[0011] An adaptive robust smoothing process is applied to the degraded remote sensing images in the time series using a time series robustness processing strategy to obtain robust remote sensing images with the first spatial resolution corresponding to each time point.

[0012] The robust remote sensing images are upsampled according to a preset fusion strategy to obtain fused remote sensing images with a second spatial resolution corresponding to each time point, wherein the second spatial resolution is higher than the first spatial resolution.

[0013] Furthermore, the standard remote sensing image is processed according to a preset downsampling strategy to obtain a degraded remote sensing image with a spatial resolution of the first spatial resolution, including:

[0014] The lowest spatial resolution among the aforementioned standard remote sensing images is determined as the first spatial resolution;

[0015] The standard remote sensing image with the first spatial resolution is determined to be the degraded remote sensing image;

[0016] Using a spatial degradation model, standard remote sensing images with a spatial resolution greater than the first spatial resolution are downsampled to obtain degraded remote sensing images mapped to the first spatial resolution.

[0017] Furthermore, the original remote sensing image is a remote sensing image that characterizes the land conditions in the target area;

[0018] After obtaining the fused remote sensing images with the second spatial resolution corresponding to each of the aforementioned time points, the method further includes:

[0019] The monitoring results determine whether there are any abnormalities in the land under the target area based on the fused remote sensing imagery.

[0020] Furthermore, the step of obtaining robust remote sensing images of the first spatial resolution corresponding to each of the aforementioned time points includes:

[0021] Using each of the aforementioned time points as the center of the window, perform the following steps:

[0022] The reference sliding time window is determined based on the window center and the preset sliding size;

[0023] The set of remote sensing images that satisfy the current reference sliding time window is determined based on the timestamps of each of the degraded remote sensing images.

[0024] When it is determined that the remote sensing image set in the window includes no less than X degraded remote sensing images without cloud cover, the current reference sliding time window is determined to be the optimal sliding time window, where X is an integer greater than 1;

[0025] When it is determined that the remote sensing image set in the window includes fewer than X degraded remote sensing images without cloud cover, the current reference sliding time window is expanded to obtain a new reference sliding time window. This process continues until it is determined that the remote sensing image set in the window corresponding to the new reference sliding time window includes no fewer than X degraded remote sensing images without cloud cover. At this point, the reference sliding time window is determined to be the optimal sliding time window.

[0026] For the degraded remote sensing image under the optimal sliding time window, a weighted average is uniformly applied to each remote sensing image band by band and pixel by pixel to obtain the predicted image for each band. Then, the predicted images for each band are combined to obtain the robust remote sensing image corresponding to the center of the window.

[0027] Furthermore, the original remote sensing image is a remote sensing image characterizing the land conditions under the target area; the step of obtaining robust remote sensing images with the first spatial resolution corresponding to each of the aforementioned time points includes:

[0028] Corresponding to the time axis, all degraded remote sensing images in the time series are used as input items and fed into a pre-trained fitting smoothing model for inference, so as to obtain robust remote sensing images with the first spatial resolution corresponding to each time point in the time axis.

[0029] The training steps of the fitted smoothing model include:

[0030] The fitting smoothing model is trained based on the first dataset and the multispectral joint consistency constraint function to adjust the parameters of the fitting smoothing model until the first preset training termination condition is reached.

[0031] The multispectral joint consistency constraint function is:

[0032] ;

[0033] This corresponds to the total loss at the first time point in the first dataset. As the first weight, As the second weight, As the third weight, The first pixel-level mean square error is determined based on the real remote sensing image in the first dataset corresponding to the first time point and the first predicted remote sensing image obtained by inference using the fitted smoothing model. This represents the first spectral angle mapping loss, and is determined based on the real remote sensing image and the first predicted remote sensing image. This represents the first vegetation index loss, and is determined based on the actual remote sensing image and the first predicted remote sensing image.

[0034] Furthermore, the original remote sensing image is a remote sensing image characterizing the land conditions in the target area; the step of obtaining the fused remote sensing image with a second spatial resolution corresponding to each of the aforementioned time points includes:

[0035] Using a pre-trained deep learning model, the robust remote sensing images are upsampled to obtain fused remote sensing images with a second spatial resolution corresponding to each time point.

[0036] The training steps of the deep learning model include:

[0037] The deep learning model is trained based on the second dataset and the first loss function to adjust the parameters of the deep learning model until the second preset training termination condition is reached; the second dataset includes multiple image pairs, each image pair including a training remote sensing image at a second spatial resolution corresponding to a second time point, and a training remote sensing image at a first spatial resolution.

[0038] The first loss function is:

[0039] ;

[0040] in, This corresponds to the total loss at the second time point in the second dataset. As the fourth weight, As the fifth weight, It is the sixth weight. The second pixel-level mean square error is determined based on the training remote sensing image at the second time point with the second spatial resolution and the second predicted remote sensing image obtained by reasoning using the deep learning model. The spatial resolution of the predicted remote sensing image is the second spatial resolution. The second spectral angle mapping loss is represented and determined based on the training remote sensing image and the second predicted remote sensing image at the second spatial resolution. The second vegetation index loss is represented and determined based on the training remote sensing image and the second predicted remote sensing image at the second spatial resolution.

[0041] Furthermore, after obtaining the predicted remote sensing image using the deep learning model, the method further includes:

[0042] The second predicted remote sensing image is processed according to the preset downsampling strategy to obtain the predicted degraded remote sensing image.

[0043] Based on the predicted degraded remote sensing image and the training remote sensing image with the first spatial resolution, the corresponding third pixel-level mean square error, third spectral angle mapping loss, and third vegetation index loss are determined, and then weighted to obtain the reversible degradation consistency loss.

[0044] Determine the reversible degradation consistency loss and The weighted sum is the comprehensive loss corresponding to the second time point in the second dataset.

[0045] Furthermore, after obtaining the predicted remote sensing image using the deep learning model, the method further includes:

[0046] The training remote sensing image with the first spatial resolution is subjected to spectral unmixing description processing to obtain the corresponding first description information;

[0047] The second predicted remote sensing image is subjected to spectral unmixing description processing to obtain the corresponding second description information;

[0048] The spectral unmixing constraint loss is determined based on the first description information and the second description information;

[0049] Determine the reversible degradation consistency loss and The weighted sum is the comprehensive loss corresponding to the second time point in the second dataset, including:

[0050] Determine the spectral unmixing constraint loss, the reversible degradation consistency loss, and... The weighted sum is the comprehensive loss corresponding to the second time point in the second dataset.

[0051] To address the aforementioned technical problems, the present invention also provides a multi-source remote sensing image fusion system, comprising:

[0052] The acquisition module is used to acquire multiple raw remote sensing images from the first remote sensing detection device and the second remote sensing detection device, respectively.

[0053] The preprocessing module is used to perform standard preprocessing operations on the original remote sensing image to obtain a standard remote sensing image.

[0054] The downsampling degradation module is used to process the standard remote sensing image according to a preset downsampling strategy to obtain a degraded remote sensing image with a spatial resolution of the first spatial resolution.

[0055] A time axis creation module is used to pre-create a time axis for smoothed observation, which includes various time points;

[0056] The temporal robustness processing module is used to perform adaptive robust smoothing processing on the degraded remote sensing images in the temporal sequence using a temporal robustness processing strategy, so as to obtain robust remote sensing images with the first spatial resolution corresponding to each time point.

[0057] The upsampling fusion module is used to upsample the robust remote sensing image according to a preset fusion strategy to obtain a fused remote sensing image with a second spatial resolution corresponding to each time point, wherein the second spatial resolution is higher than the first spatial resolution.

[0058] To address the aforementioned technical problems, the present invention also provides a multi-source remote sensing image fusion device, comprising:

[0059] Memory, used to store computer programs;

[0060] A processor is used to execute the computer program to implement the steps of the multi-source remote sensing image fusion method as described above.

[0061] The beneficial effects of this invention are as follows:

[0062] This application provides a method, system, and apparatus for multi-source remote sensing image fusion. The scheme includes acquiring multiple raw remote sensing images from a first remote sensing detection device and a second remote sensing detection device; performing standard preprocessing operations on the raw remote sensing images to obtain standard remote sensing images; processing the standard remote sensing images according to a preset downsampling strategy to obtain degraded remote sensing images with a first spatial resolution; pre-establishing a time axis for smoothing observations, including various time points; using a time-series robustness processing strategy to perform adaptive robust smoothing processing on the degraded remote sensing images of the time series to obtain robust remote sensing images with a first spatial resolution corresponding to each time point; and upsampling the robust remote sensing images according to a preset fusion strategy to obtain fused remote sensing images with a second spatial resolution corresponding to each time point, where the second spatial resolution is higher than the first spatial resolution. It is evident that this scheme can ensure geometric stability and spectral physical consistency through standard preprocessing operations, even when there are spatial resolution differences, inconsistent imaging conditions, and irregular sampling times among multiple original remote sensing images. Through a time-series robustness processing strategy, it can reliably achieve robust processing, adaptive smoothing, and information preservation of remote sensing images with degraded first spatial resolution in a time-series format, thereby suppressing the effects of cloud cover and shadow interference and improving the continuity and consistency of robust remote sensing images with first spatial resolution over long time scales. Finally, a preset fusion strategy is used to determine the fused remote sensing images with second spatial resolution corresponding to each time point, realizing cross-sensor fusion, which is beneficial for anomaly monitoring and identification in practical applications.

[0063] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0064] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0065] Figure 1 A flowchart of a multi-source remote sensing image fusion method provided by the present invention;

[0066] Figure 2 This is a schematic diagram of the structure of a multi-source remote sensing image fusion system provided by the present invention;

[0067] Figure 3 This is a schematic diagram of the structure of a multi-source remote sensing image fusion device provided by the present invention. Detailed Implementation

[0068] The core of this invention is to provide a method, system, and apparatus for multi-source remote sensing image fusion, which can achieve adaptive smoothing and information preservation of remote sensing images when there are spatial resolution differences, inconsistent imaging conditions, and irregular sampling times among the original remote sensing images from multiple sources. This suppresses the effects of cloud cover and shadow interference, improves the continuity and consistency of remote sensing images over long time scales, and realizes cross-sensor fusion.

[0069] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0070] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in this application, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0071] Please refer to Figure 1 , Figure 1 The flowchart illustrates a multi-source remote sensing image fusion method provided by this invention.

[0072] This multi-source remote sensing image fusion method includes:

[0073] S11: Acquire multiple raw remote sensing images from the first remote sensing detection device and the second remote sensing detection device respectively;

[0074] S12: Perform standard preprocessing operations on the original remote sensing image to obtain a standard remote sensing image;

[0075] S13: Process the standard remote sensing image according to the preset downsampling strategy to obtain the degraded remote sensing image with the first spatial resolution.

[0076] S14: Pre-establish a time axis for smoothing observations, including various time points;

[0077] S15: Adaptive and robust smoothing processing is performed on the degraded remote sensing images in the time series using a time series robustness processing strategy to obtain robust remote sensing images with the first spatial resolution corresponding to each time point.

[0078] S16: Upsample the robust remote sensing image according to the preset fusion strategy to obtain the fused remote sensing image with a second spatial resolution corresponding to each time point. The second spatial resolution is higher than the first spatial resolution.

[0079] In this embodiment, the fused remote sensing imagery determined by this scheme not only has high spatial resolution to accurately depict building boundaries, road structures, and small-scale changes in land features, but also maintains high observation continuity over long periods. This facilitates the effective capture of the occurrence, evolution, and disappearance of construction activities. Therefore, this scheme can be applied to application scenarios with high requirements for temporal continuity and the reliability of observational changes. Specifically, it can be applied to various land surface change monitoring scenarios such as urban construction supervision, identification of illegal or non-standard construction, and land use or land cover, without any particular limitations.

[0080] To elaborate, the first and second remote sensing detection devices here determine different spatial and temporal resolutions of the remote sensing images during operation; each original remote sensing image corresponds to a timestamp; the first remote sensing detection device can be a first remote sensing sensor, and the second remote sensing detection device can be a second remote sensing sensor; for example, the first remote sensing sensor can be a high spatial resolution commercial remote sensing sensor, which determines a high spatial resolution of the original remote sensing image, but the acquisition time is irregular, and the sampling interval is relatively sparse; the second remote sensing sensor can be a low spatial resolution public remote sensing sensor, which determines a relatively low spatial resolution of the original remote sensing image, but the sampling interval is more frequent, and the spatial resolution of the original remote sensing image determined by the public remote sensing sensor is lower than that determined by the commercial remote sensing sensor. It is understood that this scheme does not depend on a specific remote sensing sensor and can flexibly adapt to data sources from different remote sensing sensors.

[0081] Step S12 involves standard preprocessing of the multi-source raw remote sensing images to obtain comparable standard remote sensing images in terms of physical dimensions, registration accuracy, and spatial coordinate system. This satisfies the basic conditions for cross-sensor fusion, laying the foundation for subsequent cross-sensor processing and avoiding the influence of noise caused by amplified clouds, shadows, and sensor differences. It should be noted that the standard preprocessing operations here can include spectral alignment and geometric registration, such as radiometric calibration, atmospheric correction, spectral alignment, and geometric registration. Radiometric calibration and atmospheric correction ensure that the remote sensing images observed by different remote sensing sensors are comparable in physical dimensions. There is comparability; spectral alignment processing can achieve spectral consistency in different remote sensing sensor bands through spectral response function matching or linear transformation; geometric registration can specifically include geometric calibration and geometric correction and fine registration processing, which can enable multi-source remote sensing images to reach a preset spatial registration accuracy threshold under a unified projection spatial coordinate system. It is understood that the specific implementation steps of the above operations have been described in detail in related technologies, and will not be repeated here; and according to the implementation needs of standard preprocessing operations, quality indicator data, cloud mask data, or observation geometric parameters, such as solar altitude angle and observation angle, can also be obtained, without special limitations here.

[0082] In step S13, standard remote sensing images are uniformly mapped to the first spatial resolution level. Then, based on step S14, an adaptive robust and smooth processing flow for time series is constructed at this spatial resolution level, so that anomalies in the time dimension are fully suppressed before entering subsequent processing steps. Specifically, a timeline for smoothing observations is pre-established, including various time points corresponding to a complete, long-term series. Based on this, time-series robustness processing is implemented in step S15 to supplement the remote sensing images at missing time points, resulting in robust remote sensing images corresponding to each time point within the complete long-term series. The spatial resolution of these robust remote sensing images is a first spatial resolution. Finally, the fused remote sensing images at each time point are determined in step S16, with a second spatial resolution higher than the first spatial resolution. For example, the second spatial resolution is a high spatial resolution, and the first spatial resolution is a low spatial resolution. The first spatial resolution can be the minimum spatial resolution of all standard remote sensing images. Since the spatial resolution of some original remote sensing images is greater than the preset spatial resolution, these original remote sensing images are considered to have a high spatial resolution. Therefore, the second spatial resolution can be one of these high spatial resolutions, or, as needed, several high spatial resolutions. This allows the robust remote sensing images to be upgraded to different high spatial resolutions according to current requirements. No specific limitations are imposed here; flexible settings are acceptable.

[0083] In summary, this application provides a multi-source remote sensing image fusion method. This method can ensure geometric stability and spectral physical consistency through standard preprocessing operations, even when there are spatial resolution differences, inconsistent imaging conditions, and irregular sampling times among the original multi-source remote sensing images. Through a time-series robustness processing strategy, it can reliably achieve robust processing, adaptive smoothing, and information preservation of remote sensing images with degraded first spatial resolution under time-series conditions, so as to suppress the influence of cloud cover and shadow interference, improve the continuity and consistency of robust remote sensing images at the first spatial resolution over long time scales, and finally use a preset fusion strategy to determine the fused remote sensing images at the second spatial resolution corresponding to each time point, realizing cross-sensor fusion, which is beneficial for anomaly monitoring and identification in practical applications.

[0084] Based on the above embodiments:

[0085] In one optional embodiment, standard remote sensing images are processed according to a preset downsampling strategy to obtain degraded remote sensing images with a spatial resolution of the first spatial resolution, including:

[0086] The lowest spatial resolution among the standard remote sensing images is determined as the first spatial resolution.

[0087] The standard remote sensing image with the first spatial resolution is defined as the degraded remote sensing image;

[0088] Using a spatial degradation model, standard remote sensing images with a spatial resolution greater than the first spatial resolution are downsampled to obtain degraded remote sensing images mapped to the first spatial resolution.

[0089] Specifically, standard remote sensing images with a spatial resolution of first spatial resolution do not require downsampling processing. Therefore, these standard remote sensing images are directly used as degraded remote sensing images. The spatial degradation model here can include resampling / convolutional downsampling and point spread function convolution. It can be seen that through the above settings, a first spatial resolution degraded remote sensing image dataset with uniform spatial resolution but including multi-source and multi-temporal observations in the temporal dimension can be obtained.

[0090] It should also be noted that the first spatial resolution here can also be a pre-set spatial resolution, so that all standard remote sensing images are mapped to the degraded remote sensing images corresponding to that spatial resolution, without any particular limitation here; in addition, the spatial resolution of the standard remote sensing image corresponding to the largest pixel size of each of the aforementioned standard remote sensing images can also be determined as the first spatial resolution by determining the largest pixel size, without any particular limitation here.

[0091] In one optional embodiment, the original remote sensing image is a remote sensing image characterizing the land conditions under the target area;

[0092] After obtaining the fused remote sensing images at the second spatial resolution corresponding to each time point, the process also includes:

[0093] The monitoring results determine whether there are any anomalies in the land under the target area based on the fused remote sensing images.

[0094] Specifically, by uniformly addressing time series robustness at the low spatial resolution level and achieving fusion at the high spatial resolution level, fused remote sensing images at various time points constitute a fused remote sensing image dataset. This dataset exhibits higher continuity, consistency, and reliability over long time scales, achieving cross-sensor and cross-resolution fusion, and is time-continuous and robust to cloud interference. It facilitates the determination of whether there are anomalies in the land under the target area by combining anomaly detection conditions in practical applications, with a low false alarm rate. This provides stable and reliable data support for practical operations such as urban construction supervision, and possesses good scalability and engineering applicability. For example, fused remote sensing images determined in the current time period and fused remote sensing images determined in historical time periods can also be used to determine whether anomalies have occurred in the land under the target area. It is understood that the above is only a setting example, and the anomaly detection conditions can be flexibly set according to actual application needs, without any particular limitations.

[0095] In one optional embodiment, the step of obtaining robust remote sensing images with a first spatial resolution corresponding to each time point includes:

[0096] Using each time point as the center of the window, perform the following steps:

[0097] The baseline sliding time window is determined based on the window center and the preset sliding size;

[0098] The set of remote sensing images that satisfy the current baseline sliding time window is determined based on the timestamps of each degraded remote sensing image;

[0099] When the set of remote sensing images in the decision window includes no less than X degraded remote sensing images without cloud cover, the current baseline sliding time window is determined as the optimal sliding time window, where X is an integer greater than 1.

[0100] When the set of remote sensing images in the determination window includes fewer than X degraded remote sensing images without cloud cover, the current reference sliding time window is expanded to obtain a new reference sliding time window. This process continues until the set of remote sensing images in the window corresponding to the new reference sliding time window includes no fewer than X degraded remote sensing images without cloud cover. At this point, the reference sliding time window is determined to be the optimal sliding time window.

[0101] For the degraded remote sensing images under the optimal sliding time window, a weighted average is uniformly applied to each remote sensing image band by band and pixel by pixel to obtain the predicted images for each band. Then, the predicted images for each band are combined to obtain the robust remote sensing image corresponding to the center of the window.

[0102] Specifically, the preset sliding size can be flexibly set according to the actual application needs, and no special limitation is made here. Based on the timestamps corresponding to each degraded remote sensing image, the degraded remote sensing images whose timestamps are within the baseline sliding time window range can be determined to form a window remote sensing image set. When it is determined that the window remote sensing image set includes no less than X cloudless remote sensing images, it means that there are at least X valid observations that can be used to determine the center of the window. Therefore, the baseline sliding time window at this time is the optimal sliding time window. For example, assuming that the timestamps of each original remote sensing image and the corresponding degraded remote sensing image are in days, the above determination steps can be further refined as follows: determine whether there are no clouds for no less than X days in the degraded remote sensing images under the window remote sensing image set, where X is not limited to 5, and can be flexibly set according to the actual application needs.

[0103] When the set of remote sensing images in the determination window includes fewer than X images without cloud cover, it indicates that the effective observations under the current reference sliding time window are insufficient and severely affected by clouds. In this case, the reference sliding time window is adaptively expanded and re-determined. In addition, the specific expansion method of the current reference sliding window can be symmetrical, that is, expanding at equal intervals before and after the center of the window; it can also be asymmetrical, that is, expanding at unequal intervals before and after; it can also be based on the center of the window and only expand forward or only expand backward. There are no special restrictions here, and it can be flexibly set according to the actual application.

[0104] Extendably, the above determination steps can also be set as follows: determining whether the remote sensing image set in the window includes at least X degraded remote sensing images without shadow occlusion or with shadow occlusion area smaller than a preset area; or determining whether the remote sensing image set in the window includes at least X degraded remote sensing images without abnormal brightness fluctuations or with brightness fluctuations within a preset allowable fluctuation range; or determining whether the remote sensing image set in the window includes at least X degraded remote sensing images with noise within a preset allowable range, without any particular limitation here.

[0105] In addition, the weights used in the weighted average processing can be determined by the following rules: the closer the degraded remote sensing image is to the center of the window, the greater its weight. Of course, other weight determination rules can also be used, and no special limitation is made here.

[0106] As can be seen, the above settings can effectively suppress cloud cover, shadow occlusion, abnormal brightness fluctuations, and time series noise caused by irregular revisit cycles without significantly sacrificing temporal resolution. Furthermore, the adaptive adjustment of the sliding time window makes this time series robustness processing strategy adaptable to different target areas, different land cover types, and different observation conditions. This maintains the continuity and stability of remote sensing images under long-term series conditions, resulting in smoothed, robust remote sensing images corresponding to each time point, providing high-quality, low-noise basic input for subsequent cross-sensor fusion.

[0107] In one optional embodiment, the original remote sensing image is a remote sensing image characterizing the land conditions under the target area; the step of obtaining robust remote sensing images with a first spatial resolution corresponding to each time point includes:

[0108] Corresponding to the time axis, all degraded remote sensing images in the time series are used as input items and fed into a pre-trained fitting smoothing model for inference, so as to obtain robust remote sensing images with the first spatial resolution corresponding to each time point in the time axis.

[0109] The training steps for fitting the smooth model include:

[0110] The fitted smoothing model is trained based on the first dataset and the multispectral joint consistency constraint function to adjust the parameters of the fitted smoothing model until the first preset training termination condition is reached.

[0111] The multispectral joint consistency constraint function is:

[0112] ;

[0113] This corresponds to the total loss at the first time point in the first dataset. As the first weight, As the second weight, As the third weight, This represents the first pixel-level mean square error, determined based on the real remote sensing image in the first dataset corresponding to the first time point and the first predicted remote sensing image obtained by inference using a fitted smoothing model. This represents the first spectral angle mapping loss, determined based on the real remote sensing image and the first predicted remote sensing image. This represents the first vegetation index loss, determined based on real remote sensing imagery and the first predicted remote sensing imagery.

[0114] In this embodiment, based on all degraded remote sensing images and a pre-trained fitting smoothing model, a complete temporally robust remote sensing image of the first spatial resolution remote sensing image at the missing observation time points can be inferred. This image is continuous and smooth in the temporal dimension and maintains physical consistency in its spectral structure. Specifically, the fitting smoothing model here includes, but is not limited to, various model methods that can smooth point observations into curves, such as LOESS filtering and fast Fourier transform; no particular limitation is made here.

[0115] Specifically, the steps for establishing the first dataset include: selecting a first time point on the timeline, at which real original remote sensing images exist; processing the original remote sensing images using a preset downsampling strategy to obtain degraded remote sensing images for training at a first spatial resolution; using these degraded remote sensing images as real remote sensing images, i.e., the label data used for training the fitting smoothing model; and selecting degraded remote sensing images from the surrounding time series (i.e., O time points within a preset range before and after the first time point, where O is an integer greater than 1) as input time series samples for the fitting smoothing model. These input time series samples do not... Including the degraded remote sensing images used for training at the first time point itself, as mentioned above; therefore, this first dataset essentially includes surrounding temporally degraded remote sensing images and real remote sensing images at each first time point; then, based on the input temporal samples at each first time point, the current fitting smoothing model can be used to infer the first predicted remote sensing image corresponding to the first spatial resolution at the first time point; training is achieved based on the real remote sensing image and the first predicted remote sensing image, as well as the multispectral joint consistency constraint function; the first preset training termination condition can be flexibly set according to the actual application, such as being set to when When the parameters are less than the preset threshold, it is determined that the parameters of the current fitted smoothing model have reached the optimal level, and training is terminated.

[0116] In addition, here , , The settings can be flexibly adjusted according to the actual needs of the application; no specific value is specified here. Used to quantize the numerical difference between the first predicted remote sensing image and the actual remote sensing image pixel by pixel, limiting pixel value offset. Used to maintain the consistency of multi-band spectral shape, To maintain the consistency of vegetation indices across relevant bands, as for , , The specific calculation method has been detailed in the relevant technology and will not be repeated here. In addition, water index loss, snow index loss, etc. can be added to the multispectral joint consistency constraint function as needed, without special restrictions here.

[0117] As can be seen, by setting the above-mentioned multispectral joint consistency constraints, this application can avoid spectral imbalance caused by independent processing of a single band, so that the robust remote sensing image obtained by inference maintains physical rationality and intrinsic consistency in spectral structure. Multispectral joint consistency constraints realize multi-band cross-validation, avoid cross-band response imbalance caused by single band anomalies, avoid spectral imbalance, and effectively reduce the systematic bias introduced by the differences in spectral response functions of different remote sensing sensors, providing a more consistent and interpretable input basis for subsequent fusion.

[0118] In one optional embodiment, the original remote sensing image is a remote sensing image characterizing the land conditions under the target area; the step of obtaining fused remote sensing images with a second spatial resolution corresponding to each time point includes:

[0119] Using a pre-trained deep learning model, robust remote sensing images are upsampled to obtain fused remote sensing images with second spatial resolution at each time point.

[0120] The training steps for deep learning models include:

[0121] The deep learning model is trained based on the second dataset and the first loss function to adjust the parameters of the deep learning model until the second preset training termination condition is reached; the second dataset includes multiple image pairs, each image pair including training remote sensing images at the second spatial resolution corresponding to the second time point and training remote sensing images at the first spatial resolution.

[0122] The first loss function is:

[0123] ;

[0124] in, For the total loss corresponding to the second time point in the second dataset, As the fourth weight, As the fifth weight, It is the sixth weight. The second pixel-level mean square error is determined by the training remote sensing image at the second time point and the second predicted remote sensing image obtained by reasoning using a deep learning model. The spatial resolution of the second predicted remote sensing image is the second spatial resolution. This represents the second spectral angle mapping loss, determined based on the training remote sensing image and the second predicted remote sensing image at the second spatial resolution. The second vegetation index loss is represented by the training remote sensing image and the second predicted remote sensing image, which are determined based on the second spatial resolution.

[0125] Specifically, training remote sensing images at the second spatial resolution and training remote sensing images at the first spatial resolution can be selected in advance from any time (i.e., the second time point) and region within a few years or decades as a set of image pairs. P sets of image pairs are used to construct the second dataset, where P is an integer greater than 1. It should be noted that the training remote sensing images at the first spatial resolution mentioned here are obtained directly through remote sensing sensors, rather than being obtained by using a preset downsampling strategy to achieve degradation downsampling.

[0126] Using remote sensing imagery as input for training at the first spatial resolution, and employing a deep learning model for inference, a corresponding second spatial resolution, second predicted remote sensing imagery can be obtained. Furthermore, the aforementioned first loss function is used for training, enabling the deep learning model's inference to enhance spatial details while maintaining spectral and temporal consistency. Additionally, the second preset training termination condition can be flexibly set according to actual application needs, without specific limitations here. As for... , , The specific calculation steps have been detailed in the relevant technologies and will not be repeated here; , , You can set it up flexibly according to the actual needs of the application.

[0127] It is understood that the deep learning model here includes, but is not limited to, deep learning networks, and the output of the model may also include indicators such as confidence level, without any special limitation here; in addition, the fused remote sensing images of each second spatial resolution include, but are not limited to, being stored or provided externally in the form of image files, such as being sent to downstream business systems to support anomaly monitoring.

[0128] In an optional embodiment, after obtaining the predicted remote sensing image using a deep learning model, the method further includes:

[0129] The second predicted remote sensing image is processed according to a preset downsampling strategy to obtain the predicted degraded remote sensing image.

[0130] Based on the predicted degraded remote sensing image and the training remote sensing image with the first spatial resolution, the corresponding third pixel-level mean square error, third spectral angle mapping loss, and third vegetation index loss are determined, and then weighted to obtain the reversible degradation consistency loss.

[0131] Determine the reversible degradation consistency loss and The weighted average is the comprehensive loss corresponding to the second time point in the second dataset.

[0132] Specifically, by introducing reversible degradation consistency loss through the above settings, it is beneficial to ensure that the predicted degraded remote sensing image is consistent with the training remote sensing image at the first spatial resolution. The setting of the comprehensive loss is beneficial to improve the physical reliability of the cross-sensor reconstruction results, i.e., the fused remote sensing image. It is also beneficial to effectively suppress over-sharpening, texture illusion and local structural distortion, so that the fused remote sensing image is closer to the real remote sensing observation in terms of visual effect, spectral characteristics and spatial structure, which is beneficial to improve the generalization ability and stability of this scheme.

[0133] In an optional embodiment, after obtaining the predicted remote sensing image using a deep learning model, the method further includes:

[0134] The training remote sensing images at the first spatial resolution are subjected to spectral unmixing description processing to obtain the corresponding first description information;

[0135] The second predicted remote sensing image is subjected to spectral unmixing description processing to obtain the corresponding second description information;

[0136] The spectral unmixing constraint loss is determined based on the first and second descriptive information.

[0137] Determine the reversible degradation consistency loss and The weighted average is the comprehensive loss corresponding to the second time point in the second dataset, including:

[0138] Determine the spectral unmixing constraint loss, reversible degradation consistency loss, and The weighted average is the comprehensive loss corresponding to the second time point in the second dataset.

[0139] Specifically, the first descriptive information is the first endmember spectrum and the first abundance distribution corresponding to the training remote sensing image with the first spatial resolution, reflecting the composition of land features. The second descriptive information is the second endmember spectrum and the second abundance distribution corresponding to the second predicted remote sensing image, reflecting the composition of land features. The determination of the spectral unmixing constraint loss aims to constrain the two to be consistent at the land feature composition level, that is, the first endmember spectrum should match the second endmember spectrum, and the first abundance distribution should match the second abundance distribution. Therefore, the spectral unmixing constraint loss is a weighted sum of the differences between the first and second endmember spectra and the differences between the first and second abundance distributions. It can be understood that the weights used are used to balance the differences in the numerical range and dimensions of different weighting terms.

[0140] It is evident that by introducing spectral unmixing constraint loss, deep learning models can pay attention to changes in land cover composition when determining fused remote sensing images. This helps improve the stability and interpretability of cross-sensor fusion results, reduce the probability of false detections in land anomaly monitoring, and increase the reliability of judgment results.

[0141] Please refer to Figure 2 ,Figure 2 This is a schematic diagram of the structure of a multi-source remote sensing image fusion system provided by the present invention.

[0142] This multi-source remote sensing image fusion system includes:

[0143] The acquisition module 21 is used to acquire multiple raw remote sensing images from the first remote sensing detection device and the second remote sensing detection device, respectively.

[0144] The preprocessing module 22 is used to perform standard preprocessing operations on the original remote sensing image to obtain a standard remote sensing image.

[0145] The downsampling degradation module 23 is used to process the standard remote sensing image according to the preset downsampling strategy to obtain the degraded remote sensing image with the first spatial resolution.

[0146] Time axis creation module 24 is used to pre-create a time axis for smoothing observations, which includes various time points;

[0147] The temporal robustness processing module 25 is used to perform adaptive robust smoothing processing on the degraded remote sensing images under the temporal sequence using a temporal robustness processing strategy, so as to obtain robust remote sensing images with the first spatial resolution corresponding to each time point.

[0148] The upsampling fusion module 26 is used to upsample the robust remote sensing image according to a preset fusion strategy to obtain the fused remote sensing image with a second spatial resolution corresponding to each time point. The second spatial resolution is higher than the first spatial resolution.

[0149] For a description of the multi-source remote sensing image fusion system provided in this application, please refer to the embodiments of the multi-source remote sensing image fusion method described above, which will not be repeated here.

[0150] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a multi-source remote sensing image fusion device provided by the present invention.

[0151] The multi-source remote sensing image fusion device includes:

[0152] Memory 31 is used to store computer programs;

[0153] The processor 32 is used to execute the computer program to implement the steps of the multi-source remote sensing image fusion method as described above.

[0154] For a description of the multi-source remote sensing image fusion device provided in this application, please refer to the embodiments of the multi-source remote sensing image fusion method described above, which will not be repeated here.

[0155] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. Relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0156] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for fusing multi-source remote sensing images, characterized in that, include: Multiple raw remote sensing images were acquired from the first and second remote sensing devices, respectively. The original remote sensing image is subjected to standard preprocessing operations to obtain a standard remote sensing image; The standard remote sensing image is processed according to a preset downsampling strategy to obtain a degraded remote sensing image with a spatial resolution of the first spatial resolution. A time axis for smoothing observations is pre-established, and the time axis includes various time points; An adaptive robust smoothing process is applied to the degraded remote sensing images in the time series using a time series robustness processing strategy to obtain robust remote sensing images with the first spatial resolution corresponding to each time point. The robust remote sensing images are upsampled according to a preset fusion strategy to obtain fused remote sensing images with a second spatial resolution corresponding to each time point, wherein the second spatial resolution is higher than the first spatial resolution. The step of obtaining robust remote sensing images with the first spatial resolution corresponding to each of the aforementioned time points includes: Using each of the aforementioned time points as the center of the window, perform the following steps: The reference sliding time window is determined based on the window center and the preset sliding size; The set of remote sensing images that satisfy the current reference sliding time window is determined based on the timestamps of each of the degraded remote sensing images. When it is determined that the remote sensing image set in the window includes no less than X degraded remote sensing images without cloud cover, the current reference sliding time window is determined to be the optimal sliding time window, where X is an integer greater than 1; When it is determined that the remote sensing image set in the window includes fewer than X degraded remote sensing images without cloud cover, the current reference sliding time window is expanded to obtain a new reference sliding time window. This process continues until it is determined that the remote sensing image set in the window corresponding to the new reference sliding time window includes no fewer than X degraded remote sensing images without cloud cover. At this point, the reference sliding time window is determined to be the optimal sliding time window. For the degraded remote sensing images under the optimal sliding time window, a weighted average processing is performed on each remote sensing image band by band and pixel by pixel to obtain the predicted images for each band. Then, the predicted images for each band are combined to obtain the robust remote sensing image corresponding to the center of the window.

2. The multi-source remote sensing image fusion method as described in claim 1, characterized in that, The standard remote sensing image is processed according to a preset downsampling strategy to obtain a degraded remote sensing image with a spatial resolution of the first spatial resolution, including: The lowest spatial resolution among the aforementioned standard remote sensing images is determined as the first spatial resolution; The standard remote sensing image with the first spatial resolution is determined to be the degraded remote sensing image; Using a spatial degradation model, standard remote sensing images with a spatial resolution greater than the first spatial resolution are downsampled to obtain degraded remote sensing images mapped to the first spatial resolution.

3. The multi-source remote sensing image fusion method as described in claim 1, characterized in that, The original remote sensing image is a remote sensing image that characterizes the land conditions under the target area; After obtaining the fused remote sensing images with the second spatial resolution corresponding to each of the aforementioned time points, the method further includes: The monitoring results determine whether there are any abnormalities in the land under the target area based on the fused remote sensing imagery.

4. The multi-source remote sensing image fusion method as described in claim 1, characterized in that, The original remote sensing image is a remote sensing image characterizing the land conditions in the target area; the step of obtaining robust remote sensing images with the first spatial resolution corresponding to each of the aforementioned time points includes: Corresponding to the time axis, all degraded remote sensing images in the time series are used as input items and fed into a pre-trained fitting smoothing model for inference, so as to obtain robust remote sensing images with the first spatial resolution corresponding to each time point in the time axis. The training steps of the fitted smoothing model include: The fitting smoothing model is trained based on the first dataset and the multispectral joint consistency constraint function to adjust the parameters of the fitting smoothing model until the first preset training termination condition is reached. The multispectral joint consistency constraint function is: ； This corresponds to the total loss at the first time point in the first dataset. As the first weight, As the second weight, As the third weight, The first pixel-level mean square error is determined based on the real remote sensing image in the first dataset corresponding to the first time point and the first predicted remote sensing image obtained by inference using the fitted smoothing model. This represents the first spectral angle mapping loss, and is determined based on the real remote sensing image and the first predicted remote sensing image. This represents the first vegetation index loss, and is determined based on the actual remote sensing image and the first predicted remote sensing image.

5. The multi-source remote sensing image fusion method according to any one of claims 1 to 4, characterized in that, The original remote sensing image is a remote sensing image characterizing the land conditions in the target area; the step of obtaining the fused remote sensing image with a second spatial resolution corresponding to each of the aforementioned time points includes: Using a pre-trained deep learning model, the robust remote sensing images are upsampled to obtain fused remote sensing images with a second spatial resolution corresponding to each time point. The training steps of the deep learning model include: The deep learning model is trained based on the second dataset and the first loss function to adjust the parameters of the deep learning model until the second preset training termination condition is reached; the second dataset includes multiple image pairs, each image pair including a training remote sensing image at a second spatial resolution corresponding to a second time point, and a training remote sensing image at a first spatial resolution. The first loss function is: ； in, This corresponds to the total loss at the second time point in the second dataset. As the fourth weight, As the fifth weight, It is the sixth weight. The second pixel-level mean square error is determined based on the training remote sensing image at the second time point with the second spatial resolution and the second predicted remote sensing image obtained by reasoning using the deep learning model. The spatial resolution of the predicted remote sensing image is the second spatial resolution. The second spectral angle mapping loss is represented and determined based on the training remote sensing image and the second predicted remote sensing image at the second spatial resolution. The second vegetation index loss is represented and determined based on the training remote sensing image and the second predicted remote sensing image at the second spatial resolution.

6. The multi-source remote sensing image fusion method as described in claim 5, characterized in that, After obtaining the predicted remote sensing image using the deep learning model, the process further includes: The second predicted remote sensing image is processed according to the preset downsampling strategy to obtain the predicted degraded remote sensing image. Based on the predicted degraded remote sensing image and the training remote sensing image with the first spatial resolution, the corresponding third pixel-level mean square error, third spectral angle mapping loss, and third vegetation index loss are determined, and then weighted to obtain the reversible degradation consistency loss. Determine the reversible degradation consistency loss and The weighted sum is the comprehensive loss corresponding to the second time point in the second dataset.

7. The multi-source remote sensing image fusion method as described in claim 6, characterized in that, After obtaining the predicted remote sensing image using the deep learning model, the process further includes: The training remote sensing image with the first spatial resolution is subjected to spectral unmixing description processing to obtain the corresponding first description information; The second predicted remote sensing image is subjected to spectral unmixing description processing to obtain the corresponding second description information; The spectral unmixing constraint loss is determined based on the first description information and the second description information; Determine the reversible degradation consistency loss and The weighted sum is the comprehensive loss corresponding to the second time point in the second dataset, including: Determine the spectral unmixing constraint loss, the reversible degradation consistency loss, and... The weighted sum is the comprehensive loss corresponding to the second time point in the second dataset.

8. A multi-source remote sensing image fusion system, characterized in that, include: The acquisition module is used to acquire multiple raw remote sensing images from the first remote sensing detection device and the second remote sensing detection device, respectively. The preprocessing module is used to perform standard preprocessing operations on the original remote sensing image to obtain a standard remote sensing image. The downsampling degradation module is used to process the standard remote sensing image according to a preset downsampling strategy to obtain a degraded remote sensing image with a spatial resolution of the first spatial resolution. A time axis creation module is used to pre-create a time axis for smoothed observation, which includes various time points; The temporal robustness processing module is used to perform adaptive robust smoothing processing on the degraded remote sensing images in the temporal sequence using a temporal robustness processing strategy, so as to obtain robust remote sensing images with the first spatial resolution corresponding to each time point. The upsampling fusion module is used to upsample the robust remote sensing image according to a preset fusion strategy to obtain a fused remote sensing image with a second spatial resolution corresponding to each time point, wherein the second spatial resolution is higher than the first spatial resolution. Specifically, the time-series robustness processing module is used to perform the following steps, with each time point as the window center: determining a baseline sliding time window based on the window center and a preset sliding size; determining a set of remote sensing images that satisfy the current baseline sliding time window based on the timestamps of each degraded remote sensing image; determining the current baseline sliding time window as the optimal sliding time window when the set of remote sensing images includes at least X degraded remote sensing images without cloud cover, where X is an integer greater than 1; and determining the optimal sliding time window when the set of remote sensing images includes fewer than X degraded remote sensing images without cloud cover. When dealing with degraded remote sensing images, the current reference sliding time window is expanded to obtain a new reference sliding time window. This process continues until the set of remote sensing images corresponding to the new reference sliding time window includes at least X degraded remote sensing images without cloud cover. At this point, the reference sliding time window is determined to be the optimal sliding time window. For the degraded remote sensing images under the optimal sliding time window, a weighted average is uniformly applied to each remote sensing image band by band and pixel by pixel to obtain predicted images for each band. These predicted images are then combined to obtain the robust remote sensing image corresponding to the center of the window.

9. A multi-source remote sensing image fusion device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the multi-source remote sensing image fusion method as described in any one of claims 1 to 7 when executing the computer program.

Images