A method and system for enhancing low-resolution remote sensing images based on multiple temporal phases

Abstract

This invention provides a method and system for enhancing low-resolution remote sensing images based on multiple temporal phases, belonging to the field of image processing technology. The method includes: using the imaging time point of a target high-resolution image as a reference, acquiring K temporal phases of low-resolution images corresponding to the corresponding geographical location through dynamically adjusted time windows, and performing data preprocessing to form a remote sensing image enhancement dataset; wherein, each training data in the remote sensing image enhancement dataset includes... K A low-resolution image patch at each time phase and a high-resolution image patch of the target at the corresponding geographic location. Y The method involves inputting low-resolution image patches from K temporal phases into a resolution enhancement model for feature extraction, encoding, fusion, and decoding to generate high-resolution image patches. A hybrid loss function is then used to guide the training of the resolution enhancement model to generate the final resolution enhancement model. This invention improves the accuracy and robustness of multi-temporal image enhancement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a low-resolution remote sensing image enhancement method and system based on multi-temporal. BACKGROUND

[0002] Remote sensing images have important value in the fields of river and lake shoreline protection monitoring, urban planning, disaster tracking and evaluation, etc. However, due to the satellite revisit period and imaging cost, the acquisition frequency of high-resolution (HR) images is much lower than that of low-resolution (LR) images. How to use the easily acquired multi-temporal LR images to enhance or reconstruct the missing HR images has become a research hotspot in the field of remote sensing, i.e. multi-temporal super-resolution (MT-SR) technology.

[0003] The existing MT-SR methods usually have the following shortcomings:

[0004] 1. Single data enhancement method: The existing methods mostly use geometric transformation (such as flipping and rotating), which cannot simulate the real changes of remote sensing images in different time phases, such as illumination, season, and atmosphere, resulting in insufficient robustness of the model to real environmental changes.

[0005] 2. Insufficient feature extraction: Traditional convolutional neural networks (CNNs) are good at extracting local texture features, but have weak perception ability for global structure and long-distance dependence of images. Although the Transformer can capture global information, it may ignore fine local details. The simple combination of the two cannot achieve complementary advantages.

[0006] 3. Rough temporal fusion mechanism: The existing fusion methods (such as splicing, 3D convolution, and simple attention mechanism) often couple the spatio-temporal features for processing, which is difficult to accurately distinguish and utilize the effective information (such as cloud-free areas) in different temporal images, and is likely to introduce noise or cause information redundancy.

[0007] 4. Single loss function design: The loss function usually only focuses on the difference between the final reconstruction result and the real HR image, ignoring the contribution of each single temporal LR image, especially not considering the temporal proximity of the image and the target HR image, resulting in an insufficiently fine model training target.

[0008] Therefore, there is an urgent need to propose a new technical solution to solve at least one of the above problems and improve the precision and robustness of multi-temporal image enhancement. SUMMARY

[0009] The present application provides a low-resolution remote sensing image enhancement method and system based on multi-temporal, which solves at least one of the defects in the prior art.

[0010] In a first aspect, the present application provides a low-resolution remote sensing image enhancement method based on multi-temporal, comprising:

[0011] Step 1: Using the imaging time point of the target high-resolution image as a reference, acquire the corresponding geographical location through a dynamically adjusted time window. K Low-resolution images from various time phases were collected and preprocessed to form a remote sensing image enhancement dataset; each training data in the remote sensing image enhancement dataset includes... K A low-resolution image patch at each time phase and a high-resolution image patch of the target at the corresponding geographic location. Y ;

[0012] Step 2: K Low-resolution image patches from each time phase are input into a resolution enhancement model for feature extraction, encoding, fusion, and decoding to generate high-resolution image patches. The resolution enhancement model includes an encoder, a feature fusion module, and a decoder.

[0013] Step 3: Set a hybrid loss function to guide the training of the resolution enhancement model to generate the final resolution enhancement model. Specifically:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019] in, For structural loss, For time-weighted single-phase loss, and To balance the hyperparameters of different loss terms, For multi-scale structural similarity loss, For the first k The corresponding time point of each hour The imaging time point of the target high-resolution image. These are preset parameters. It is the first k Low-resolution image patches of each time phase Preliminary high-resolution image patches are formed through feature encoding and decoding reconstruction.

[0020] According to the present invention, a low-resolution remote sensing image enhancement method based on multi-temporal phases is provided. The data preprocessing in step 1 includes: performing standard preprocessing on all acquired images, which includes dehazing, color alignment, and geometric registration; cutting the low-resolution image after standard preprocessing into patches of size H×W, and cutting the high-resolution image after standard preprocessing into patches of size (s×H)×(s×W), where s is the upsampling factor.

[0021] According to the multi-temporal low-resolution remote sensing image enhancement method provided by the present invention, step 2, after data preprocessing, further includes: performing data enhancement processing on low-resolution image patches of some temporal phases. The data enhancement processing includes one or more of the following processing methods: seasonal style transfer processing, illumination and shadow simulation processing, and controllable interference synthesis processing. Among them, seasonal style transfer processing refers to performing seasonal style transfer on low-resolution image patches and then generating new low-resolution image patches through image hybridization; illumination and shadow simulation processing refers to physically rendering low-resolution image patches to simulate the illumination and shadow effects at different times of the day; and controllable interference synthesis processing refers to randomly adding interference to clear low-resolution image patches.

[0022] According to the present invention, a low-resolution remote sensing image enhancement method based on multiple temporal phases is provided. The encoder of the resolution enhancement model is a parallel dual-stream feature extraction encoder. For each temporal low-resolution image patch, the dual-stream feature extraction encoder first uses shallow feature extraction to reduce the feature dimension, then uses a parallel dual-stream structure for feature extraction, then obtains multi-feature attention weights through a channel attention calculation module, and finally achieves the complementarity of local details and global context information through weighting to generate the LR feature map of each temporal low-resolution image patch.

[0023] According to the present invention, a low-resolution remote sensing image enhancement method based on multiple temporal phases is provided, wherein the feature fusion module is a spatiotemporal decoupling fusion module: spatial self-attention calculation is performed independently on the feature map of each temporal phase to enhance spatial structure consistency; the LR feature map of the latest time point is used as the query, and others... K The spatially enhanced LR feature maps of -1 time phases are used as keys and values, cross-attention calculation is performed, weights are adaptively assigned, and weighted summation is performed to generate a fused feature map.

[0024] According to the present invention, a low-resolution remote sensing image enhancement method based on multi-temporal phases is provided, wherein the decoder is a densely cascaded decoder: the total upsampling factor s is decomposed into multiple stages. Each stage corresponds to one upsampling block; the first l The input of the upsampling block not only includes the first lThe output after upsampling one block is further combined with the output from the first to the second block through a concatenation operation. l -1 Output feature maps of all upsampled blocks.

[0025] According to the present invention, a low-resolution remote sensing image enhancement method based on multi-temporal phases is provided, wherein the parallel dual-stream structure includes a CNN stream and a Transformer stream; wherein the CNN stream is used to extract local feature information, and the Transformer stream is used to extract global feature information.

[0026] In a second aspect, the present invention also provides a low-resolution remote sensing image enhancement system based on multiple temporal phases, comprising: a first processing module, a second processing module and a third processing module;

[0027] The first processing module is used to acquire data from the corresponding geographical location using a dynamically adjusted time window, based on the imaging time point of the target's high-resolution image. K Low-resolution images from various time phases were collected and preprocessed to form a remote sensing image enhancement dataset; each training data in the remote sensing image enhancement dataset includes... K A low-resolution image patch at each time phase and a high-resolution image patch of the target at the corresponding geographic location. Y ;

[0028] The second processing module is used to... K Low-resolution image patches from each time phase are input into a resolution enhancement model for feature extraction, encoding, fusion, and decoding to generate high-resolution image patches. The resolution enhancement model includes an encoder, a feature fusion module, and a decoder.

[0029] The third processing module is used to set a hybrid loss function to guide the training of the resolution enhancement model to generate the final resolution enhancement model. Specifically:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] in, For structural loss, For time-weighted single-phase loss, and To balance the hyperparameters of different loss terms, For multi-scale structural similarity loss, For the first k The corresponding time point of each hour The imaging time point of the target high-resolution image. These are preset parameters. It is the first k Low-resolution image patches of each time phase Preliminary high-resolution image patches are formed through feature encoding and decoding reconstruction.

[0036] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described multi-temporal low-resolution remote sensing image enhancement methods.

[0037] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-temporal low-resolution remote sensing image enhancement method as described above.

[0038] The low-resolution remote sensing image enhancement method and system based on multi-temporal phases provided by this invention have the following advantages compared with the prior art:

[0039] (1) This invention solves the problems of poor robustness of the model to changes in the real environment, insufficient feature extraction, inaccurate use of temporal information, easy loss of reconstruction details and imprecise training target by innovative data augmentation, feature extraction, temporal fusion, decoding reconstruction and loss function design, thereby significantly improving the quality and realism of the enhanced image.

[0040] (2) Through innovative virtual temporal data enhancement, the model can adapt to more complex real-world changes (seasons, lighting, clouds and fog), and its generalization ability and robustness are significantly improved.

[0041] (3) This invention combines the local detail perception capability of CNN and the global structure capture capability of Transformer through a parallel dual-stream encoder, and achieves a more comprehensive and efficient representation of remote sensing image features through adaptive fusion via a gating mechanism.

[0042] (4) The present invention can accurately evaluate and aggregate the most effective information in different time phases through the spatiotemporal decoupling attention fusion module, effectively suppress the interference of noise such as clouds, fog, and shadows, and improve the signal-to-noise ratio of the fusion features.

[0043] (5) The dense cascaded decoder in this invention effectively prevents information loss during the upsampling process by using cross-level feature reuse, making the final generated image clearer and richer in details such as texture and edge.

[0044] (6) The time-weighted hybrid loss function provided by the present invention provides a more refined supervision signal for the model, guides the model to learn information of temporally adjacent images first, conforms to physical laws, accelerates model convergence, and improves the final reconstruction accuracy. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is one of the flowcharts of the low-resolution remote sensing image enhancement method based on multiple temporal phases provided by the present invention;

[0047] Figure 2 This is the second flowchart of the low-resolution remote sensing image enhancement method based on multiple temporal phases provided by the present invention;

[0048] Figure 3 This is a structural diagram of the parallel dual-stream feature extraction encoder provided by the present invention;

[0049] Figure 4 This is a schematic diagram of spatiotemporal decoupling attention fusion provided by the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of the low-resolution remote sensing image enhancement system based on multiple temporal phases provided by the present invention;

[0051] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0054] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0055] The following is combined with Figures 1-6 This invention describes a low-resolution remote sensing image enhancement method and system based on multi-temporal phases provided by embodiments of the present invention.

[0056] Figure 1 This is one of the flowcharts illustrating the low-resolution remote sensing image enhancement method based on multi-temporal phases provided by the present invention. Figure 2 This is the second flowchart illustrating the low-resolution remote sensing image enhancement method based on multi-temporal phases provided by this invention. See also... Figure 1 and Figure 2 The present invention includes, but is not limited to, the following steps:

[0057] Step 1: Using the imaging time point of the target high-resolution image (i.e., the target HR image) as a reference, acquire corresponding geographical locations through a dynamically adjusted time window. K Low-resolution (LR) images from various time phases were collected and preprocessed to form a remote sensing image augmentation dataset. Each training data point in the remote sensing image augmentation dataset includes... K A low-resolution image patch (i.e., LR image patch) at each time phase and a high-resolution image patch of the target at the corresponding geographical location. Y (i.e., target HR image patch).

[0058] Step 1 involves acquiring data from existing high-resolution HR and low-resolution LR image sources. After the following processing stages, a remote sensing image enhancement dataset is created. Each training dataset consists of... K Temporal LR image patch , … A target HR image patch with a corresponding geographical location Y Composition. Specifically:

[0059] (1) Time-adaptive acquisition: For each target HR image acquired from the HR image source, its imaging time point is used as the basis for the acquisition. t Based on this, an initial time window t0 is set, and LR images corresponding to the corresponding geographic locations are filtered within the range [t-t0, t]. A maximum number of LR images, K, is set. If the number of images acquired within this window is less than [the specified value], the LR image acquisition method will be used. K Then, t0 is gradually increased by a preset step size until the desired value is obtained. K Up to the last LR image, corresponding K Each phase. This dynamic windowing mechanism ensures that sufficient time-series information can be obtained under different data densities.

[0060] (2) Standard preprocessing: Dehaze, color alignment and high-precision geometric registration are performed on all acquired HR and LR images.

[0061] (3) Synchronous tiling: The standard preprocessed LR image and the target HR image are synchronously tiled into patches. If the size of the LR image patch is H×W and the upsampling factor is s, then the size of the corresponding target HR image patch is (s×H)×(s×W).

[0062] (4) Data Augmentation: In addition to conventional geometric augmentation (rotation, flipping), this invention introduces a "virtual temporal" data augmentation strategy, which aims to simulate real-world changes and improve the model's generalization ability. This strategy includes:

[0063] Seasonal style transfer: This technique utilizes unpaired image style transfer networks (such as CycleGAN) to learn the feature distributions of images from different seasons (e.g., summer and winter). During training, LR image patches from certain time periods are randomly transferred from one seasonal style to another, generating "virtual seasonal" LR image patches. New image patches are then generated using an "image hybridization" method. (The text then abruptly shifts to a different topic: Assuming image patches...) A "virtual season" image tile was generated using a style transfer network. New image tiles can then be generated using the following "image hybridization" method. :

[0064] (1)

[0065] (2)

[0066] in, For the style transfer network of season 1, The adjustment factor aims to give the new image patch diverse features without deviating too much from the real image; a larger value makes it more like a virtual image patch. In practical applications, this image patch can replace the original image patch, increasing the diversity of the image.

[0067] Lighting and Shadow Simulation: Based on image metadata (such as solar elevation angle and azimuth angle) and a simplified digital elevation model (DEM), physically render LR image tiles to simulate lighting and shadow effects at different times of day, generating “virtual temporal” images.

[0068] Controlled interference synthesis: Synthesized, semi-transparent thin clouds or fog layers and random noise are randomly added to clear LR image patches, forcing the model to learn to extract effective information from other phases even in the presence of slight interference.

[0069] Data augmentation steps can randomly employ one or more of the strategies described above in combination to further enhance the richness of the data.

[0070] Step 2: K Low-resolution image patches from each time phase are input into a resolution enhancement model for feature extraction, encoding, fusion, and decoding to generate high-resolution image patches. .

[0071] In this step, the invention designs an end-to-end resolution enhancement network, the core of which consists of an encoder, a feature fusion module, and a decoder, and its input is... K The output is a generated HR image patch, which is obtained from LR image patches at each time phase. The following is a description of each part.

[0072] (1) The encoder of the resolution enhancement model is a parallel two-stream feature extraction encoder:

[0073] For each temporal LR image patch, a shallow feature extraction operation is first used to reduce the feature dimension, then a parallel two-stream structure is used for feature extraction, then a multi-feature attention weight is obtained through a channel attention calculation module, and finally a weighted summation is used to achieve complementarity between local details and global contextual information.

[0074] Figure 3 This is a structural diagram of the parallel dual-stream feature extraction encoder provided by the present invention, as shown below. Figure 3 As shown, the parallel dual-stream feature extraction encoder includes the following main components.

[0075] CNN stream: Employs lightweight convolutional neural network branches (such as EfficientNet or MobileNet) to efficiently extract local texture, edge, and spatial detail features of images.

[0076] Transformer flow: Using the Swing Transformer or Vision Transformer branch, the tiles are divided into non-overlapping windows / sequences, and long-distance dependencies and global structural information between regions within the image are captured through a self-attention mechanism.

[0077] Gated-Attention Interaction Fusion: In the encoder, a gated-attention fusion module is designed. This module takes features from two streams as input and fuses the local features of the CNN, the global features of the Transformer, and shallow features through attention weights to generate more expressive multi-scale feature maps.

[0078] The process is as follows:

[0079] (3)

[0080] (4)

[0081] (5)

[0082] (6)

[0083] (7)

[0084] (8)

[0085]

[0086] in, This represents a specific temporal phase LR image patch from the input. This indicates that a feature reduction network (consisting of several convolutional layers) is used to extract shallow features in the initial stage of the model. This reduces the feature dimension and the amount of subsequent computation. and These represent CNN stream and Transformer stream, respectively. and These represent the features they extracted, and their feature dimensions are consistent. This indicates a scaling operation, which scales the feature dimensions to... and The dimension; Indicates the characteristics after fusion; The first characteristic represents the feature m One channel, M Represents the total number of channels. , as well as These are preset parameters. This represents the dot product operation, used to calculate the similarity of normalized features.

[0087] (2) The feature fusion module is a spatiotemporal decoupling fusion module:

[0088] Figure 4 This is a schematic diagram of spatiotemporal decoupling attention fusion provided by the present invention, as shown below. Figure 4 Explain it, K The LR image patches from each time phase are obtained after being processed by a parallel two-stream feature extraction encoder. K Each feature map is input into this module for fusion. To avoid confusion of spatiotemporal information, this invention proposes a decoupling process:

[0089] Spatial introspection enhancement: First, spatial self-attention is calculated independently for the feature map of each time phase. This step aims to enhance the consistency of spatial structure within each feature map and highlight salient targets.

[0090] Time alignment aggregation: Then, using the LR feature map features corresponding to the LR image patch at the latest time point (i.e., the largest time point among phases 1 to K) as the query (Q), other... K -1 spatially augmented LR feature maps are used as keys and values ​​(V) for a cross-temporal attention computation. The model generates attention weights by calculating the similarity between the query vector and features at each time phase, thereby adaptively assigning importance to different time phases (the purpose is to compensate for the shortcomings of the latest time phase relative to the true results, and to make up for the shortcomings through this weighting). Finally, a weighted sum is obtained to obtain a single feature map that integrates the essence of information from multiple time phases, namely the fused feature map.

[0091] (3) The decoder is a densely concatenated decoder:

[0092] To maximize detail retention during decoding, this invention employs a densely connected cascaded decoder:

[0093] Multi-level upsampling: Decompose the total upsampling factor s into multiple stages, such as... The decoder contains n One upsampling block.

[0094] Dense Feature Fusion: The l The input of the upsampling block not only includes the first... l The output after upsampling one block is further combined with the output from the first to the second block through a concatenation operation. l-1 is the output feature map of all upsampled blocks (appropriately resized). That is, the higher-resolution reconstruction process can "directly connect" and utilize feature information from all low-resolution stages. This DenseNet-like dense connection mechanism creates a direct path for information, effectively mitigating the problem of detail loss in deep networks.

[0095] Step 3: Set a hybrid loss function to guide the training of the resolution enhancement model to generate the final resolution enhancement model. The hybrid loss function... Specifically:

[0096] (10)

[0097] Among them, structural loss : Calculate the final generated HR image patch With real HR image blocks The difference between them. The loss is composed of a weighted average of L1 loss and multi-scale structural similarity (MS-SSIM) loss to simultaneously ensure pixel-level accuracy and perceptual quality.

[0098] (11)

[0099] Time-weighted single-phase loss It supervises the model from each individual LR image patch. Starting from this point, a preliminary reconstruction of the HR image patch can also be generated. (Depend on The encoded features are directly input into the densely concatenated upsampling decoder for generation, such as Figure 2 (As shown by the dashed line). The core of this loss lies in the loss weight for each time phase. With this LR image patch Time difference with target HR image patch hook up.

[0100] (12)

[0101] Weighting coefficient Defined by an exponentially decaying function:

[0102] (13)

[0103] in, It is a normal number used to control the decay rate.

[0104] Understandably, the closer the LR image is to the target time point t, the more reliable the information it contains, and the larger the proportion of its single-phase reconstruction loss in the total loss should be. This makes the optimization direction of the model more consistent with physical reality. λ and α in formulas (10) and (11) are hyperparameters that balance different loss terms.

[0105] Figure 5 This is a schematic diagram of the structure of the low-resolution remote sensing image enhancement system based on multi-temporal phases provided by the present invention, including:

[0106] The first processing module 510 is used to acquire data from the corresponding geographical location using a dynamically adjusted time window, based on the imaging time point of the target high-resolution image. K Low-resolution images from various time phases were collected and preprocessed to form a remote sensing image enhancement dataset; each training data in the remote sensing image enhancement dataset includes... K A low-resolution image patch at each time phase and a high-resolution image patch of the target at the corresponding geographic location. Y ;

[0107] The second processing module 520 is used to process... K Low-resolution image patches from each time phase are input into a resolution enhancement model for feature extraction, encoding, fusion, and decoding to generate high-resolution image patches. The resolution enhancement model includes an encoder, a feature fusion module, and a decoder.

[0108] The third processing module 530 is used to set a hybrid loss function to guide the training of the resolution enhancement model to generate the final resolution enhancement model. Specifically:

[0109]

[0110]

[0111]

[0112]

[0113]

[0114] in, For structural loss, For time-weighted single-phase loss, and To balance the hyperparameters of different loss terms, For multi-scale structural similarity loss, For the first k The corresponding time point of each hour The imaging time point of the target high-resolution image. These are preset parameters. It is directly composed of low-resolution image tiles Preliminary high-resolution image patches are formed through feature encoding and decoding reconstruction.

[0115] It should be noted that the low-resolution remote sensing image enhancement system based on multi-temporal phases provided in this embodiment of the invention can execute the low-resolution remote sensing image enhancement method based on multi-temporal phases described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0116] In summary, the multi-temporal low-resolution remote sensing image enhancement method and system provided by this invention have the following advantages compared with the prior art:

[0117] (1) This invention solves the problems of poor robustness of the model to changes in the real environment, insufficient feature extraction, inaccurate use of temporal information, easy loss of reconstruction details and imprecise training target by innovative data augmentation, feature extraction, temporal fusion, decoding reconstruction and loss function design, thereby significantly improving the quality and realism of the enhanced image.

[0118] (2) Through innovative virtual temporal data enhancement, the model can adapt to more complex real-world changes (seasons, lighting, clouds and fog), and its generalization ability and robustness are significantly improved.

[0119] (3) This invention combines the local detail perception capability of CNN and the global structure capture capability of Transformer through a parallel dual-stream encoder, and achieves a more comprehensive and efficient representation of remote sensing image features through adaptive fusion via a gating mechanism.

[0120] (4) The present invention can accurately evaluate and aggregate the most effective information in different time phases through the spatiotemporal decoupling attention fusion module, effectively suppress the interference of noise such as clouds, fog, and shadows, and improve the signal-to-noise ratio of the fusion features.

[0121] (5) The dense cascaded decoder in this invention effectively prevents information loss during the upsampling process by using cross-level feature reuse, making the final generated image clearer and richer in details such as texture and edge.

[0122] (6) The time-weighted hybrid loss function provided by the present invention provides a more refined supervision signal for the model, guides the model to learn information of temporally adjacent images first, conforms to physical laws, accelerates model convergence, and improves the final reconstruction accuracy.

[0123] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute a multi-temporal low-resolution remote sensing image enhancement method.

[0124] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the multi-temporal low-resolution remote sensing image enhancement method provided in the above embodiments.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-temporal based low resolution remote sensing image enhancement method, characterized in that, Comprising: Step 1: Collect low-resolution images of corresponding geographical positions of the target high-resolution image at the imaging time point of the target high-resolution image through a dynamically adjusted time window, and perform data preprocessing to form a remote sensing image enhancement data set. K Step 1: Collect low-resolution images of corresponding geographical positions of the target high-resolution image at the imaging time point of the target high-resolution image through a dynamically adjusted time window, and perform data preprocessing to form a remote sensing image enhancement data set. Wherein each training data in the remote sensing image enhanced data set comprises K a low-resolution image tile of one time phase and a target high-resolution image tile of a corresponding geographical position Y ; Step 2: K Low-resolution image patches from each time phase are input into a resolution enhancement model for feature extraction, encoding, fusion, and decoding to generate high-resolution image patches. The resolution enhancement model includes an encoder, a feature fusion module, and a decoder. Step 3: Set a hybrid loss function to guide the training of the resolution enhancement model to generate the final resolution enhancement model, the hybrid loss function Specifically: in, For structural loss, For time-weighted single-phase loss, and To balance the hyperparameters of different loss terms, For multi-scale structural similarity loss, For the first k The corresponding time point of each hour The imaging time point of the target high-resolution image. These are preset parameters. It is the first k Low-resolution image patches of each time phase Preliminary high-resolution image patches are formed through feature encoding and decoding reconstruction.

2. The multi-temporal based low resolution remote sensing imagery enhancement method according to claim 1, wherein, The data preprocessing in step 1 includes: Standard preprocessing is performed on all collected images, including cloud and fog removal, color alignment, and geometric registration; The low-resolution images after standard preprocessing are cut into tiles with a size of HxW, and the high-resolution images after standard preprocessing are cut into tiles with a size of (s x H) x (s x W), where s is the upsampling multiple.

3. The multi-temporal based low resolution remote sensing imagery enhancement method of claim 1, wherein, After data preprocessing in step 2, it also includes: Data augmentation is performed on the low-resolution image tiles of some time phases, and the data augmentation methods include one or more of the following methods: seasonal style transfer processing, light and shadow simulation processing, controllable interference synthesis processing; Wherein, the seasonal style transfer processing refers to performing seasonal style transfer on the low-resolution image tiles, and then generating new low-resolution image tiles through image hybridization; The light and shadow simulation processing refers to physically rendering the low-resolution image tiles to simulate the light and shadow effects at different times of the day; The controllable interference synthesis processing refers to randomly adding interference to the clear low-resolution tiles.

4. The multi-temporal based low resolution remote sensing image enhancement method of claim 1, wherein, The encoder of the resolution enhancement model is a parallel double-flow feature extraction encoder; For each time phase low-resolution image tile, the double-flow feature extraction encoder first reduces the feature dimension through shallow feature extraction operation, then extracts features through parallel double-flow structure, then obtains multi-feature attention weight through channel attention calculation module, and finally realizes the complementarity of local details and global context information through weighting to generate the LR feature map of each time phase low-resolution image tile.

5. The multi-temporal based low resolution remote sensing image enhancement method according to claim 4, wherein, The feature fusion module is a space-time decoupling fusion module: Spatial self-attention calculation is performed on each time phase feature map to enhance spatial structure consistency; Take the LR feature map at the latest time point as the query, and the other K -1 temporal phase of the spatially enhanced LR feature map as the key and value, cross-attention calculation, adaptive weight allocation and weighted sum to generate a fusion feature map.

6. The multi-temporal based low resolution remote sensing image enhancement method according to claim 5, wherein, The decoder is a dense cascading decoder: decomposing the total up-sampling factor s into a plurality of stages, each stage corresponding to an up-sampling block; No. l The input of the upsampling block not only includes the first l The output after upsampling one block is further combined with the output from the first to the second block through a concatenation operation. l -1 Output feature maps of all upsampled blocks.

7. The multi-temporal based low resolution remote sensing imagery enhancement method of claim 4, wherein, The parallel double-flow structure includes: CNN flow and Transformer flow; Wherein, the CNN flow is used to extract local feature information, and the Transformer flow is used to extract global feature information.

8. A multi-temporal based low resolution remote sensing image enhancement system, characterized in that, Comprising: The first processing module, the second processing module, and the third processing module; The first processing module is used to acquire data from the corresponding geographical location using a dynamically adjusted time window, based on the imaging time point of the target's high-resolution image. K Low-resolution images from different time phases were collected and preprocessed to form a remote sensing image enhancement dataset. Wherein each training data in the remote sensing image enhanced data set comprises K a low-resolution image tile of one time phase and a target high-resolution image tile of a corresponding geographical position Y ; The second processing module is used to... K Low-resolution image patches from each time phase are input into a resolution enhancement model for feature extraction, encoding, fusion, and decoding to generate high-resolution image patches. The resolution enhancement model includes an encoder, a feature fusion module, and a decoder. The third processing module is configured to set a hybrid loss function to guide training of the resolution enhancement model to generate a final resolution enhancement model, and the hybrid loss function comprises a first loss function and a second loss function. Specifically: in, For structural loss, For time-weighted single-phase loss, and To balance the hyperparameters of different loss terms, For multi-scale structural similarity loss, For the first k The corresponding time point of each hour The imaging time point of the target high-resolution image. These are preset parameters. It is the first k Low-resolution image patches of each time phase Preliminary high-resolution image patches are formed through feature encoding and decoding reconstruction.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the multi-time phase low-resolution remote sensing image enhancement method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the multi-time phase low-resolution remote sensing image enhancement method according to any one of claims 1-7.

Images