Real-time seamless splicing method for multi-source remote sensing images based on adaptive generative adversarial network

Abstract

The present application relates to the technical field of multi-source remote sensing image real-time seamless splicing, and particularly relates to a multi-source remote sensing image real-time seamless splicing method based on an adaptive generative adversarial network, comprising the following steps: a data preprocessing step, acquiring multi-temporal or multi-source remote sensing images, performing radiation correction and geometric correction on the images, an adaptive generative adversarial network model construction step, constructing a generator and a discriminator, the generator using a convolutional neural network to receive a preprocessed image feature map and generate a spliced image, and the present application dynamically adjusts model parameters in the training and reasoning process by introducing an adaptive generative adversarial network and a space-time module to adapt to different sensor data (such as optical, radar or thermal infrared) and time sequence changes.

Description

Technical Field

[0001] This invention relates to the field of real-time seamless stitching technology for multi-source remote sensing images, specifically a method for real-time seamless stitching of multi-source remote sensing images based on adaptive generative adversarial networks. Background Technology

[0002] In the field of remote sensing image mosaicking, existing technologies mainly focus on mosaicking methods for single-temporal images, typically based on geometric correction and traditional image registration techniques. Spatial alignment of images is achieved through feature point matching (such as SIFT or ORB algorithms) and affine transformations, and the mosaicking result is generated by combining pixel-level fusion algorithms (such as weighted averaging or multi-band fusion). In optical remote sensing image processing, existing technologies often employ a block-based processing strategy, using manually set parameters for radiometric and geometric correction to eliminate sensor differences and terrain effects. However, these methods have significant technical limitations when processing multi-temporal or multi-sensor images, such as optical, radar, and thermal infrared remote sensing images.

[0003] Multi-temporal images, due to factors such as illumination variations, seasonal fluctuations, and cloud cover caused by time differences, make it difficult for traditional registration methods to guarantee spatiotemporal consistency. The stitching results often exhibit seams, color mismatches, or artifacts, affecting visual coherence and analytical accuracy. Due to the heterogeneous characteristics of sensors in multi-source images (e.g., optical images depend on illumination conditions, and radar images are susceptible to noise interference), traditional methods lack adaptive fusion capabilities and cannot effectively integrate the complementary features of different source data, resulting in inconsistencies in texture, structure, and radiometric information in the stitched images. Existing technologies mostly rely on serial processing architectures, which have low computational efficiency and are difficult to meet the real-time processing needs of large-scale remote sensing data. For example, in environmental monitoring or disaster emergency response, it is impossible to achieve stitching output at the second or minute level. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for real-time seamless stitching of multi-source remote sensing images based on adaptive generative adversarial networks, thus solving the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a real-time seamless stitching method for multi-source remote sensing images based on adaptive generative adversarial networks, comprising the following steps: data preprocessing, construction of an adaptive generative adversarial network model, introduction of a spatiotemporal module, training and generation, and real-time stitching processing, using an efficient parallel computing architecture to achieve real-time stitching processing of large-scale remote sensing images.

[0006] Preferably, the data preprocessing involves, for multi-temporal images, acquiring remote sensing images from different times, performing radiometric and geometric corrections to eliminate sensor differences and illumination effects; for multi-source images, acquiring remote sensing images from different sensors, including optical, radar, or thermal infrared images, performing radiometric and geometric corrections, aligning images from different sources using registration techniques, and extracting feature maps from the multi-source images for subsequent fusion.

[0007] Preferably, the generator is constructed using a multi-scale convolutional neural network. It generates a high-resolution and seamless stitched image by extracting and fusing feature maps of multi-source remote sensing images at multiple levels. The generator's input is the feature maps of multi-temporal or multi-source images, and its output is the fused stitched image. Image details are restored through deconvolution or upsampling operations.

[0008] Preferably, the discriminator adopts a convolutional neural network structure to judge the authenticity of the stitched image output by the generator. It provides gradient feedback by calculating the adversarial loss function to optimize the generator parameters. During the training process, the discriminator continuously learns the distribution differences between real images and generated images to ensure the naturalness and consistency of the stitched image in terms of texture, color and structure.

[0009] Preferably, the spatiotemporal module constrains the spliced ​​images output by the generator during the training process to maintain spatial continuity without seams and reflect the dynamic changes of the image sequence in time. This module avoids unnatural splicing phenomena caused by factors such as season, lighting, or cloud cover by learning the spatiotemporal correlation between images.

[0010] Preferably, the training generation adopts an adaptive training strategy, which dynamically adjusts the model hyperparameters, including learning rate, batch size, or network depth, according to the sensor type, time interval, or spatial resolution of the input image, in order to enhance the model's adaptability to multi-source heterogeneous data. Adversarial loss and reconstruction loss are used during the training process.

[0011] Preferably, the real-time stitching process adopts a high-efficiency parallel computing architecture based on GPU or distributed computing framework, which divides the image into multiple sub-regions for parallel processing, and uses pipeline technology to achieve seamless connection between data reading, preprocessing, model inference and result output, ensuring that real-time stitching performance at the second or minute level can still be achieved in large-scale remote sensing data scenarios.

[0012] Preferably, the method is applicable to the stitching of multi-temporal remote sensing images. The resulting stitched image has no obvious seams and can accurately reflect the temporal change characteristics between images, such as vegetation growth, urban expansion, or disaster evolution. It is suitable for long-term environmental monitoring and dynamic analysis applications.

[0013] Preferably, the stitching of the multi-source remote sensing images results in images with high visual quality and geometric accuracy. It can integrate data from multiple sensors such as optical, radar, and thermal infrared, and retain the characteristic advantages of each source data in the stitching result. It is suitable for comprehensive analysis and decision support of multimodal remote sensing data.

[0014] Preferably, the system includes: a data preprocessing module for acquiring multi-temporal or multi-source remote sensing images and performing radiometric correction, geometric correction, image segmentation, and normalization operations; an adaptive generative adversarial network (GAN) model module, including a generator and a discriminator, wherein the generator uses a convolutional neural network to generate stitched images, and the discriminator is used to evaluate the realism of the stitched images; a spatiotemporal module for introducing spatiotemporal consistency constraints to optimize the stitching results; a training module for performing adversarial training and adaptive parameter adjustment; and a real-time stitching processing module that employs a parallel computing architecture to achieve high-speed image processing. The system works collaboratively to achieve high-precision, real-time, seamless stitching of multi-source remote sensing images.

[0015] Compared with the prior art, the beneficial effects of the present invention are:

[0016] This invention provides a method for real-time seamless stitching of multi-source remote sensing images based on adaptive generative adversarial networks:

[0017] 1. This invention introduces an adaptive generative adversarial network and a spatiotemporal module to dynamically adjust model parameters during training and inference to adapt to different optical, radar, or thermal infrared and time-series variations. The spatiotemporal consistency loss constraint generator outputs a smooth spatial transition and a coherent temporal change, avoiding unnatural stitching phenomena caused by seasonality, lighting, or cloud cover. This results in visually seamless stitched images that accurately reflect dynamic changes in ground features, improving stitching accuracy and reliability, and making it suitable for long-term environmental monitoring. 2. This invention employs a highly efficient parallel computing architecture based on GPUs or distributed computing frameworks to segment images into multiple sub-regions for parallel processing. Combined with pipeline technology to optimize data flow, it significantly reduces processing latency, enabling the stitching of GB-level data within seconds or minutes. This supports disaster emergency response and large-scale operational applications, improving the timeliness of remote sensing data processing. 3. This invention uses a multi-scale convolutional neural network to fuse feature maps from multiple sources to generate high-resolution stitched results. The discriminator optimizes image realism through adversarial training. The adaptive training strategy combines adversarial loss, reconstruction loss, and spatiotemporal consistency loss to enhance the model's ability to handle heterogeneous data. This results in stitched results that are naturally consistent in texture, color, and structure, providing reliable support for multimodal remote sensing data analysis and making it suitable for environmental monitoring and disaster assessment. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the observation process of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0022] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "configuration" should be interpreted broadly. For example, they can refer to a fixed connection or configuration, a detachable connection or configuration, or an integral connection or configuration. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0023] like Figure 1As shown, the present invention proposes a real-time seamless stitching method for multi-source remote sensing images based on adaptive generative adversarial networks, comprising the following steps: a data preprocessing step, acquiring multi-temporal or multi-source remote sensing images, performing radiometric and geometric corrections on the images, and segmenting the images; an adaptive generative adversarial network model construction step, constructing a generator and a discriminator, the generator using a convolutional neural network to receive the preprocessed image feature maps and generate stitched images, the discriminator classifying the generated stitched images to determine their authenticity and providing feedback signals; the introduction of a spatiotemporal module, ensuring the consistency of the stitched images in space and time by introducing spatiotemporal consistency constraints; training generation, optimizing the parameters of the generator and discriminator using adversarial training, and dynamically adjusting the model parameters through an adaptive training strategy to improve robustness; and real-time stitching processing, employing an efficient parallel computing architecture to achieve real-time stitching processing of large-scale remote sensing images.

[0024] In practical application, during the data preprocessing stage, after acquiring multi-temporal or multi-source remote sensing images, radiometric correction is first performed to eliminate radiometric distortion caused by atmospheric conditions, sensor response differences, or illumination variations. Geometric correction is then performed to correct geometric deviations caused by terrain undulations, sensor attitude errors, or projection distortion. For multi-temporal images, further segmentation is performed, dividing large-scale images into several smaller blocks to facilitate parallel processing. Standardized pixel value distribution reduces the impact of illumination and climate fluctuations. For multi-source images, feature-level registration aligns images from different sensors (such as optical and radar) and improves image quality. By extracting multi-scale feature maps, in the construction of the adaptive generative adversarial network model, the generator uses a convolutional neural network (such as the U-Net structure) to receive the preprocessed feature maps, fuses multi-source information through an encoding-decoding mechanism and generates a preliminary stitched image. At the same time, it combines adaptive learning rate adjustment and multi-task loss (such as adversarial loss, pixel-level reconstruction loss and spatiotemporal consistency loss) to optimize network parameters, distributes image data to multiple computing units, utilizes the parallel computing capabilities of GPUs to process multiple sub-regions simultaneously, and generates seamless and high-precision stitched images, which are suitable for large-scale environmental monitoring or disaster emergency response.

[0025] Data preprocessing involves acquiring remote sensing images from different times for multi-temporal images, performing radiometric and geometric corrections to eliminate sensor differences and illumination effects, and acquiring remote sensing images from different sensors, including optical, radar, or thermal infrared images, after performing radiometric and geometric corrections, aligning images from different sources using registration techniques, and extracting feature maps from the multi-source images for subsequent fusion.

[0026] In practical applications, for multi-temporal imagery, after acquiring remote sensing data from different time points, radiometric correction uses an atmospheric correction model to eliminate interference from aerosols, water vapor, etc., while geometric correction uses a digital model and ground control points for precise registration. Subsequently, overlapping areas are segmented to reduce illumination differences caused by seasonal or diurnal variations. For multi-source imagery, after acquiring optical, radar, or thermal infrared data, radiometric correction is performed separately for each sensor's characteristics (e.g., radiometric calibration for radar and atmospheric correction for optical sensors). Geometric correction uses affine transformations or polynomial models to unify the coordinate system. Then, feature point matching achieves spatial alignment of the multi-source imagery. Finally, a convolutional feature extraction network generates a multi-source feature map. This feature map integrates complementary information from different sensors, fundamentally reducing the sources of stitching errors.

[0027] The generator is constructed using a multi-scale convolutional neural network. It extracts and fuses feature maps from multi-source remote sensing images at multiple levels to generate high-resolution and seamless stitched images. The generator takes feature maps from multiple time phases or multiple sources as input and outputs the fused stitched image. It also restores image details through deconvolution or upsampling operations.

[0028] In practical use, the encoder extracts multi-scale features from the input image through multiple convolutional layers and downsampling operations, from low-level edge textures to high-level semantic information. The decoder gradually restores the spatial resolution through deconvolution or upsampling layers, and combines skip connections to fuse the encoder's feature map with the corresponding layer of the decoder to preserve detailed information. For multi-source images, after receiving the feature maps of each source image, the generator fuses the features of different sources through attention mechanisms or channel weighting to ensure that the fused features contain both the color authenticity of optical images and the terrain structure of radar images. The generator finally outputs a stitched image, and its training objective is to minimize the difference from the real image while meeting the authenticity requirements of the discriminator. The multi-scale design enables the generator to process remote sensing data of different resolutions.

[0029] The discriminator uses a convolutional neural network structure to judge the authenticity of the stitched image output by the generator. It provides gradient feedback by calculating the adversarial loss function to optimize the generator parameters. During training, the discriminator continuously learns the distribution differences between real images and generated images to ensure the naturalness and consistency of the stitched image in terms of texture, color and structure.

[0030] In practical use, the input is either a real stitched image or a stitched image output by the generator. Image features are extracted through multiple convolutional and fully connected layers, and a scalar value is output to represent the probability that the input image is real. The discriminator distinguishes the distribution differences between real and generated images. Real images usually have natural texture transitions and consistent lighting, while generated images may have seams or color mismatches. The adversarial loss provided by the discriminator guides the generator to optimize its parameters through backpropagation, so that the generated image gradually approaches the real distribution. The discriminator can also combine multi-scale discrimination strategies to evaluate image quality at different resolutions, further improving the visual realism of the stitched result.

[0031] The spatiotemporal module constrains the spliced ​​images output by the generator during training to maintain spatial continuity without seams and reflect the dynamic changes of the image sequence in time. By learning the spatiotemporal correlation between images, this module avoids unnatural splicing caused by factors such as season, lighting, or cloud cover.

[0032] In practical use, the spatiotemporal module typically calculates two parts: spatial loss and temporal loss. It estimates the motion vector between adjacent time phases using optical flow and constrains the changes in the generated image in the time dimension to conform to physical laws (such as the gradual change of vegetation index). The module integrates the spatiotemporal loss into the overall target data of the generative adversarial network. Through joint optimization, the generator not only focuses on the quality of individual images but also takes into account the coherence of the sequence of images, thereby effectively avoiding unnatural stitching caused by factors such as cloud cover and seasonal changes.

[0033] The training generation adopts an adaptive training strategy, which dynamically adjusts the model hyperparameters, including learning rate, batch size, or network depth, according to the sensor type, time interval, or spatial resolution of the input image, to enhance the model's adaptability to multi-source heterogeneous data. Adversarial loss and reconstruction loss are used during the training process.

[0034] In practical use, during training, the model automatically adjusts hyperparameters based on the characteristics of the input data (such as sensor type and spatiotemporal resolution). For high-noise radar data, the learning rate is appropriately reduced to avoid gradient explosion. For multi-temporal data, the weights of the temporal loss are adjusted according to the time interval. The reconstruction loss is calculated by comparing the pixel differences between the generated image and the real image. The spatiotemporal consistency loss is calculated by a dedicated module. These losses are jointly optimized by gradient descent Adam or SGD. The model can adapt to different data scenarios and improve its generalization ability on multi-source heterogeneous remote sensing data.

[0035] The real-time stitching process employs a high-efficiency parallel computing architecture based on GPUs or distributed computing frameworks. The image is divided into multiple sub-regions for parallel processing, and pipeline technology is used to achieve seamless integration of data reading, preprocessing, model inference, and result output, ensuring that real-time stitching performance at the second or minute level can still be achieved in large-scale remote sensing data scenarios.

[0036] In practical use, the system employs GPU clusters or a distributed architecture to segment large-scale input remote sensing images into multiple sub-blocks, which are then distributed to different computing nodes for parallel processing. Each sub-block independently performs data loading, preprocessing, model inference, and result generation tasks. Data synchronization is achieved through message queues or shared memory. Pipeline technology divides the entire processing flow into data reading, correction, stitching, and output, with each stage executed in parallel, significantly reducing overall latency. Utilizing the parallel computing capabilities of GPUs, the forward inference speed of convolutional networks is greatly improved, enabling stitching to be completed within seconds even when processing gigabyte-level remote sensing data, meeting real-time monitoring requirements.

[0037] The method is applicable to the stitching of multi-temporal remote sensing images. The final stitched image has no obvious seams and can accurately reflect the temporal change characteristics between images, such as vegetation growth, urban expansion or disaster evolution. It is suitable for long-term environmental monitoring and dynamic analysis applications.

[0038] In practical use, during the stitching process, the generator not only integrates spatial information, but also captures temporal dependencies through recurrent neural networks (RNN) or 3D convolution. The stitched images are visually seamless and can clearly reflect changes in land features over time, such as crop growth cycles, water area fluctuations, or urban building expansion.

[0039] Multi-source remote sensing image stitching produces images with high visual quality and geometric accuracy. It can integrate data from multiple sensors such as optical, radar, and thermal infrared, and retain the characteristic advantages of each source data in the stitching result. It is suitable for comprehensive analysis and decision support of multimodal remote sensing data.

[0040] In practical use, during the stitching process, the generator integrates the rich colors of optical images, the penetration capability of radar images (such as clouds and vegetation), and the temperature information of thermal infrared images. It highlights the advantages of each source data through feature weighting or attention mechanisms, while the discriminator ensures that the fused image is visually natural and consistent, with no obvious differences between sensors. The stitching result is not only seamless, but also retains the comprehensive information of multi-source data, making it suitable for remote sensing analysis in complex scenarios, such as damaged area detection or multi-resource surveys in disaster assessment.

[0041] The system comprises several modules: a data preprocessing module for acquiring multi-temporal or multi-source remote sensing images and performing radiometric correction, geometric correction, image segmentation, and normalization; an adaptive generative adversarial network (GAN) model module, including a generator and a discriminator (the generator uses a convolutional neural network to generate stitched images, and the discriminator evaluates the realism of the stitched images); a spatiotemporal module for introducing spatiotemporal consistency constraints to optimize the stitching results; a training module for performing adversarial training and adaptive parameter adjustment; and a real-time stitching processing module that employs a parallel computing architecture to achieve high-speed image processing. The system works collaboratively to achieve high-precision, real-time, seamless stitching of multi-source remote sensing images.

[0042] In practical use, the data preprocessing module is responsible for image acquisition and correction to ensure the quality of input data. The adaptive generative adversarial network module optimizes the stitching effect through the generator and discriminator. The spatiotemporal module introduces external constraints to improve the consistency of results and updates network parameters through backpropagation and adaptive strategies. The real-time stitching processing module uses parallel computing to accelerate the overall process. All modules are connected through standardized interfaces to achieve seamless transmission of data flow and control flow. Finally, the system can efficiently and automatically complete the real-time seamless stitching of multi-source remote sensing images, which is suitable for large-scale operational applications.

[0043] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for real-time seamless stitching of multi-source remote sensing images based on adaptive generative adversarial networks, characterized in that, Includes the following steps: Step 1: Data preprocessing. Acquire multi-temporal or multi-source remote sensing images, perform radiometric and geometric corrections on the images, and segment the images. Step 2: Construction of the adaptive generative adversarial network model, including a generator and a discriminator. The generator uses a convolutional neural network to receive preprocessed image feature maps and generate stitched images. The discriminator classifies the generated stitched images to determine their authenticity and provides feedback signals. Step 3: Introduction of the spatiotemporal module. Spatiotemporal consistency constraints are introduced to ensure the consistency of the stitched images in space and time. Step 4: Training and Generation. The parameters of the generator and discriminator are optimized using adversarial training, and the model parameters are dynamically adjusted through an adaptive training strategy to improve performance. Step 5: Real-time stitching processing, using an efficient parallel computing architecture to achieve real-time stitching processing of large-scale remote sensing images; The data preprocessing involves acquiring remote sensing images from different times for multi-temporal images, performing radiometric and geometric corrections to eliminate sensor differences and illumination effects, and acquiring remote sensing images from different sensors, including optical, radar, or thermal infrared images, performing radiometric and geometric corrections, aligning images from different sources using registration techniques, and extracting feature maps from the multi-source images for subsequent fusion. The generator is constructed using a multi-scale convolutional neural network. It extracts and fuses feature maps from multi-source remote sensing images at multiple levels to generate high-resolution and seamless stitched images. The generator's input is the feature maps of multi-temporal or multi-source images, and its output is the fused stitched image. It also restores image details through deconvolution or upsampling operations. The discriminator adopts a convolutional neural network structure to judge the authenticity of the stitched image output by the generator. It provides gradient feedback by calculating the adversarial loss function to optimize the generator parameters. During the training process, the discriminator continuously learns the distribution differences between real images and generated images to ensure the naturalness and consistency of the stitched image in terms of texture, color and structure. The spatiotemporal module constrains the spliced ​​images output by the generator during training to maintain spatial continuity without seams and reflects the dynamic changes of the image sequence in time. By learning the spatiotemporal correlation between images, this module avoids unnatural splicing phenomena caused by seasonality, lighting, or cloud cover.

2. The method for real-time seamless stitching of multi-source remote sensing images based on adaptive generative adversarial networks according to claim 1, characterized in that: The training generation adopts an adaptive training strategy, which dynamically adjusts the model hyperparameters, including learning rate, batch size, or network depth, according to the sensor type, time interval, or spatial resolution of the input image, in order to enhance the model's adaptability to multi-source heterogeneous data. Adversarial loss and reconstruction loss are used during the training process.

3. The method for real-time seamless stitching of multi-source remote sensing images based on adaptive generative adversarial networks according to claim 1, characterized in that: The real-time stitching process adopts a high-efficiency parallel computing architecture based on GPU or distributed computing framework, which divides the image into multiple sub-regions for parallel processing, and uses pipeline technology to achieve seamless connection between data reading, preprocessing, model inference and result output, ensuring that real-time stitching performance at the second or minute level can still be achieved in large-scale remote sensing data scenarios.

4. The method for real-time seamless stitching of multi-source remote sensing images based on adaptive generative adversarial networks according to claim 1, characterized in that: The method is applicable to the stitching of multi-temporal remote sensing images. The resulting stitched images have no obvious seams and can accurately reflect the temporal change characteristics between images, vegetation growth, urban expansion or disaster evolution processes. It is suitable for long-term environmental monitoring and dynamic analysis applications.

5. The method for real-time seamless stitching of multi-source remote sensing images based on adaptive generative adversarial networks according to claim 1, characterized in that: The stitching of the multi-source remote sensing images results in images with high visual quality and geometric accuracy. It can integrate data from multiple sensors, including optical, radar, and thermal infrared sensors, and retain the characteristic advantages of each source data in the stitching result. It is suitable for comprehensive analysis and decision support of multimodal remote sensing data.

Images