Optical and sar paired image generation method and system

Abstract

This invention discloses a method and system for generating paired optical and SAR images. The method includes: Step S1: constructing and preprocessing an optical and SAR image dataset; Step S2: constructing an optical and SAR paired image generation model based on feature decoupling technology; Step S3: constructing a loss function to constrain the image generation process and training the model using data from the dataset; Step S4: inputting the optical image to be paired and the SAR image into the trained optical and SAR paired image generation model to obtain paired optical and SAR images. This invention can be trained without relying on paired optical and SAR image datasets, and can simultaneously achieve image conversion from optical to SAR and from SAR to optical. While reducing the cost of data construction and model training, the generated images maintain good consistency in detail and overall structure.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method and system for generating paired optical and SAR images. Background Technology

[0002] In the field of modern remote sensing, optical images and SAR (Synthetic Aperture Radar) images exhibit significant differences due to variations in imaging mechanisms and environmental factors. For example, while optical images can capture more information, they are greatly affected by lighting and weather conditions; SAR images, on the other hand, are not constrained by these conditions and can acquire surface information under all weather conditions. Utilizing the complementary information from these two modalities to fuse and analyze multimodal remote sensing data has significant implications for many fields, including military reconnaissance, disaster monitoring, urban planning, and environmental research.

[0003] However, acquiring paired optical-SAR images of corresponding targets simultaneously is extremely difficult, mainly due to the following limitations: First, optical images rely on the reflection of sunlight off the ground, and their quality and clarity are affected by factors such as light intensity, atmospheric transparency, and cloud cover, making it challenging to acquire continuous, directly paired, high-quality optical images at the same geographical location. Second, even when optical and SAR images are captured at the same location, differences in satellite transit time, orbital position, and imaging angle make it very difficult to acquire corresponding optical and SAR images simultaneously or nearly simultaneously. Finally, in existing remote sensing satellite systems, few satellites are equipped with both optical and SAR sensors, and even those that exist have limitations in imaging frequency, coverage, and resolution due to technological and equipment constraints, further increasing the difficulty of acquiring paired image data.

[0004] In recent years, some techniques have used image translation to generate paired optical-SAR images. These techniques can be divided into two categories: paired image-based methods and methods that do not require paired images. Paired image-based methods, such as pix2pix, rely on paired training data, where each pair of images corresponds to the same scene in both optical and SAR modes. The main problem with this approach is the extreme difficulty and high cost of acquiring large amounts of high-quality paired optical-SAR image data. Furthermore, due to the limitation of paired training data, the generalization ability of these methods may be limited when faced with unfamiliar scenes or dynamically changing environments. In contrast, paired image-free translation methods, such as CycleGAN, allow the model to perform style transfer between images without any paired training data. However, the main challenges of this approach include effectively capturing and preserving the fine features of the source images, especially when direct constraints are lacking; the model may perform poorly in maintaining detail and overall structural consistency. Moreover, because it is not directly guided by paired data, CycleGAN may produce inaccurate or inconsistent translation results when handling complex scenes and large-scale spatial transformations.

[0005] For example, prior art 1: Patent publication number CN116721264A, entitled "SAR to optical image translation method, system, device and medium based on improved CGAN", discloses: Several sets of SAR images and corresponding optical images from different scenes were randomly collected; The SAR image is input into a generator network with a style-based recalibration module to obtain the corresponding optical image; The acquired optical image and the collected optical image are input into a multi-scale discriminator network for comparison to obtain a similarity score between the two. The generator network and discriminator network are continuously optimized, and steps 2 and 3 are repeated, using similarity scores as the evaluation criteria, until the highest quality optical image is obtained.

[0006] This invention proposes a pix-to-pixel SAR-to-optical image translation method, system, apparatus, and medium. The invention improves the quality and realism of the translated optical image by minimizing color differences between the output image and the real optical image through a pattern-based calibration module. A multi-scale strategy is introduced into the discriminator to enhance the texture and edge information of the image, considering both local and global levels, further improving the quality and sharpness of the generated optical image. However, this invention requires paired SAR and optical image datasets for training, demanding a large amount of high-quality paired optical-SAR image data, which is difficult to meet practical needs. Furthermore, this invention can only perform a one-way SAR-to-optical image translation process, limiting its application scenarios.

[0007] Prior art 2: The invention patent with publication number CN111047525A, entitled "Translation method from SAR remote sensing image to optical remote sensing image", discloses: (1) Construct an adversarial neural network; (2) Acquire multiple SAR remote sensing images of the same size and their corresponding optical remote sensing images; (3) Based on the different ground features to which multiple SAR remote sensing images belong, the training set is divided into several subsets, with each subset corresponding to one ground feature; (4) For any subset, obtain the adversarial neural network prediction model corresponding to that subset.

[0008] This invention proposes a method for translating SAR remote sensing images into optical remote sensing images. By employing an adversarial neural network (ANN) prediction model, it can generate more refined translation results with high accuracy and good visual quality, making the method applicable to the translation of high-resolution SAR remote sensing images. However, the translation process from SAR to optical remote sensing images is a one-to-many process, and the ANN can only learn one-to-one mappings, often resulting in noisy generated images. Furthermore, this invention can only perform unidirectional translation from SAR to optical images, limiting its application scenarios.

[0009] Existing technology 3: CN111784560A A method for bidirectional translation of SAR and optical images based on cascaded residual generative adversarial networks (1) Select a portion of SAR images and their corresponding optical images from the dataset and strictly register them; (2) Preprocess all SAR and optical images to be used, including noise reduction, normalization and slicing; (3) Supervised learning is adopted, and strictly paired SAR and optical images are fed into a multi-scale residual generative adversarial network to iteratively train the model; (4) Using unsupervised learning, the images to be tested are iteratively trained using unpaired optical or SAR images in the extended dataset to improve the translation results; This invention can efficiently convert data between two distinct sensing modes, SAR and optical, transforming commonly visible information from one image to another, and generating new content based on prior knowledge. However, this invention still requires the pre-preparation of rigorously registered SAR and optical images, and the data preparation process is costly. Summary of the Invention

[0010] The technical problem to be solved by the embodiments of the present invention is to provide a method and system for generating optical-SAR paired images, so as to generate high-quality optical-SAR paired images based on unpaired data of the same target.

[0011] To address the aforementioned technical problems, this invention proposes a method for generating paired optical and SAR images, comprising: Step S1: Construct optical and SAR image datasets and perform preprocessing; Step S2: Construct an optical and SAR paired image generation model based on feature decoupling technology; Step S3: Construct a loss function to constrain the image generation process, and train the model using data from the dataset; Step S4: Input the optical image to be paired and the SAR image into the trained optical and SAR paired image generation model to obtain the optical and SAR paired image.

[0012] Accordingly, embodiments of the present invention also provide an optical and SAR paired image generation system, comprising: Dataset building module: Constructs and preprocesses optical and SAR image datasets; Model building module: Constructs an optical and SAR paired image generation model based on feature decoupling technology; Training module: Constructs a loss function to constrain the image generation process, and trains the model using data from the dataset; Generation module: Input the optical image to be paired and the SAR image into the trained optical and SAR paired image generation model to obtain the optical and SAR paired image.

[0013] The beneficial effects of this invention are as follows: The generated paired images exhibit good performance in maintaining detail and overall structural consistency. This invention employs a feature-based decoupling technique, which performs well in one-to-many conversion processes such as SAR to optical, reducing noise introduction and improving generation quality. This invention can simultaneously achieve optical to SAR and SAR to optical conversions, merging them into a single framework without the need to train two different models, thus reducing the complexity of model training and inference. This invention does not require training with already paired optical-SAR image datasets, significantly reducing the additional overhead of constructing paired datasets. Attached Figure Description

[0014] Figure 1 This is a schematic flowchart of the optical and SAR paired image generation method according to an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of the processing flow of the optical and SAR paired image generation model according to an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] In this embodiment of the invention, directional indicators (such as up, down, left, right, front, back, etc.) are only used to explain the relative positional relationship and movement of each component in a specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0018] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features.

[0019] Please refer to Figure 1 The optical-SAR paired image generation method of this invention includes steps S1 to S4. This invention is used to support the training data requirements of tasks such as multimodal target detection based on optical-SAR paired images.

[0020] Step S1, Data Preparation and Preprocessing: Construct a sufficiently diverse dataset of optical and SAR images and perform preprocessing to help the algorithm understand and capture the inherent relationships and transformation rules between the two modalities. Although these images are not strictly one-to-one pairs, their overall statistical properties should reflect the representational characteristics of the same target under different observation methods. The optical and SAR images in the dataset of this invention do not require pairing.

[0021] In one implementation, step S1 includes steps S11 to S15.

[0022] Step S11: Collect a large number of optical images (such as RGB satellite images) and SAR images (such as radar echo images) covering different scenes and geographical regions to construct an optical and SAR image dataset. The dataset contains images of different scenes and geographical regions, but importantly, they cover similar spatial extents and scene types, although a one-to-one correspondence between the images is not required.

[0023] Step S12, Data Filtering and Cleaning: Perform quality control on the dataset, removing blurry, noisy, or obviously abnormal images. Ensure that there are no obvious skew or projection differences in each dataset; if necessary, perform corrections and unify the coordinate system.

[0024] Step S13, Data Standardization: Preprocess the optical and SAR images in the dataset (including but not limited to normalizing brightness and contrast, removing background noise, enhancing image edges or specific features, etc.) to ensure that the images entering the model have similar data distribution characteristics. Step S14, Data Splitting: Divide the entire dataset into a training set, a validation set, and a test set to facilitate model training, hyperparameter tuning, and performance evaluation.

[0025] Step S15, Data Augmentation: Crops or scales the images in the dataset to a uniform size and converts them to the file format required by the model. Furthermore, data augmentation techniques such as random rotation, flipping, and scaling are applied to increase the model's generalization ability and adaptability to different viewpoints and scales, especially in unpaired cases. This helps the model learn a wider range of image transformation rules.

[0026] Step S2: Construct an optical and SAR paired image generation model based on feature decoupling technology. This model design allows for the simulation of the mutual conversion between optical and SAR images by effectively decomposing and recombining features from different modalities, even without direct paired data, thereby achieving virtual paired image generation from unpaired images. Specifically, as... Figure 2 As shown, the process is as follows: (1) Feature Decoupling Encoding: Three different encoders, including an optical-specific encoder, a SAR-specific encoder, and a modality-shared encoder, are used to decouple optical and SAR features. The optical-specific encoder extracts high-frequency details unique to optical images; the SAR-specific encoder extracts specific features from SAR images; and the modality-shared encoder extracts shared features from both optical and SAR images that are unaffected by specific imaging methods, such as the shape and outline of objects and structural information. Each type of original image is processed through its corresponding specific encoder and shared encoder to obtain optical-specific features, optical shared features, SAR-specific features, and SAR shared features, respectively.

[0027] (2) Feature fusion and reconstruction: The decoupled features are recombined. For example, the shared features of the optical image are combined with the specific features of the SAR image and input into the optical decoder; at the same time, the shared features of the SAR image are combined with the specific features of the optical image and input into the SAR decoder. In this way, while keeping the inherent features of the target in the original mode unchanged, the mode-specific features are replaced, which is equivalent to applying a transformation of specific style, pattern and other elements.

[0028] (3) Image generation: After processing by the decoder, paired optical and SAR images can be generated. The optical decoder is responsible for converting the received mixed features into an output image that approximates the optical mode, while the SAR decoder is responsible for generating an image that approximates the SAR mode.

[0029] (4) Multi-scale residual decoupling coding: As a further implementation, optical-specific encoders, SAR-specific encoders, and modality-shared encoders can all adopt a multi-scale residual coding structure, that is, they all include multi-scale residual coding units. For the input optical image and SAR images Modal shared encoder output , Optical specific encoder output SAR-specific encoder output The superscript 'c' indicates a shared feature, and the superscript 'p' indicates a modality-specific feature. The encoded features at each level satisfy the following: ,in Represents the residual mapping. Through multi-scale features... The model can simultaneously represent the global outline, local edges, and detailed textures of targets such as ports, airports, roads, buildings, and ships.

[0030] (5) Shared Feature Distribution Alignment: To enable the modal shared encoder to extract target structural information independent of the imaging method, this invention can introduce distribution alignment constraints in the shared feature layer. For a batch of optical shared features... Shared features with SAR Calculate their mean mu(·) and covariance Sigma(·) respectively. The shared feature distribution alignment loss satisfies: .

[0031] This constraint allows the two modes to have similar structural representations in a shared feature space, reducing feature domain offset caused by sensor differences.

[0032] (6) Cross-modal attention fusion: To avoid the mixing of shared structural information and modal style information caused by simple splicing, this invention introduces a cross-modal attention mechanism in the feature fusion stage. For source modal shared features... and target modality specific features ,structure , , and calculate Therefore, the model can adaptively select the SAR scattering features or optical texture features to be injected based on the importance of the target region, edge region, and background region. Specifically: Indicates modal sharing features, Indicates modality-specific features, express and Feature dimensions, This represents the query matrix obtained from the modality-shared feature mapping. This represents the key matrix obtained by mapping specific features of the target mode. This represents the value matrix obtained by mapping specific features of the target mode. Indicates attention weights, Indicates attention fusion features, T This represents the matrix transpose operation. K superscript T This represents the transpose of K. , , They represent the methods used for generation. Q, K, V Learnable linear transformation weight matrix, This represents the normalized exponential function.

[0033] Step S3: Construct a loss function to constrain the image generation process, and train the model using data from the dataset.

[0034] To ensure that the generated images maintain good detail and overall structural consistency, this invention constrains the image generation process through reconstruction loss, cyclic loss, and adversarial loss. Specifically: Reconstruction loss: Allowing the features that are forced to decouple to reconstruct their original image can guarantee the feature extraction capability of the encoder and the image generation capability of the decoder.

[0035] ; in, , , These represent mode-shared encoders, optically specific encoders, and SAR-specific encoders, respectively. and These represent the optical and SAR decoders, respectively. and These represent optical and SAR images, respectively. By constraining the distance between the reconstructed image and the original image, the capabilities of the model's encoder and decoder are ensured.

[0036] Cyclic Loss: Since paired optical-SAR images are not used for training, there is a lack of effective supervision over the generated new images. This invention uses cyclic loss for constraint, i.e., rgb->sar->rgb and sar->rgb->sar, to generate images, and applies distance constraints to the original image.

[0037] ; , These represent visible light images converted to SAR images and then back to visible light images, and SAR images converted to visible light images and then back to SAR images, respectively.

[0038] Adversarial Loss: Constraining solely by reconstruction loss and cyclic loss leads to image blurring; therefore, this invention introduces adversarial loss for constraint. ; , These represent SAR images converted from visible light images and visible light images converted from SAR images, respectively. , These represent the visible light image discriminator and the SAR image discriminator, respectively.

[0039] Step S4: Input the optical image to be paired and the SAR image into the trained optical and SAR paired image generation model to obtain optical and SAR paired images (i.e., optical images and SAR images that are paired with each other).

[0040] This invention decomposes unpaired data of the same target into modal-shared and modal-specific features using feature decoupling technology. The decoupled features are then recombined, and generation constraints are applied. Finally, paired optical-SAR images are generated through two different decoders. This invention eliminates the need for training with paired optical-SAR images and performs well in maintaining detail and overall structural consistency, enabling accurate matching and mutual conversion between optical-SAR images.

[0041] The optical and SAR paired image generation system of this invention includes a dataset construction module, a model construction module, a training module, and a generation module.

[0042] Dataset building module: Builds optical and SAR image datasets and performs preprocessing.

[0043] Model building module: Constructs an optical and SAR paired image generation model based on feature decoupling technology.

[0044] Training module: Constructs a loss function to constrain the image generation process, and trains the model using data from the dataset.

[0045] Generation module: Input the optical image to be paired and the SAR image into the trained optical and SAR paired image generation model to obtain the optical and SAR paired image.

[0046] As one implementation method, the dataset construction module includes: Data collection unit: Collects optical and SAR images containing different scenes and geographical areas to construct optical and SAR image datasets; Data filtering and cleaning unit: performs quality control on the dataset, removing blurry, noisy, or obviously abnormal images; Data standardization unit: Preprocesses optical and SAR images in the dataset to ensure that the images entering the model have similar data distribution characteristics; Data partitioning unit: Divides the entire dataset into training set, validation set, and test set; Data augmentation unit: Crops or scales images within the dataset to a uniform size and converts them to the file format required by the model.

[0047] As one implementation method, the model processes the input image according to the following steps: Three different encoders are used to decouple optical and SAR features. The encoders include an optical-specific encoder, a SAR-specific encoder, and a mode-sharing encoder. The optical-specific encoder is used to extract high-frequency details from the optical image to obtain optical-specific features. The SAR-specific encoder is used to extract corresponding specific features from the SAR image to obtain SAR-specific features. The mode-sharing encoder is used to extract optical-shared features and SAR-shared features from both optical and SAR images. The decoupled features are recombined, and feature fusion and reconstruction are performed to obtain hybrid features; A decoder is used to process the mixed features and generate optical and SAR paired images. The decoder includes an optical decoder and a SAR decoder. The optical decoder is responsible for converting the received mixed features into an output image that approximates the optical mode, while the SAR decoder is responsible for generating an image that approximates the SAR mode.

[0048] As one implementation method, the feature fusion and reconstruction specifically includes: Using modality-shared features as the query vector and target modality-specific features as the key and value vectors, attention weights are calculated to obtain attention fusion features; where the attention weights satisfy: ; Attention fusion features satisfy: ; in: Indicates modal sharing features, Indicates modality-specific features, Indicates the feature dimension.

[0049] This invention can be trained without being based on paired optical and SAR image datasets, and can simultaneously achieve optical to SAR and SAR to optical image conversion. While reducing the cost of data construction and model training, the generated images perform well in maintaining details and overall structural consistency.

[0050] Current visible light-SAR paired image generation techniques either require training with paired datasets and often only support unidirectional transformation, or they employ CycleGAN-based image style transfer. However, these methods introduce significant generation noise in the one-to-many context of SAR optical images, resulting in poor performance in maintaining detail and overall structural consistency. This invention adds feature decoupling technology to the optical-SAR paired image generation process, overcoming the shortcomings of both of the above approaches. It can easily generate high-quality optical-SAR paired images to support the training data requirements for tasks such as multimodal target detection based on optical-SAR paired images.

[0051] Example 1 illustrates the optical and SAR paired image generation method of the present invention using typical remote sensing target scenes such as ports, airports, roads, buildings, and ships as examples.

[0052] First, optical remote sensing image datasets and SAR remote sensing image datasets covering similar geographical areas and scene types are collected. The optical remote sensing images can be RGB satellite images, and the SAR remote sensing images can be single-polarization or multi-polarization radar echo images. Due to differences in satellite transit time, imaging angle, and sensor type, the collected optical images and SAR images are not required to be registered one-to-one, nor are they required to strictly correspond to the same shooting time. They are only required to have similar statistical characteristics in terms of target category, land cover type, and spatial distribution.

[0053] Then, the optical and SAR images are preprocessed. For the optical images, brightness normalization, contrast adjustment, cloud or anomalous region removal, and cropping are performed. For the SAR images, speckle noise suppression, amplitude normalization, anomalous echo region removal, and cropping are performed. The processed images are then uniformly cropped or scaled to a preset size, such as 256×256 pixels or 512×512 pixels, and divided into training, validation, and test sets.

[0054] Next, the optical images from the training set are input into the modality-shared encoder and the optical-specific encoder to obtain optical shared features and optical-specific features, respectively. Similarly, the SAR images from the training set are input into the modality-shared encoder and the SAR-specific encoder to obtain SAR shared features and SAR-specific features, respectively. Through feature fusion, the optical shared features and SAR-specific features are combined and input into the SAR decoder to generate a SAR image corresponding to the input optical image on the target structure. Simultaneously, the SAR shared features and optical-specific features are combined and input into the optical decoder to generate an optical image corresponding to the input SAR image on the target structure.

[0055] During training, the original modal images can be reconstructed by reconstructing loss-constrained optical images and SAR images respectively; the original structure of optical images is maintained after "optical→SAR→optical" transformation by cyclic loss constraints, and the original structure of SAR images is maintained after "SAR→optical→SAR" transformation; the distribution of optical images and SAR images generated by adversarial loss constraints are close to the distribution of real optical images and real SAR images respectively.

[0056] After training, the optical or SAR image to be paired is input into the trained model to generate the corresponding SAR or optical image. The resulting optical-SAR paired images can be used as training samples for multimodal target detection, target recognition, change detection, or image fusion tasks, thereby reducing the cost of manually constructing strictly registered optical-SAR paired datasets.

[0057] Example 2 illustrates the application of this invention using flood disaster area monitoring as an example. During floods, optical remote sensing images are easily affected by cloud cover, rainfall, and lighting conditions, making it difficult to obtain clear images in a timely manner. While SAR images can be imaged under cloudy, rainy, and nighttime conditions, their visual representation differs significantly from optical images, making them unsuitable for direct integration with existing optical image analysis models.

[0058] When using the method of this invention, historical optical and SAR images of the disaster area and similar surface areas are first collected to construct a non-paired training dataset. The modality-sharing encoder of this invention extracts structural information such as rivers, roads, buildings, and water body boundaries; the optical-specific encoder extracts optical features such as color and texture; and the SAR-specific encoder extracts SAR features such as radar scattering and echo intensity. After training, SAR images acquired during the disaster can be input into the model to generate optical modal images corresponding to the structure of those SAR images; alternatively, optical images from before the disaster can be input into the model to generate corresponding SAR modal images.

[0059] The above methods enable the formation of optical-SAR paired samples even in the absence of strictly paired data, assisting in tasks such as disaster area identification, water body extent extraction, and pre- and post-disaster change detection, thereby improving the training data scale and generalization ability of multimodal remote sensing analysis models.

[0060] Comparative Example 1: Using a paired supervised image translation method: As a comparison, a supervised image translation method based on paired training data is employed, such as a SAR-to-optical image translation method based on a pix2pix model. This method requires the pre-acquisition of strictly registered SAR and optical images, using one-to-one corresponding image pairs as training samples. During training, the model directly learns the mapping relationship between the input SAR image and the corresponding optical image.

[0061] Compared with the present invention, this comparison model has at least the following shortcomings: First, it requires a large number of strictly registered optical-SAR image pairs, resulting in high data acquisition and calibration costs; Second, since training relies on fixed paired samples, its generalization ability is easily limited when facing unseen scenes, different imaging angles, or different geographical areas; Third, it usually only achieves one-way conversion from SAR to optical or from optical to SAR. If two-way conversion is required, it often requires training another model.

[0062] This invention does not require strict pairing of optical and SAR images in the training data. By decoupling and recombining modal shared features and modal-specific features, it achieves bidirectional generation from optical to SAR and from SAR to optical within the same framework, reducing the cost of pairing data construction and model training.

[0063] Comparative Example 2: Using an undecoupled cycle-consistent image translation method: As another example, a standard CycleGAN-like unpaired image translation method is used, treating the optical and SAR image domains as two separate image domains. Cyclic consistency constraints are used to achieve mutual conversion between optical and SAR images. While this method does not require strictly paired data, it typically learns the overall mapping relationship between the two image domains directly, without explicitly distinguishing between modality-shared structural features and modality-specific imaging features.

[0064] Compared with the present invention, this comparative example has at least the following shortcomings: First, when the imaging mechanisms of optical images and SAR images are significantly different, the model is prone to mixing target structure information with modal style information during learning, resulting in structural shifts, texture distortion, or increased noise in the generated images; Second, in the one-to-many conversion process from SAR to optical, ordinary cyclic consistency constraints are difficult to fully limit the generation results, which may lead to unstable details in the generated images; Third, for complex scenes or large-scale spatial changes, the consistency of the generated results in terms of target contours, edges, and spatial layout is weak.

[0065] This invention decouples target structural information from modal-specific information by setting up a modal shared encoder, an optical specific encoder, and a SAR specific encoder, and recombines the features of different modalities during the generation stage. This allows the generated image to maintain the overall structural consistency of the target while possessing the imaging features of the target modality, thereby improving the realism, stability, and usability of the generated image.

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating paired optical and SAR images, characterized in that, include: Step S1: Construct optical and SAR image datasets and perform preprocessing; Step S2: Construct an optical and SAR paired image generation model based on feature decoupling technology; Step S3: Construct a loss function to constrain the image generation process, and train the model using data from the dataset; Step S4: Input the optical image to be paired and the SAR image into the trained optical and SAR paired image generation model to obtain the optical and SAR paired image; The model processes the input image according to the following steps: Three different encoders are used to decouple optical and SAR features. The encoders include an optical-specific encoder, a SAR-specific encoder, and a mode-sharing encoder. The optical-specific encoder is used to extract high-frequency details from the optical image to obtain optical-specific features. The SAR-specific encoder is used to extract corresponding specific features from the SAR image to obtain SAR-specific features. The mode-sharing encoder is used to extract optical-shared features and SAR-shared features from both optical and SAR images. The decoupled features are recombined, and feature fusion and reconstruction are performed to obtain hybrid features; A decoder is used to process the mixed features and generate optical and SAR paired images. The decoder includes an optical decoder and a SAR decoder. The optical decoder is responsible for converting the received mixed features into an output image that approximates the optical mode, while the SAR decoder is responsible for generating an image that approximates the SAR mode.

2. The optical and SAR paired image generation method as described in claim 1, characterized in that, Step S1 includes: Step S11: Collect optical and SAR images containing different scenes and geographical areas to construct an optical and SAR image dataset; Step S12: Perform quality control on the dataset, removing blurry, noisy, or obviously abnormal images; Step S13: Preprocess the optical and SAR images in the dataset to ensure that the images entering the model have similar data distribution characteristics; Step S14: Divide the entire dataset into a training set, a validation set, and a test set; Step S15: Crop or scale the images in the dataset to a uniform size and convert them to the file format required by the model.

3. The optical and SAR paired image generation method as described in claim 1, characterized in that, The loss functions include a reconstruction loss function, a recurrence loss function, and an adversarial loss function, wherein, Reconstruction loss function satisfy: ； in, , , These represent mode-shared encoders, optically specific encoders, and SAR-specific encoders, respectively. and These represent the optical decoder and the SAR decoder, respectively. and These represent optical images and SAR images, respectively. Cyclic loss function satisfy: ； in, , These represent the visible light image converted to a SAR image and then back to a visible light image, and the SAR image converted to a visible light image and then back to a SAR image, respectively. Adversarial loss function satisfy: ； in, , These represent SAR images converted from visible light images and visible light images converted from SAR images, respectively. , These represent the visible light image discriminator and the SAR image discriminator, respectively.

4. The optical and SAR paired image generation method as described in claim 1, characterized in that, The modality-shared encoder, optical-specific encoder, and SAR-specific encoder all include a multi-scale residual coding unit. This multi-scale residual coding unit is used to extract target contour, edge texture, and scattering intensity features at different downsampling scales. The l-th level coding feature of the multi-scale residual coding unit... satisfy: ； in Represents the l-th level convolution mapping, This represents the residual mapping.

5. The optical and SAR paired image generation method as described in claim 1, characterized in that, The feature fusion and reconstruction specifically refers to: Using modality-shared features as the query vector and target modality-specific features as the key and value vectors, attention weights are calculated to obtain attention fusion features; where the attention weights satisfy: ； Attention fusion features satisfy: ； in: Indicates modal sharing features, Indicates modality-specific features, express and Feature dimensions, This represents the query matrix obtained from the modality-shared feature mapping. This represents the key matrix obtained by mapping specific features of the target mode. This represents the value matrix obtained by mapping specific features of the target mode. Indicates attention weights, Indicates attention fusion features, T This represents the matrix transpose operation. K superscript T This represents the transpose of K. , , They represent the methods used for generation. Q, K, V Learnable linear transformation weight matrix, This represents the normalized exponential function.

6. An optical and SAR paired image generation system, characterized in that, include: Dataset building module: Constructs and preprocesses optical and SAR image datasets; Model building module: Constructs an optical and SAR paired image generation model based on feature decoupling technology; Training module: Constructs a loss function to constrain the image generation process, and trains the model using data from the dataset; Generation module: Input the optical image to be paired and the SAR image into the trained optical and SAR paired image generation model to obtain the optical and SAR paired image; The model processes the input image according to the following steps: Three different encoders are used to decouple optical and SAR features. The encoders include an optical-specific encoder, a SAR-specific encoder, and a mode-sharing encoder. The optical-specific encoder is used to extract high-frequency details from the optical image to obtain optical-specific features. The SAR-specific encoder is used to extract corresponding specific features from the SAR image to obtain SAR-specific features. The mode-sharing encoder is used to extract optical-shared features and SAR-shared features from both optical and SAR images. The decoupled features are recombined, and feature fusion and reconstruction are performed to obtain hybrid features; A decoder is used to process the mixed features and generate optical and SAR paired images. The decoder includes an optical decoder and a SAR decoder. The optical decoder is responsible for converting the received mixed features into an output image that approximates the optical mode, while the SAR decoder is responsible for generating an image that approximates the SAR mode.

7. The optical and SAR paired image generation system as described in claim 6, characterized in that, The dataset building module includes: Data collection unit: Collects optical and SAR images containing different scenes and geographical areas to construct optical and SAR image datasets; Data filtering and cleaning unit: performs quality control on the dataset, removing blurry, noisy, or obviously abnormal images; Data standardization unit: Preprocesses optical and SAR images in the dataset to ensure that the images entering the model have similar data distribution characteristics; Data partitioning unit: Divides the entire dataset into training set, validation set, and test set; Data augmentation unit: Crops or scales images within the dataset to a uniform size and converts them to the file format required by the model.

8. The optical and SAR paired image generation system as described in claim 6, characterized in that, The loss functions include a reconstruction loss function, a recurrence loss function, and an adversarial loss function, wherein, Reconstruction loss function satisfy: ； in, , , These represent mode-shared encoders, optically specific encoders, and SAR-specific encoders, respectively. and These represent the optical decoder and the SAR decoder, respectively. and These represent optical images and SAR images, respectively. Cyclic loss function satisfy: ； in, , These represent the visible light image converted to a SAR image and then back to a visible light image, and the SAR image converted to a visible light image and then back to a SAR image, respectively. Adversarial loss function satisfy: ； in, , These represent SAR images converted from visible light images and visible light images converted from SAR images, respectively. , These represent the visible light image discriminator and the SAR image discriminator, respectively.

Images