Optical-SAR image translation method based on scattering feature enhancement strategy

Abstract

The invention discloses an optical-SAR (Synthetic Aperture Radar) image translation method based on a scattering feature enhancement strategy, the method provides an image translation framework SFEG based on the scattering feature enhancement strategy, the image translation framework SFEG mainly comprises an SCG module and a GFE module, the SCG module firstly performs edge extraction on an input image, generates an edge mask in combination with target annotation information, and translates the edge mask to the GFE module; a scattering field is modeled by combining priori knowledge of SAR target scattering characteristics, finally, the generated scattering field and a corresponding optical image are input into a generator together, and the module can effectively restore the scattering characteristics of a target in the input image. The GFE module comprises a channel attention branch and a space attention branch, the channel attention branch and the space attention branch can carry out information enhancement on input features from the channel dimension and the space dimension respectively, feature recombination is achieved in combination with a nonlinear function, the nonlinear and refined output capacity of SFEG is further enhanced, and the module can improve the actual effect of SAR image generation.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and relates to a Synthetic Aperture Radar (SAR) image translation method, in particular to an optical-SAR image translation method based on a scattering feature enhancement strategy. Background Art

[0002] SAR, an active microwave remote sensing imaging technology, boasts the advantages of 24 / 7 operation and can stably acquire target and background information in complex weather conditions, making it of great value for remote sensing applications. However, currently publicly available SAR image datasets are limited, particularly those containing high-value targets such as aircraft. This severely undermines research and application in both military and civilian fields. In contrast, optical remote sensing image databases, leveraging mature imaging technology and widespread application demand, have accumulated a large amount of sample data containing aircraft targets. These samples exhibit advantages such as sharp edges, intuitive textures, and rich colors. Therefore, SAR image transfer generation methods based on optical data have emerged as an exploratory approach to overcome the bottleneck of SAR data generation. Currently, researchers have proposed a series of optical-SAR image translation methods based on image style transfer techniques, which have alleviated the problem of insufficient SAR image data to some extent. Image translation technology involves mapping a source image into a target image domain, ensuring that the converted image possesses the visual characteristics of the target domain while preserving the structure and semantic content of the source image as much as possible.

[0003] However, current optical-SAR image translation technology still has the following key flaws: (1) SAR images generated by existing methods can only ensure that the global statistical characteristics of the generated SAR images are close to the real SAR images, and cannot effectively restore the scattering characteristics of key targets in the input image. (2) Existing methods have difficulty in suppressing image background interference while enhancing the local feature representation of the target in the generated image. The actual effect of generating SAR images cannot meet the relevant research needs. Summary of the Invention

[0004] To address the problems of existing optical-SAR image translation methods, which lack key target scattering properties and inadequately represent local image features, this paper proposes an optical-SAR image translation method based on a scattering feature enhancement strategy. This method effectively provides a priori guidance on the physical properties of aircraft targets, while preserving the target's true physical scattering properties in the SAR image and ensuring pixel-level matching between the generated image and the input image, enabling high-performance SAR image generation.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] An optical-SAR image translation method based on a scattering feature enhancement strategy comprises the following steps:

[0007] Step 1: Build an image translation framework called Scattering Feature Enhancement GAN (SFEG) based on a scattering feature enhancement strategy. Utilize GAN loss, cycle consistency loss, and identity loss to constrain SFEG's training process. This allows SFEG to accept unpaired optical and SAR image datasets for training and generate corresponding SAR images based on the input optical images.

[0008] Step 2: Construct a Scattering Characteristic Guidance (SCG) module. Considering the similarities in the geometric structure of the same target in optical and SAR images, the module first extracts edges from the input optical image. Then, it combines the target annotation information to generate an edge mask focused on the aircraft target area. Next, it models the scattering field based on prior knowledge of the scattering characteristics of aircraft targets in SAR images. Finally, the generated scattering field and the corresponding optical image are input into the generator of the image translation model for subsequent training.

[0009] Step 3: Construct a Guided Feature Enhancement (GFE) module to enhance the nonlinear and refined output capabilities of SFEG and improve the actual effect of generating SAR images. This module consists of two attention branches. The channel attention branch compresses the channel dimension to varying degrees while maintaining the spatial resolution of the input depth features. It then combines the feature information with a nonlinear function to obtain channel-enhanced features. The spatial attention branch compresses the spatial dimension of the output features and combines the feature information with a nonlinear function to obtain spatial-enhanced features.

[0010] Compared with the prior art, the present invention has the following advantages:

[0011] (1) A SCG module is proposed that can simulate the scattering characteristics of the target in the input optical image. The module first extracts the edge of the input image, generates an edge mask based on the target annotation information, and models the scattering field based on the prior knowledge of the SAR target scattering characteristics. Finally, the generated scattering field and the corresponding optical image are input into the generator. This module can effectively restore the scattering characteristics of the target in the input image.

[0012] (2) A GFE module is proposed that can integrate and enhance the target scattering features of the input image. This module consists of a channel attention branch and a spatial attention branch, which can respectively enhance the input features from the channel dimension and the spatial dimension. It also combines nonlinear functions to achieve feature reorganization, further enhancing the nonlinear and refined output capabilities of SFEG. This module can enhance the target feature representation in the generated image while suppressing background interference, thus improving the actual effect of generating SAR images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of the optical-SAR image translation model structure based on scattering feature enhancement strategy;

[0014] Figure 2 Comparison of SAR image results generated by different unpaired image translation methods. DETAILED DESCRIPTION

[0015] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0016] The present invention provides an optical-SAR image translation method based on a scattering feature enhancement strategy. The method designs an optical-SAR image translation model SFEG based on a scattering feature enhancement strategy, which mainly includes a scattering feature guidance SCG module and a guided feature enhancement GFE module. The model can establish a mapping relationship between the optical domain and the SAR domain, and generate a SAR image that matches the input optical image at the pixel level and retains the target's real physical scattering characteristics. Figure 1 As shown, the specific steps include:

[0017] Step 1: Construct an image translation framework SFEG based on a scattering feature enhancement strategy, and use GAN loss, cycle consistency loss, and identity loss to constrain the SFEG training process. Ultimately, SFEG can receive unpaired optical and SAR image datasets for training and generate corresponding SAR images based on the input optical images. The specific steps are as follows:

[0018] Step 1: Build a basic unpaired image translation framework. The SFEG model consists of two generators and two discriminators, where the generator The discriminator generates the corresponding SAR image according to the input optical image. To evaluate whether the input image is a real SAR image or a generated SAR image, another set of generators and the discriminator It performs the exact opposite function.

[0019] Step 1 and 2: The two sets of generators and discriminators perform an adversarial process during training, and the GAN loss function can be used. It is expressed as follows:

[0020]

[0021] Where, and represent image samples in the optical domain and SAR domain respectively, and Represent the data distribution of optical domain and SAR domain respectively. The construction of GAN loss enables the generator to generate target domain images based on source domain images.

[0022] Step 13: Propose cycle consistency loss To further restrict the generated target domain image, this loss can ensure that the details of the generated image, such as the direction and shape of the target, remain pixel-level corresponding to the input image. The loss is defined as follows:

[0023]

[0024] The role of the cycle consistency loss is to ensure and , thus ensuring that the generated image corresponds to the input image at the pixel level in local details.

[0025] Step 14: Raise Identity Loss , to ensure the continuity and consistency of the generated target domain image. The loss is defined as follows:

[0026]

[0027] Combine the above three types of loss functions to construct an overall loss function , ensuring that the generated image is as close as possible to the real SAR image in terms of overall feature distribution. The overall loss function of the SFEG model is defined as follows:

[0028]

[0029] Where, , and is the weight hyperparameter.

[0030] Step 2: Construct the scattering characteristic guidance module (SCG). Considering the similarity in the geometric structure of the same target in optical and SAR images, the input optical image is first edge extracted. Then, the edge mask focused on the aircraft target area is generated by combining the target annotation information. Next, the scattering field is modeled by combining the prior knowledge of the scattering characteristics of the aircraft target in the SAR image. Finally, the generated scattering field and the corresponding optical image are input into the generator of SFEG to execute the subsequent training process. The specific steps are as follows:

[0031] Step 21: In order to simulate the characteristic that the SAR imaging results of aircraft targets are sensitive to the incident azimuth of the radar wave, the gradient vector field generated by the Sobel operator in the edge extraction process is used. To construct direction-sensitive features , and on this basis, we get the direction-sensitive weight :

[0032]

[0033] in, represents a 1×1 convolutional layer, represents a full connection operation, represents the hyperbolic tangent function, and are learnable parameters.

[0034] Step 2: In order to simulate the effect of the target's scattering intensity attenuating with increasing distance during SAR imaging, a negative exponential spatial attenuation distribution weight is constructed. Specifically, based on the edge mask Construct an initial distance field and extract spatial local features through multi-level dilated convolution:

[0035]

[0036] Where, is a local average pooling operation that maintains spatial resolution, The expansion rate is Then, the obtained multi-level spatial local features are aggregated to generate negative exponential spatial attenuation distribution weights , as follows:

[0037]

[0038] Where, is a learnable parameter.

[0039] Step 2 and 3: Adjust the overall scattering intensity to obtain the final output of the aircraft target scattering field :

[0040]

[0041] Where, represents global average pooling, represents the Hadamard multiplication operator, is a learnable parameter.

[0042] Step 24: Input the target scattered field and the corresponding optical image into the generator of SFEG to carry out the model training process.

[0043] Step 3: Construct the guided feature enhancement module GFE to enhance the nonlinear and refined output capabilities of SFEG and improve the actual effect of generating SAR images. This module contains two attention branches. The channel attention branch compresses the channel dimension to varying degrees while maintaining the spatial resolution of the input depth feature, and reorganizes the feature information with a nonlinear function to obtain channel information enhanced features; the spatial attention branch compresses the spatial dimension of the output feature and reorganizes the feature information with a nonlinear function to obtain spatial information enhanced features. The specific steps are as follows:

[0044] Step 31: First, for a given depth feature map, use the channel attention branch The operation is performed, where R represents a real number set, C represents the number of channels, H represents the image height, and W represents the image width. The feature is transferred to two 1×1 convolutional layers, and the channel dimension is compressed to different degrees while keeping the feature space resolution unchanged to obtain the output feature and , and use the softmax function to The information in is reorganized; then, the reorganized features are combined with Perform matrix multiplication to achieve feature enhancement; then, send the calculation results to the 1×1 convolution layer in sequence , layerNorm (LN) layer and sigmoid function to obtain the channel weight vector , the formula is as follows:

[0045]

[0046] Where, represents the sigmoid function, represents the LayerNorm (LN) layer, represents the softmax function, Represents a matrix multiplication operation.

[0047] Finally, and Perform channel-by-channel multiplication to obtain deep features with enhanced channel information , the equation is as follows:

[0048]

[0049] In the formula Represents a channel-wise multiplication operator.

[0050] Step 32: First, the spatial attention branch is used to input the deep feature map Perform the operation and input it into two 1×1 convolutional layers. Perform global average pooling (GAP) on the output features to achieve spatial dimension compression and provide new output features. and , in getting and Then, use the softmax function to The information in is reorganized; then, the reorganized features are combined with Perform matrix multiplication; then, input the calculation result into the sigmoid function to obtain the spatial weight distribution vector , the formula is as follows:

[0051]

[0052] Where, Represents a feature dimension transformation operation.

[0053] Finally, yes and Perform spatial multiplication to obtain deep features enhanced by spatial information , the equation is expressed as follows:

[0054]

[0055] Where, represents the multiplication operator in spatial directions.

[0056] Through the processing of step 31 and step 32, a SAR output image corresponding to the pixel level of the input optical image can be obtained.

[0057] Step 4: Use the CORS-ADD dataset, an optical aircraft target dataset used in remote sensing scenarios, and the SAR-AIRcraft-1.0 dataset, an aircraft dataset based on high-resolution SAR remote sensing images, to verify the performance of the SFEG proposed in this paper. The specific steps are as follows:

[0058] Validation was performed using the CORS-ADD and SAR-AIRcraft-1.0 datasets. The CORS-ADD dataset has image resolutions ranging from 0.31 to 1.02 meters, with a fixed image size of 640×640 pixels. A total of 32,285 aircraft instances were annotated. This dataset was split into training and test sets in a 7:3 ratio, with the training set containing 3,764 images and the test set containing 1,722 images. SAR-AIRcraft-1.0 is an aircraft dataset based on high-resolution SAR remote sensing imagery. The polarization mode is single polarization, the imaging mode is beam focusing, and the spatial resolution is 1 meter. The dataset contains 4,368 images and 16,463 aircraft instances. The dataset image sizes include 800×800, 1000×1000, 1200×1200, and 1500×1500 pixels.

[0059] The model training process was implemented on an Ubuntu 23.04 server based on the PyTorch 1.9.1 framework. The CPU was an Intel (R) Xeon (R) Platinum 8375C @ 2.90GHz and the GPU was an Nvidia RTX 4090 24G. All images of the source and target domains were input at their original sizes, the batch size was set to 1, and the weights were updated using the Adam optimizer. The training process lasted for 50 epochs. The first 25 epochs were trained with an initial learning rate of 0.0002, and the learning rate was linearly decayed to 0 for the next 25 epochs. The experimental results are shown in Figure 2. Figure 2 As shown in the figure, it can be found that the optical-SAR image translation method based on the scattering feature enhancement strategy proposed in the present invention has excellent performance, can provide prior guidance on the physical properties of the aircraft target, and can achieve pixel-level matching between the generated image and the input image while maintaining the real physical scattering properties of the target in the SAR image.

Claims

1. An optical-SAR image translation method based on scattering feature enhancement strategy, characterized by The method comprises the following steps: Step 1: Construct an image translation framework SFEG based on a scattering feature enhancement strategy. Utilize GAN loss, cycle consistency loss, and identity loss to constrain the SFEG training process. This allows SFEG to receive unpaired optical and SAR image datasets for training and generate corresponding SAR images based on the input optical images. Step 2: Construct a scattering characteristic-guided SCG module. Considering the similarities in the geometric structures of the same target in optical and SAR images, the input optical image is first edge-extracted. Then, an edge mask focused on the aircraft target region is generated by combining the target annotation information. Next, the scattering field is modeled by combining prior knowledge of the scattering characteristics of aircraft targets in SAR images. Finally, the generated scattering field and the corresponding optical image are input into the generator of the image translation model for subsequent training. Step 3: Construct a guided feature enhancement (GFE) module to enhance the nonlinear and refined output capabilities of SFEG and improve the actual effect of generating SAR images. This module contains two attention branches. The channel attention branch compresses the channel dimension to varying degrees while keeping the spatial resolution of the input depth feature unchanged, and reorganizes the feature information with a nonlinear function to obtain channel information enhanced features; the spatial attention branch compresses the spatial dimension of the output features, and reorganizes the feature information with a nonlinear function to obtain spatial information enhanced features.

2. The optical-SAR image translation method based on the scattering feature enhancement strategy according to claim 1 is characterized in that The specific steps of step one are as follows: Step 1: Build a basic unpaired image translation framework: The SFEG model consists of two generators and two discriminators, where the generator The discriminator generates the corresponding SAR image according to the input optical image. To evaluate whether the input image is a real SAR image or a generated SAR image, another set of generators and the discriminator It performs the exact opposite function; Step 1 and 2: The two sets of generators and discriminators perform an adversarial learning process during training, using the GAN loss function It is expressed as follows: Where, and represent image samples in the optical domain and SAR domain respectively, and Representing the data distribution of the optical domain and SAR domain respectively, the construction of GAN loss enables the generator to generate target domain images based on source domain images; Step 13: Propose cycle consistency loss Further restricting the generated target domain images, the loss is defined as follows: The role of the cycle consistency loss is to ensure and , thus ensuring that the generated image corresponds to the input image at the pixel level in local details; Step 14: Raise Identity Loss , to ensure the continuity and consistency of the generated target domain image, the loss is defined as follows: Combine the above three types of loss functions to construct an overall loss function , to ensure that the generated image is as close as possible to the real SAR image in terms of overall feature distribution. The overall loss function is defined as follows: Where, , and is the weight hyperparameter.

3. The optical-SAR image translation method based on scattering feature enhancement strategy according to claim 1 is characterized in that The specific steps of step 2 are as follows: Step 21: Use the gradient vector field generated by the Sobel operator during edge extraction Constructing direction-sensitive features , and on this basis, we get the direction-sensitive weight : in, represents a 1×1 convolutional layer, represents a full connection operation, represents the hyperbolic tangent function, and is a learnable parameter; Step 22: Construct negative exponential spatial decay distribution weights : Step 2 and 3: Adjust the overall scattering intensity to obtain the final output of the aircraft target scattering field : Where, represents global average pooling, represents the Hadamard multiplication operator, is a learnable parameter; Step 24: Input the target scattered field and the corresponding optical image into the generator of SFEG to carry out the model training process.

4. The optical-SAR image translation method based on scattering feature enhancement strategy according to claim 3 is characterized in that The specific steps of step 22 are as follows: First, based on the edge mask Construct an initial distance field and extract spatial local features through multi-level dilated convolution: Where, is a local average pooling operation that maintains spatial resolution, The expansion rate is Convolution operation; Subsequently, the obtained multi-level spatial local features are aggregated to generate negative exponential spatial attenuation distribution weights , as follows: Where, is a learnable parameter.

5. The optical-SAR image translation method based on scattering feature enhancement strategy according to claim 1 is characterized in that The specific steps of step three are as follows: Step 31: First, for a given depth feature, use the channel attention branch The operation is performed, where R represents a real number set, C represents the number of channels, H represents the image height, and W represents the image width. The feature is transferred to two 1×1 convolutional layers, and the channel dimension is compressed to different degrees while keeping the feature space resolution unchanged to obtain the output feature. and , and use the softmax function to The information in is reorganized; then, the reorganized features are combined with Perform matrix multiplication to achieve feature enhancement; then, send the calculation results to the 1×1 convolution layer in sequence , LN layer and sigmoid function, get the channel weight vector , the formula is as follows: Where, represents the sigmoid function, Indicates the LN layer, represents the softmax function, Represents matrix multiplication operation; Finally, and Perform channel-by-channel multiplication to obtain deep features with enhanced channel information , the equation is as follows: In the formula represents the channel-level multiplication operator; Step 32: First, use the spatial attention branch to analyze the deep features of the input Perform the operation and input it into two 1×1 convolutional layers. Perform global average pooling on the output features to achieve spatial dimension compression and provide new output features. and , in getting and Then, use the softmax function to The information in is reorganized; then, the reorganized features are combined with Perform matrix multiplication; then, input the calculation result into the sigmoid function to obtain the spatial weight distribution vector , the formula is as follows: Where, Represents the feature dimension transformation operation; Finally, yes and Perform spatial multiplication to obtain deep features enhanced by spatial information , the equation is expressed as follows: Where, represents the multiplication operator according to the spatial direction; Through the processing of step 31 and step 32, a SAR output image corresponding to the pixel level of the input optical image is obtained.

Images