Light-SAR image conversion method based on coordinate attention and histogram loss

Abstract

The invention provides a light-SAR image conversion method based on coordinate attention and histogram loss, and relates to the technical field of SAR image data enhancement, and the method comprises the steps: obtaining a to-be-converted optical image; inputting the optical image to be converted into a pre-trained SAR image generator, and converting to obtain a target SAR image; a coordinate attention module is arranged in the pre-trained SAR image generator, and the pre-trained SAR image generator is obtained based on a pre-trained bidirectional conversion framework; the coordinate attention module is used for performing global pooling from the horizontal direction and the vertical direction respectively, and constructing a global feature vector corresponding to the optical image to be converted based on a global pooling result; histogram loss is set in an overall loss function of the pre-training bidirectional conversion framework; histogram losses are constructed based on kernel density estimates and differences between image histograms are determined using square earth motion distances. In combination with the method, the realistic degree of the finally generated target SAR image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of SAR image data enhancement, and in particular to a light-SAR image conversion method based on coordinate attention and histogram loss. Background Art

[0002] With the continuous development of Earth observation technology, synthetic aperture radar (SAR), with its unique all-day, all-weather imaging capabilities, has demonstrated irreplaceable value in key areas such as disaster monitoring, maritime surveillance, and environmental management. However, the widespread application of deep learning technology in SAR image interpretation tasks faces significant challenges: the urgent need for large-scale, high-quality training data for deep neural networks is in sharp conflict with the high cost of acquiring SAR imagery and the limited size of publicly available datasets. This contradiction directly restricts the performance improvement of deep learning-based SAR image target detection, classification, and change detection technologies, becoming a core bottleneck limiting breakthrough progress in this field. How to effectively expand the scale and diversity of SAR image datasets while ensuring the authenticity and availability of the generated data has become a key scientific issue in promoting the intelligent development of SAR remote sensing technology.

[0003] To address these challenges, existing approaches have proposed data-driven SAR image enhancement methods, with the Cycle Generative Adversarial Network (CycleGAN) emerging as a mainstream approach. This method utilizes unpaired optical and SAR image datasets by constructing a bidirectional generative network (G_O2S and G_S2O) to achieve cross-modal image translation. Specifically, G_O2S converts optical images into synthetic images with the style of SAR images, while G_S2O performs the reverse translation. To ensure semantic consistency during the translation process, CycleGAN incorporates an adversarial training mechanism. Domain discriminators (D_OPT and D_SAR) constrain the modal characteristics of the generated images. A cycle consistency loss (L_cycle) is also incorporated to enforce that the synthetic images reconstruct the original input after reverse translation, thereby achieving a balance between style transfer and semantic preservation without the constraints of paired data. By leveraging a publicly available optical image database and a small number of SAR image samples, this method avoids the reliance of traditional electromagnetic simulation methods on prior target knowledge and significantly reduces data generation costs.

[0004] Although CycleGAN has made progress in the field of unpaired image conversion, its core defects still restrict the quality and practicality of generated SAR images. Specifically, there are essential differences in the imaging mechanisms of optical images and SAR images: the former relies on the spectral characteristics of ground object reflection, while the latter reflects the target scattering intensity and phase information, resulting in significant nonlinear correlation between the two modalities in semantic features (such as ground object structure) and style features (such as scattering distribution). The domain discriminator relied on by existing methods can only impose coarse-grained constraints on image-level modal differences and cannot accurately model the complex mapping relationship between optical and SAR images. In the absence of pixel-level correspondence supervision, the generator is prone to mode collapse, resulting in the following problems in the synthesized images:

[0005] 1. Semantic distortion: The ability to extract key features such as edges and textures of objects is insufficient, resulting in blurred target structures or geometric distortion in the generated images.

[0006] 2. Style shift: The unique scattering characteristics of SAR images (such as coherent speckle noise and multi-polarization characteristics) are not fully learned, and the amplitude distribution of the generated image has significant statistical differences from the real SAR data.

[0007] 3. Quality fluctuation: The generated results are unstable in different scenarios (such as cities, water areas, and vegetation), making it difficult to meet the stringent requirements for data consistency in tasks such as disaster monitoring.

[0008] The above defects cause the SAR images generated by existing methods to show obvious deficiencies in quantitative evaluation indicators (such as structural similarity index SSIM, peak signal-to-noise ratio PSNR) and expert visual interpretation, which seriously limits their direct application in high-precision remote sensing tasks. Summary of the Invention

[0009] In order to solve the above problems existing in the prior art, the present invention provides a light-SAR image conversion method based on coordinate attention and histogram loss.

[0010] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0011] The present invention provides a method for optical-SAR image conversion based on coordinate attention and histogram loss, comprising:

[0012] acquiring an optical image to be converted;

[0013] The optical image to be converted is input into a pre-trained SAR image generator to obtain a target SAR image; a coordinate attention module is provided in the pre-trained SAR image generator, and the pre-trained SAR image generator is obtained based on a pre-trained bidirectional conversion framework; the coordinate attention module is used to perform global pooling in the horizontal and vertical directions respectively, and construct a global feature vector corresponding to the optical image to be converted based on the result of global pooling; the pre-trained bidirectional conversion framework is provided with a conversion and discrimination architecture for converting optical images to SAR images and SAR images to optical images, and a histogram loss is provided in the overall loss function of the pre-trained bidirectional conversion framework; the histogram loss is constructed based on kernel density estimation and uses squared ground motion distance to determine the difference between image histograms.

[0014] Optionally, the pre-trained SAR image generator is provided with a feature encoder module, a semantic enhancement module and a feature decoder module connected in sequence;

[0015] The feature encoder module is provided with a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence;

[0016] The semantic enhancement module is provided with multiple semantic enhancement submodules, which are connected in sequence. Each semantic enhancement submodule is provided with a fourth convolutional layer, a coordinate attention module, and a fifth convolutional layer connected in sequence. The input end of each semantic enhancement submodule is connected to the output end of the corresponding semantic enhancement submodule.

[0017] The feature decoder module is provided with a first transposed convolution layer, a second transposed convolution layer and a third transposed convolution layer connected in sequence; the output end of the first convolution layer is also connected to the input end of the third transposed convolution layer; the output end of the second convolution layer is also connected to the input end of the second transposed convolution layer.

[0018] Optionally, the pre-trained bidirectional conversion framework is provided with: a pre-trained SAR image generator, a pre-trained optical image generator, a pre-trained SAR image discriminator, and a pre-trained optical image discriminator;

[0019] The pre-trained SAR image generator and the pre-trained SAR image discriminator are obtained based on adversarial training between the two; the pre-trained optical image generator and the pre-trained optical image discriminator are obtained based on adversarial training between the two; the pre-trained SAR image generator and the pre-trained optical image generator have the same structure; the pre-trained SAR image discriminator and the pre-trained optical image discriminator have the same structure.

[0020] Optionally, the pre-trained SAR image discriminator and the pre-trained optical image discriminator both adopt a PatchGAN structure, in which a plurality of preset convolutional layers connected in sequence are provided, and a convolution kernel size of the preset convolutional layer is 4×4.

[0021] Optionally, the processing of the semantic enhancement module includes:

[0022] Using the fourth convolutional layer to perform feature dimension reduction processing on the feature encoding result to obtain a first processing result; the feature encoding result is the output result of the feature encoder module;

[0023] Performing global average pooling operations on the first processing result along the horizontal direction and the vertical direction respectively, to obtain a horizontal feature vector and a vertical feature vector respectively;

[0024] Perform feature concatenation on the horizontal feature vector and the vertical feature vector to obtain the global feature vector;

[0025] Performing feature dimension reduction processing on the global feature vector to obtain a second processing result;

[0026] Perform feature separation processing on the second processing result along the horizontal direction and the vertical direction respectively, and obtain horizontal separation features and vertical separation features respectively;

[0027] Perform channel recovery processing on the horizontal separation features and the vertical separation features respectively, and obtain the horizontal direction attention map and the vertical direction attention map respectively;

[0028] Perform weighted fusion processing on the horizontal attention map, the vertical attention map and the first processing result to obtain a weighted feature;

[0029] The fifth convolutional layer is used to perform feature dimension recovery processing on the weighted features to obtain a third processing result;

[0030] The third processing result and the feature encoding result are superimposed to obtain the first semantic enhancement sub-module feature; the first semantic enhancement sub-module feature is the output feature corresponding to the first semantic enhancement sub-module in the semantic enhancement module.

[0031] Optionally, the overall loss function corresponding to the pre-trained bidirectional transformation framework is the sum of adversarial loss, cycle consistency loss, and histogram loss.

[0032] Optionally, the adversarial loss includes: a generator adversarial loss and a discriminator adversarial loss;

[0033] The generator adversarial loss is expressed as:

[0034]

[0035] Among them, L G represents the value of the generator's adversarial loss, Represents the initial SAR image generator based on the real optical image x O The generated fake SAR image, express The discrimination results in the initial SAR image discriminator, Represents the initial optical image generator based on the real SAR image x S The generated false optical image, express The discrimination results in the initial optical image discriminator;

[0036] The discriminator adversarial loss is expressed as:

[0037]

[0038]

[0039] Among them, L D represents the value of the discriminator's adversarial loss, x O represents the real optical image, x S represents the real SAR image, D OPT (x O ) represents x O The discrimination result in the initial optical image discriminator, D SAR (x S ) represents x S The discrimination results in the initial SAR image discriminator;

[0040] The cycle consistency loss is expressed as:

[0041]

[0042] Among them, L Cycl∑ represents the value of cycle consistency loss, λ represents the first weight parameter, Indicates that the initial optical image generator is based on the false SAR image The generated image, Indicates that the initial SAR image generator is based on the false optical image For the generated image, ‖·‖1 represents the L1 norm;

[0043] The histogram loss is expressed as:

[0044]

[0045] Among them, L Histo represents the value of histogram loss, β represents the second weight parameter, express The corresponding histogram features, Represents x O The corresponding histogram features, express The corresponding histogram features, Represents x S The corresponding histogram features, express and The corresponding squared ground motion distance, express and The corresponding squared earthquake distance.

[0046] Alternatively, the squared ground motion distance is expressed as:

[0047]

[0048] Among them, SEMD(H1,H2) represents the squared ground motion distance between the histogram feature H1 of the first input image and the histogram feature H2 of the second input image, M represents the total number of intervals divided for image pixels, m represents the mth pixel subinterval, and CDF m (.) represents the cumulative distribution function;

[0049]

[0050] in, Represents the pixel distribution intensity value of the first input image in the mth pixel subinterval, Indicates the value of the left boundary of the m-th pixel subinterval, represents the value of the right boundary of the mth pixel subinterval, h represents the smoothness parameter of the kernel function, n represents the total number of image pixels corresponding to the first input image, i represents the i-th image pixel corresponding to the first input image, and x i represents the pixel value of the i-th image pixel corresponding to the first input image, and σ(.) represents the Sigmoid function.

[0051] Optionally, the training process of the pre-trained bidirectional conversion framework includes:

[0052] Acquire a training sample set; the training sample set includes a plurality of optical sample images and SAR sample images;

[0053] Input the optical sample image into the initial bidirectional conversion framework for training;

[0054] The initial bidirectional conversion frame corresponding to the preset stopping condition is used as the pre-trained bidirectional conversion frame;

[0055] The preset stopping conditions include: the number of training times meets the preset iteration threshold or the value of the overall loss function is continuously less than the loss function threshold;

[0056] Among them, the structure of the initial bidirectional conversion framework is the same as that of the pre-trained bidirectional conversion framework.

[0057] The present invention provides a method for optical-SAR image conversion based on coordinate attention and histogram loss. The method comprises the following steps: obtaining an optical image to be converted; inputting the optical image to be converted into a pre-trained SAR image generator to convert the image into a target SAR image; the pre-trained SAR image generator is provided with a coordinate attention module, and the pre-trained SAR image generator is obtained based on a pre-trained bidirectional conversion framework; the coordinate attention module is used to perform global pooling in the horizontal and vertical directions, and construct a global feature vector corresponding to the optical image to be converted based on the results of the global pooling; the pre-trained bidirectional conversion framework is provided with a conversion and discrimination architecture for converting optical images into SAR images and SAR images into optical images, and the overall loss function of the pre-trained bidirectional conversion framework is provided with a histogram loss; the histogram loss is constructed based on kernel density estimation and uses squared ground motion distance to determine the difference between image histograms. In this paper, a coordinate attention module is incorporated into a pre-trained SAR image generator. This module utilizes global pooling operations in both horizontal and vertical directions to obtain feature vectors along the corresponding coordinate axes. This module retains the semantic positional information of a specific coordinate axis while also obtaining feature vectors along the other axis. The feature vectors from both directions are then fused to generate a global feature vector, improving the pre-trained SAR image generator's ability to capture the semantic content of the original optical image. Furthermore, by leveraging the weak correlation between histogram features and the spatial semantic distribution of an image, a histogram loss is constructed for use in a bidirectional transformation framework to measure the difference between unpaired images. To address the non-differentiable nature of the histogram loss calculation, kernel density estimation is used to construct the histogram loss, making it differentiable. The squared ground motion distance is then used to describe the differences in image histogram features. By using the histogram loss as an auxiliary learning objective for unpaired optical-SAR image conversion, along with the coordinate attention module and bidirectional transformation framework, the generator's learning of the overall image distribution is enhanced, reducing semantic distortion, style shift, and image quality fluctuation, thereby improving the fidelity of the resulting target SAR images.

[0058] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of a method for optical-SAR image conversion based on coordinate attention and histogram loss provided by an embodiment of the present invention;

[0060] Figure 2The schematic diagram of the structure of the pre-trained SAR image generator is shown as an example;

[0061] Figure 3 The structural diagram of the coordinate attention module is shown as an example;

[0062] Figure 4 The structural diagram of the bidirectional conversion framework is shown as an example;

[0063] Figure 5 Some SAR image results obtained based on comparative experiments are shown as examples;

[0064] Figure 6 The figure shows an example of a histogram comparison of the generated results of the baseline model before and after using the histogram loss. DETAILED DESCRIPTION

[0065] To address the inaccurate semantic extraction and incomplete SAR image style learning caused by optical-SAR image differences and unpaired images, this paper proposes a SAR image generation method based on a coordinate attention module and a differentiable histogram loss, based on the CycleGAN training framework. The coordinate attention module uses global pooling operations in both horizontal and vertical directions to obtain global feature vectors for the corresponding coordinate axes, preserving the semantic position information of a specific coordinate axis while acquiring global information in the other direction. By concatenating and fusing the global feature vectors from the two coordinate directions, an attention map is constructed and image semantic features are enhanced, improving the generator's ability to accurately capture the semantic content of the original optical image. Furthermore, by leveraging the weak correlation between histogram features and the spatial semantic distribution of images, a histogram loss function is constructed for unpaired image difference measurement. To address the non-differentiable nature of the histogram calculation process, kernel density estimation is used to construct differentiable histogram features, and ground motion distance is used to describe the differences in image histogram features. By using the histogram loss function as an auxiliary learning objective for unpaired optical-SAR image conversion, the generator is encouraged to accurately learn the overall distribution of SAR images, improving the fidelity of the generated images.

[0066] In summary, the method proposed in this paper accurately captures the semantics of the original image by utilizing the coordinate attention module and constructs a differentiable histogram as an auxiliary supervised learning objective, providing a more advanced and efficient solution for optical-SAR image conversion in unpaired situations.

[0067] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0068] In order to improve the realism of the target SAR image finally generated, an embodiment of the present invention provides a light-SAR image conversion method based on coordinate attention and histogram loss. Figure 1 A flow chart of a method for converting light-SAR images based on coordinate attention and histogram loss is provided in an embodiment of the present invention, as shown in FIG. Figure 1 Shown, including:

[0069] S101: Acquire an optical image to be converted.

[0070] S102: Input the optical image to be converted into a pre-trained SAR image generator to obtain a target SAR image.

[0071] It should be noted that a coordinate attention module is provided in the pre-trained SAR image generator, and the pre-trained SAR image generator is obtained based on a pre-trained bidirectional conversion framework; the coordinate attention module is used to perform global pooling in the horizontal and vertical directions respectively, and construct a global feature vector corresponding to the optical image to be converted based on the result of global pooling; the pre-trained bidirectional conversion framework is provided with a conversion and discrimination architecture for optical image to SAR image and SAR image to optical image, and the overall loss function of the pre-trained bidirectional conversion framework is provided with a histogram loss; the histogram loss is constructed based on kernel density estimation and uses the squared ground motion distance to determine the difference between image histograms.

[0072] An embodiment of the present invention provides a method for optical-SAR image conversion based on coordinate attention and histogram loss. By setting a coordinate attention module in a pre-trained SAR image generator, the coordinate attention module uses global pooling operations in both horizontal and vertical directions to obtain feature vectors in the corresponding coordinate axis directions. This module retains the semantic position information of a specific coordinate axis while obtaining feature vectors in the other direction. The feature vectors in the two directions are then fused to generate a global feature vector, thereby improving the ability of the pre-trained SAR image generator to capture the semantic content of the original optical image. Furthermore, by leveraging the weak correlation between histogram features and the spatial semantic distribution of images, a histogram loss that can be used to measure the difference between unpaired images is constructed for the bidirectional conversion framework. To address the problem that the histogram loss calculation process is non-differentiable, kernel density estimation is used to construct the histogram loss, making it differentiable. The squared ground motion distance is used to describe the difference in image histogram features. By using histogram loss as an auxiliary learning objective for unpaired light-SAR image conversion, as well as the setting of coordinate attention module and bidirectional conversion framework, the overall learning of the generator on the overall distribution of the image is promoted, the occurrence of semantic distortion, style shift and image quality fluctuation problems is reduced, and the realism of the final generated target SAR image is improved.

[0073] Figure 2 The schematic diagram of the structure of the pre-trained SAR image generator is shown as an example. Figure 2As shown, the pre-trained SAR image generator is provided with a feature encoder module, a semantic enhancement module and a feature decoder module connected in sequence;

[0074] The feature encoder module is provided with a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence;

[0075] The semantic enhancement module is provided with multiple semantic enhancement submodules, which are connected in sequence. Each semantic enhancement submodule is provided with a fourth convolutional layer, a coordinate attention module, and a fifth convolutional layer connected in sequence. The input end of each semantic enhancement submodule is connected to the output end of the corresponding semantic enhancement submodule.

[0076] The feature decoder module is provided with a first transposed convolution layer, a second transposed convolution layer and a third transposed convolution layer connected in sequence; the output end of the first convolution layer is also connected to the input end of the third transposed convolution layer; the output end of the second convolution layer is also connected to the input end of the second transposed convolution layer.

[0077] In addition, the pre-trained SAR image generator uses instance normalization (IN) and LeakyReLU function with a slope of 0.2 as the activation function.

[0078] The processing of the pre-trained SAR image generator is performed as follows:

[0079] 1. The feature encoder is used to reduce the spatial dimension of the feature map and extract the semantic features of the image. Its input is the original image x∈R 3×H×W , R 3×H×W Represents the dimensions of image x in terms of channels, pixel height, and pixel width, where 3 represents the number of image channels, H and W are the pixel height and width of the original image respectively. The encoder network structure contains three convolutional layers (the first convolutional layer, the second convolutional layer, and the third convolutional layer) with a convolution kernel size of 4×4 and a stride of 2. The input of each convolutional layer is the output of the previous convolutional layer. The first convolutional layer has 3 input channels and 64 output channels, while the second and third convolutional layers have 128 and 256 output channels respectively. The second convolutional layer uses normalization and activation functions; the third convolutional layer uses activation functions. After three layers of downsampling, a high-dimensional semantic feature f0∈R with a resolution of 1 / 8 of the original image is obtained. 256×H / 8×W / 8 .

[0080] 2. The semantic enhancement module is used to extract and enhance the image semantics of the high-dimensional semantic features extracted by the feature encoder. Its input is the output f0∈R of the feature encoder. 256×H / 8×W / 8 , the output is the enhanced semantic feature f0 of the same dimension size ′ ∈R 256×H / 8×W / 8. The network structure of the semantic enhancement module is composed of N identical basic modules (semantic enhancement submodules) stacked together. Each basic module includes two convolutional layers (the fourth convolutional layer and the fifth convolutional layer), a coordinate attention module and a residual connection. The convolution kernel size of the fourth and fifth convolutional layers is 3×3, the step size is 1, and normalization and activation functions are used. The fourth convolutional layer is used for channel dimensionality reduction, and the number of output channels is half of the input; the fifth convolutional layer is used for channel recovery, and the number of output channels is twice the number of input channels. The coordinate attention module is located between the fourth and fifth convolutional layers, and is used to mine the spatial position information of the input image content and enhance the global semantic expression ability of the features. Figure 3 The structural diagram of the coordinate attention module is shown as an example. For an input feature map with a channel number of c0 and a feature height and width of h and w respectively The calculation process of the semantic enhancement module is as follows:

[0081] 2.1. Use the fourth convolutional layer to process the output features of the feature encoder module and reduce the feature channel dimension to half of the output feature dimension c0 of the feature encoder module, c=c0 / 2.

[0082] 2.2、For the feature f∈R after dimensionality reduction c×h×w , use the global average pooling operation in the horizontal and vertical directions along the coordinate axis to obtain the vertical feature vector f h ∈R c×h and the horizontal eigenvector f w ∈R c ×w c, h, and w represent the number of feature map channels, feature map height, and feature map width, respectively.

[0083] 2.3. Concatenate the vertical feature vector with the horizontal feature vector to change the feature dimension to R c ×(h+w) The concatenated features are then dimensionality reduced using a 1×1 convolutional layer, reducing their channel dimension from c to c / r, where r is the channel compression ratio. This convolutional layer can fuse the feature vector information in both horizontal and vertical directions, resulting in a dimension of R c / r×(h+w) The fusion feature is then separated along the second dimension based on the length h and w to obtain the vertical feature f h ′ ∈R c / r×h and the horizontal feature f w ′ ∈R c / r×w .

[0084] 2.4. Use 1×1 convolutional layers to perform channel recovery on vertical features and horizontal features respectively to obtain the vertical attention map A H ∈R c×h And the horizontal attention map A W ∈R c×w Multiply the vertical attention map and the horizontal attention map with the original feature map (i.e., the first processing result obtained by the fourth convolutional layer performing feature dimension reduction on the feature encoding result), and use the attention score to weight the strength of the feature points at different positions to obtain weighted features, thereby enhancing the semantic expression ability of the feature map. For example: for the feature value with channel q and coordinates (i, j), i and j represent the horizontal and vertical coordinates of the attention map respectively. The specific calculation process is as follows:

[0085] f ′ (q,i,j)=f(q,i,j)×σ(A H (q,i))×σ(A W (q,j))

[0086] f ′ (q,i,j) represents the weighted feature, f(q,i,j) represents the first processing result, A H (q,i) represents the eigenvalue of the vertical attention map at the channel q and the horizontal coordinate i, A W (q, j) represents the eigenvalue of the horizontal attention map at channel q and vertical coordinate j. σ(·) is the Sigmoid function used to generate the normalized attention map.

[0087] 2.5. Use the fifth convolutional layer to weight the feature f ′ ∈R c×h×w Perform channel recovery and restore its feature channel dimension to c0 to obtain the third processing result

[0088] 2.6. To avoid the gradient vanishing problem, a residual connection structure is used to add the feature encoding result to the third processing result to obtain the output feature of the first semantic enhancement submodule.

[0089] 2.7. To enhance the generator network's ability to extract and express semantic meaning from the input optical image, stack N identical basic modules. The specific computational process involves repeating steps 2.1 through 2.6 N times, using the output features of each semantic enhancement submodule as the input to the next. Through multiple coordinate attention calculations and feature weighting, the image's semantic features are effectively enhanced.

[0090] 3. The feature decoder is used to decode the enhanced image semantic features and generate a fake image in the target domain style. The feature decoder consists of three transposed convolutional layers with a convolution kernel size of 4×4 and a stride of 2. The input of the first transposed convolutional layer is the enhanced semantic features (the output of the semantic enhancement module), the number of input channels is 256, the number of output channels is 128, and the first transposed convolutional layer uses an activation function; the input of the second transposed convolutional layer is the output of the first transposed convolutional layer and the output of the second convolutional layer; the input of the third transposed convolutional layer is the output of the second transposed convolutional layer and the output of the first convolutional layer, and the number of output channels of the third transposed convolutional layer is 3.

[0091] Based on the bidirectional conversion training framework, an optical to SAR image generator (also referred to as SAR image generator) with the same structure is constructed. O2S and SAR to optical image generator (also referred to as optical image generator) G S2O . Figure 4 The schematic diagram of the bidirectional conversion framework is shown as an example. Figure 4 As shown, the bidirectional conversion framework is equipped with a SAR image generator G O2S , optical image generator G S2O , SAR image discriminator D SAR and optical image discriminator D OPT . Real optical image input G O2S Get the false SAR image, and input the real SAR image into the optical image generator G S2O Get a fake optical image, calculate the cycle consistency loss between the real optical image and the fake optical image, and calculate the cycle consistency loss between the fake SAR image and the real SAR image. The real SAR image and the fake SAR image are input to the SAR image discriminator D SAR Calculate the adversarial loss, the fake optical image and the real optical image are input to the optical image discriminator D OPT Calculate the adversarial loss.

[0092] Correspondingly, the pre-trained bidirectional conversion framework is provided with: a pre-trained SAR image generator, a pre-trained optical image generator, a pre-trained SAR image discriminator, and a pre-trained optical image discriminator;

[0093] The pre-trained SAR image generator and the pre-trained SAR image discriminator are obtained based on adversarial training between the two; the pre-trained optical image generator and the pre-trained optical image discriminator are obtained based on adversarial training between the two; the pre-trained SAR image generator and the pre-trained optical image generator have the same structure; the pre-trained SAR image discriminator and the pre-trained optical image discriminator have the same structure.

[0094] Optionally, the pre-trained SAR image discriminator and the pre-trained optical image discriminator both adopt a PatchGAN structure, in which a plurality of preset convolutional layers connected in sequence are provided, and a convolution kernel size of the preset convolutional layer is 4×4.

[0095] Optionally, the processing of the semantic enhancement module includes:

[0096] Using the fourth convolutional layer to perform feature dimension reduction processing on the feature encoding result to obtain a first processing result; the feature encoding result is the output result of the feature encoder module;

[0097] Performing global average pooling operations on the first processing result along the horizontal direction and the vertical direction respectively, to obtain a horizontal feature vector and a vertical feature vector respectively;

[0098] Perform feature concatenation on the horizontal feature vector and the vertical feature vector to obtain the global feature vector;

[0099] Performing feature dimension reduction processing on the global feature vector to obtain a second processing result;

[0100] Perform feature separation processing on the second processing result along the horizontal direction and the vertical direction respectively, and obtain horizontal separation features and vertical separation features respectively;

[0101] Perform channel recovery processing on the horizontal separation features and the vertical separation features respectively, and obtain the horizontal direction attention map and the vertical direction attention map respectively;

[0102] Perform weighted fusion processing on the horizontal attention map, the vertical attention map and the first processing result to obtain a weighted feature;

[0103] The fifth convolutional layer is used to perform feature dimension recovery processing on the weighted features to obtain a third processing result;

[0104] The third processing result and the feature encoding result are superimposed to obtain the first semantic enhancement sub-module feature; the first semantic enhancement sub-module feature is the output feature corresponding to the first semantic enhancement sub-module in the semantic enhancement module.

[0105] Optionally, the overall loss function corresponding to the pre-trained bidirectional transformation framework is the sum of adversarial loss, cycle consistency loss, and histogram loss.

[0106] Optionally, the adversarial loss includes: a generator adversarial loss and a discriminator adversarial loss;

[0107] The generator adversarial loss is expressed as:

[0108]

[0109] Among them, LG represents the value of the generator's adversarial loss, Represents the initial SAR image generator based on the real optical image x O The generated fake SAR image, express The discrimination results in the initial SAR image discriminator, Represents the initial optical image generator based on the real SAR image x S The generated false optical image, express The discrimination results in the initial optical image discriminator;

[0110] The discriminator adversarial loss is expressed as:

[0111]

[0112] Among them, L D represents the value of the discriminator's adversarial loss, x O represents the real optical image, x S represents the real SAR image, D OPT (x O ) represents x O The discrimination result in the initial optical image discriminator, D SAR (x S ) represents x S The discrimination results in the initial SAR image discriminator;

[0113] The cycle consistency loss is expressed as:

[0114]

[0115] Among them, L Cycle represents the value of cycle consistency loss, λ represents the first weight parameter, Indicates that the initial optical image generator is based on the false SAR image The generated image, Indicates that the initial SAR image generator is based on the false optical image For the generated image, ‖·‖1 represents the L1 norm;

[0116] The histogram loss is expressed as:

[0117]

[0118] Among them, L Histo represents the value of histogram loss, β represents the second weight parameter, express The corresponding histogram features, Represents x OThe corresponding histogram features, express The corresponding histogram features, Represents x S The corresponding histogram features, express and The corresponding squared ground motion distance, express and The corresponding squared earthquake distance.

[0119] Alternatively, the squared ground motion distance is expressed as:

[0120]

[0121] Among them, SEMD(H1,H2) represents the squared ground motion distance between the histogram feature H1 of the first input image and the histogram feature H2 of the second input image, M represents the total number of intervals divided for image pixels, m represents the mth pixel subinterval, and CDF m (.) represents the cumulative distribution function;

[0122]

[0123] in, Represents the pixel distribution intensity value of the first input image in the mth pixel subinterval, Indicates the value of the left boundary of the m-th pixel subinterval, represents the value of the right boundary of the mth pixel subinterval, h represents the smoothness parameter of the kernel function, n represents the total number of image pixels corresponding to the first input image, i represents the i-th image pixel corresponding to the first input image, and x i represents the pixel value of the i-th image pixel corresponding to the first input image, and σ(.) represents the Sigmoid function.

[0124] It should be noted that in this example, to improve the learning of the target domain image style, the histogram difference between the fake image and the unpaired real image is used as the optimization target, guiding the generator to produce fake images with consistent pixel distribution characteristics with the real image. Therefore, a kernel density estimation method is used to construct a differentiable histogram feature, using the derivative of the sigmoid function as the kernel function.

[0125] Optionally, the training process of the pre-trained bidirectional conversion framework includes:

[0126] Acquire a training sample set; the training sample set includes a plurality of optical sample images and SAR sample images;

[0127] Input the optical sample image into the initial bidirectional conversion framework for training;

[0128] The initial bidirectional conversion frame corresponding to the preset stopping condition is used as the pre-trained bidirectional conversion frame;

[0129] The preset stopping conditions include: the number of training times meets the preset iteration threshold or the value of the overall loss function is continuously less than the loss function threshold;

[0130] Among them, the structure of the initial bidirectional conversion framework is the same as that of the pre-trained bidirectional conversion framework.

[0131] In order to illustrate the effectiveness of the optical-SAR image conversion method based on coordinate attention and histogram loss provided by the embodiment of the present invention, a simulation experiment was also conducted as follows:

[0132] 1. Experimental conditions include:

[0133] The dataset uses optical and SAR images from the public datasets SEN1-2 as training and testing datasets. Based on the image capture time, four sub-datasets from the original dataset were selected as training datasets for the optical-SAR image conversion network. All images are 256×256 in size, and the number of images is shown in Table 1.

[0134] Table 1 Number of images in training and test datasets

[0135]

[0136]

[0137] The experimental hardware platform used an RTX 4090 GPU, and the software environment was based on the PyTorch 2.2.0 and CUDA 12.1 frameworks. The model was trained on the dataset for 100 epochs with a batch size of 1. The initial learning rate of the Adam optimizer was set to 0.0002, and the exponential decay factors of the first-order and second-order moments were 0.5 and 0.999, respectively. The learning rate began to decrease linearly from the 51st epoch and reached 0 after the 100th epoch. The channel compression ratio r in the semantic enhancement module was 8, and the number of stacked basic modules N was 9. The kernel function smoothness parameter h of the histogram loss was 2.5, and the pixel interval was divided into M = 256 subintervals.

[0138] To verify the effect of generating SAR images by the present invention, the image quality evaluation indicators structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) are used to measure the similarity between the generated image and the real image.

[0139] 2. The experimental results are as follows:

[0140] 2.1 Comparative test, the baseline model CycleGAN and the method proposed in this invention are trained on four sub-datasets, and the final quantitative evaluation results of image quality are shown in Table 2. The comparison results on different sub-datasets show that the method proposed in this invention can generate higher quality SAR images in various scenarios compared with the baseline model CycleGAN. In addition, Figure 5 Part of the SAR image results is shown as an example. Figure 5 The first column is the real optical image input to the generator, the second column is the fake SAR image generated based on the baseline model CycleGAN, the third column is the fake SAR image generated based on the method of the present invention, and the fourth column is the real SAR image. Figure 5 It can be seen that the fake SAR image generated by the method of the present invention retains the semantics of the original optical image while having a higher similarity with the real SAR image in terms of overall visual style.

[0141] Table 2 Quantitative evaluation results of the quality of generated fake SAR images

[0142]

[0143] 2.2 Ablation Experiment,In order to verify the influence of the semantic enhancement module and histogram loss proposed in this,an ablation experiment is carried out. The results are shown in Table 3.,It can be seen from Table 3 that adding the semantic enhancement module and,using the histogram loss on the basis of the baseline model can effectively improve the PSNR and,SSIM indicators of the generated results,indicating that the similarity between the generated results and the real SAR images,has been effectively improved. Figure 6 Shows a histogram comparison of the generated results of the baseline model before and after using histogram loss. Figure 6 Figure (a) shows the overall pixel distribution comparison between the real SAR image and the generated virtual SAR image before the baseline model uses the histogram loss. Figure 6 Figure (b) shows the overall pixel distribution comparison between the real SAR image and the generated virtual SAR image after the baseline model uses the histogram loss. Figure 6 It can be seen that before the baseline model CycleGAN uses the histogram loss, there is a significant difference in the overall pixel distribution between the generated image and the real SAR image. However, after using the histogram loss to guide the generator learning, this image pixel distribution difference can be effectively eliminated.

[0144] Table 3. Comparison results of ablation experiments

[0145]

[0146] 2.3 Parameter Experiments. To study the impact of different histogram loss weights on the generation results, experiments were conducted using different weighting factors β. The results are shown in Table 4. As the loss weight increases, the similarity between the generated virtual SAR images and the real SAR images gradually increases. As the loss weight continues to increase, the generation quality begins to decline. This is because histogram differences associated with different semantics always exist in unpaired images. Excessively large histogram losses force the generator to focus on meaningless semantic differences, potentially undermining the learning of overall style features. Therefore, to balance the learning effect of the generator, a histogram loss weight of β = 5.0 is used.

[0147] Table 4 Effect of different histogram loss weights β on image evaluation indicators

[0148]

[0149]

[0150] In summary, the method of the present invention can effectively retain the real semantic information of ground objects in the original optical image by designing a generative network structure that integrates stacked coordinate attention modules; it solves the SAR style learning problem in an unsupervised situation and effectively improves the image quality of the generated SAR images by constructing a differentiable histogram loss through kernel density estimation and ground motion distance.

[0151] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic devices can be desktop computers, portable computers, smart mobile terminals, servers, etc., which are not limited in the embodiment of the present invention.

[0152] It should be noted that the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0153] In the description of this specification, the reference terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0154] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the above-mentioned disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the word "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0155] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention cannot be considered to be limited to these descriptions. For those skilled in the art of the present invention, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the scope of protection of the present invention.

Claims

1. A light-SAR image conversion method based on coordinate attention and histogram loss, characterized in that: include: acquiring an optical image to be converted; Inputting the optical image to be converted into a pre-trained SAR image generator to convert the image into a target SAR image; The pre-trained SAR image generator is provided with a coordinate attention module, and the pre-trained SAR image generator is obtained based on a pre-trained bidirectional conversion framework; the coordinate attention module is used to perform global pooling in the horizontal and vertical directions respectively, and construct a global feature vector corresponding to the optical image to be converted based on the result of the global pooling; the pre-trained bidirectional conversion framework is provided with a conversion and discrimination architecture for converting optical images to SAR images and SAR images to optical images, and the overall loss function of the pre-trained bidirectional conversion framework is provided with a histogram loss; the histogram loss is constructed based on kernel density estimation and uses squared ground motion distance to determine the difference between image histograms.

2. The optical-SAR image conversion method based on coordinate attention and histogram loss according to claim 1, characterized in that: The pre-trained SAR image generator is provided with a feature encoder module, a semantic enhancement module and a feature decoder module connected in sequence; The feature encoder module is provided with a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence; The semantic enhancement module is provided with a plurality of semantic enhancement submodules, and the plurality of semantic enhancement submodules are connected in sequence, and each of the semantic enhancement submodules is provided with a fourth convolutional layer, a coordinate attention module, and a fifth convolutional layer connected in sequence; the input end of each semantic enhancement submodule is connected to the output end of the corresponding semantic enhancement submodule; The feature decoder module is provided with a first transposed convolutional layer, a second transposed convolutional layer and a third transposed convolutional layer connected in sequence; the output end of the first convolutional layer is also connected to the input end of the third transposed convolutional layer; the output end of the second convolutional layer is also connected to the input end of the second transposed convolutional layer.

3. The optical-SAR image conversion method based on coordinate attention and histogram loss according to claim 1, characterized in that: The pre-trained bidirectional conversion framework is provided with: a pre-trained SAR image generator, a pre-trained optical image generator, a pre-trained SAR image discriminator, and a pre-trained optical image discriminator; The pre-trained SAR image generator and the pre-trained SAR image discriminator are obtained based on adversarial training between the two; The pre-trained optical image generator and the pre-trained optical image discriminator are obtained based on adversarial training between the two; The pre-trained SAR image generator and the pre-trained optical image generator have the same structure; the pre-trained SAR image discriminator and the pre-trained optical image discriminator have the same structure.

4. The optical-SAR image conversion method based on coordinate attention and histogram loss according to claim 3, characterized in that: The pre-trained SAR image discriminator and the pre-trained optical image discriminator both adopt a PatchGAN structure, in which a plurality of sequentially connected preset convolutional layers are provided, and the convolution kernel size of the preset convolutional layer is 4×4.

5. The optical-SAR image conversion method based on coordinate attention and histogram loss according to claim 2, characterized in that: The processing process of the semantic enhancement module includes: Using the fourth convolutional layer to perform feature dimension reduction processing on the feature encoding result to obtain a first processing result; the feature encoding result is the output result of the feature encoder module; Performing a global average pooling operation on the first processing result along the horizontal direction and the vertical direction respectively, to obtain a horizontal feature vector and a vertical feature vector respectively; Performing feature splicing processing on the horizontal feature vector and the vertical feature vector to obtain a global feature vector; Performing feature dimension reduction processing on the global feature vector to obtain a second processing result; Performing feature separation processing on the second processing result along the horizontal direction and the vertical direction respectively, to obtain horizontal separation features and vertical separation features respectively; Performing channel recovery processing on the horizontal separation features and the vertical separation features respectively to obtain a horizontal direction attention map and a vertical direction attention map; Performing weighted fusion processing on the horizontal attention map, the vertical attention map, and the first processing result to obtain a weighted feature; Performing feature dimension recovery processing on the weighted features using the fifth convolutional layer to obtain a third processing result; The third processing result is superimposed on the feature encoding result to obtain a first semantic enhancement submodule feature; the first semantic enhancement submodule feature is the output feature corresponding to the first semantic enhancement submodule in the semantic enhancement module.

6. The optical-SAR image conversion method based on coordinate attention and histogram loss according to claim 1, characterized in that: The overall loss function corresponding to the pre-trained bidirectional conversion framework is the sum of adversarial loss, cycle consistency loss and histogram loss.

7. The optical-SAR image conversion method based on coordinate attention and histogram loss according to claim 6, characterized in that: The adversarial loss includes: generator adversarial loss and discriminator adversarial loss; The generator adversarial loss is expressed as: Among them, L G represents the value of the generator's adversarial loss, Represents the initial SAR image generator based on the real optical image x O The generated fake SAR image, express The discrimination results in the initial SAR image discriminator, Represents the initial optical image generator based on the real SAR image x S The generated false optical image, express The discrimination results in the initial optical image discriminator; The discriminator adversarial loss is expressed as: Among them, L D represents the value of the discriminator's adversarial loss, x O represents the real optical image, x S represents the real SAR image, D OPT (x O ) represents x O The discrimination result in the initial optical image discriminator, D SAR (x S ) represents x S The discrimination results in the initial SAR image discriminator; The cycle consistency loss is expressed as: Among them, L Cycle represents the value of cycle consistency loss, λ represents the first weight parameter, Indicates that the initial optical image generator is based on the false SAR image The generated image, Indicates that the initial SAR image generator is based on the false optical image For the generated image, ‖·‖1 represents the L1 norm; The histogram loss is expressed as: Among them, L Histo represents the value of histogram loss, β represents the second weight parameter, express The corresponding histogram features, Represents x O The corresponding histogram features, express The corresponding histogram features, Represents x S The corresponding histogram features, express and The corresponding squared ground motion distance, express and The corresponding squared earthquake distance.

8. The optical-SAR image conversion method based on coordinate attention and histogram loss according to claim 7, characterized in that: The squared ground motion distance is expressed as: Among them, SEMD(H1,H2) represents the squared ground motion distance between the histogram feature H1 of the first input image and the histogram feature H2 of the second input image, M represents the total number of intervals divided for image pixels, m represents the mth pixel subinterval, and CDF m (.) represents the cumulative distribution function; in, Represents the pixel distribution intensity value of the first input image in the mth pixel subinterval, Indicates the value of the left boundary of the m-th pixel subinterval, represents the value of the right boundary of the mth pixel subinterval, h represents the smoothness parameter of the kernel function, n represents the total number of image pixels corresponding to the first input image, i represents the i-th image pixel corresponding to the first input image, and x i represents the pixel value of the i-th image pixel corresponding to the first input image, and σ(.) represents the Sigmoid function.

9. The optical-SAR image conversion method based on coordinate attention and histogram loss according to claim 1, characterized in that: The training process of the pre-trained bidirectional conversion framework includes: Acquire a training sample set; the training sample set includes a plurality of optical sample images and SAR sample images; Inputting the optical sample image into an initial bidirectional conversion framework for training; The initial bidirectional conversion framework corresponding to the preset stop condition is used as the pre-trained bidirectional conversion framework; The preset stopping condition includes: the number of training times meets the preset iteration threshold or the value of the overall loss function is continuously less than the loss function threshold; The initial bidirectional conversion framework has the same structure as the pre-trained bidirectional conversion framework.

Images