Method for translating high-resolution synthetic aperture radar image to multispectral image

Abstract

The invention discloses a method for translating a high-resolution synthetic aperture radar image into a multispectral image, which comprises the following steps: constructing a generator network and a recognizer network wherein the generator network comprises a TV-BM3D module and a generator module; the recognizer network comprises a thermodynamic constraint module and a recognizer module; obtaining a training data set and training a generator network by using the training data set and the identifier network to obtain a trained identifier network; and inputting a to-be-processed original high-resolution SAR image into the trained recognizer network to obtain a corresponding multispectral image. According to the method, edge and texture details are reserved while high-resolution SAR speckle noise is suppressed by using a variational block matched filtering technology; conservation constraint is carried out on the energy flow in the generation process with the first law of thermodynamics as the prior, cross-modal mapping can be stabilized, and spectral fidelity and channel coordination of the multispectral image can be improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for translating high-resolution synthetic aperture radar images into multispectral images. Background Technology

[0002] Existing high-altitude remote sensing reconnaissance imagery includes synthetic aperture radar (SAR) imaging and multispectral imagery, which are of significant value in tasks such as forest resource assessment, agricultural monitoring, disaster response, and land cover monitoring. SAR employs a microwave active transmission and echo reception mechanism, enabling all-weather, day-and-night observation and stable image acquisition under complex atmospheric conditions. However, SAR images suffer from inherent speckle noise and high-frequency texture fluctuations, making direct interpretation difficult. Multispectral imaging, on the other hand, can simultaneously acquire multiple spectral bands, including visible and near-infrared light. Multispectral images offer higher spectral resolution and more intuitive interpretation, but are limited by sensor imaging conditions, such as at night and in adverse weather conditions like fog, haze, and cloud cover, making it difficult to acquire high-quality images. Therefore, how to acquire easily interpretable remote sensing images under adverse weather conditions has become a key research focus and challenge in remote sensing reconnaissance operations.

[0003] Currently, image conversion mainly employs inter-domain image translation methods. However, the mainstream approach is translation between natural images, without specifically addressing high-resolution remote sensing images. High-resolution SAR images are particularly prone to speckle noise and high-frequency disturbances, often inducing artifacts and detail distortion, resulting in low image quality and stability. Furthermore, multispectral images are constrained by correlation and spectral consistency, easily leading to false colors and channel inconsistencies. Additionally, publicly available data often comes from inconsistent sources, has low resolution, and is difficult to pair at the pixel level. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a method for translating high-resolution synthetic aperture radar images into multispectral images. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a method for translating high-resolution synthetic aperture radar images into multispectral images, comprising: S1: Construct a generator network and a recognizer network. The generator network includes a TV-BM3D module and a generator module. The TV-BM3D module performs coupled denoising processing on the input SAR image to obtain a denoised SAR image. The generator module uses the denoised SAR image to obtain a corresponding predicted multispectral image. The recognizer network includes a thermodynamic constraint module and a recognizer module. The thermodynamic constraint module performs thermodynamic constraint processing on the real multispectral image and the predicted multispectral image during the generator network training process to obtain constrained real multispectral images and predicted multispectral images. The recognizer module uses the constrained real multispectral images and predicted multispectral images to obtain recognition results and update the parameters of the generator network. S2: Obtain a training dataset and train the generator network using the training dataset and the recognizer network to obtain the trained recognizer network; S3: Input the original SAR image to be processed into the trained recognizer network to obtain the corresponding multispectral image.

[0005] In one embodiment of the present invention, the TV-BM3D module specifically includes a variational regularization submodule, a 3D block matching filter submodule, and a Laplacian pyramid submodule, wherein, The variational regularization submodule is used to perform preliminary denoising on the input SAR image by constraining the minimization of the image gradient norm, thereby obtaining a preliminary denoised SAR image. The three-dimensional block matching filtering submodule is used to filter and denoise the input SAR image by utilizing the statistical correlation of a large number of similar blocks in the SAR image, and obtain a filtered and denoised SAR image. The Laplacian pyramid submodule is used to perform image fusion on the initially denoised SAR image and the filtered denoised SAR image to obtain the final denoised SAR image.

[0006] In one embodiment of the present invention, the processing expression of the variational regularization submodule is: , in, This represents the input SAR image containing noise. This represents the SAR image after initial denoising by the variational regularization submodule. This represents total variational regularization, with parameters... is the regularization coefficient.

[0007] In one embodiment of the present invention, the three-dimensional block matching filter submodule includes a first SSIM block matching unit, an adaptive threshold transformation unit, a first inverse transform and aggregation unit, a second SSIM block matching unit, a collaborative filtering unit, and a second inverse transform and aggregation unit, wherein, The first SSIM block matching unit is used to divide the input SAR image into multiple image blocks of the same size, and to classify the image blocks using the SSIM method to form a first three-dimensional array; The adaptive threshold transformation unit is used to perform separable transformations on the first three-dimensional array in two spatial dimensions and one grouping dimension to obtain three-dimensional transformation coefficients, and to perform threshold filtering on the three-dimensional transformation coefficients using a set adaptive threshold to obtain threshold-filtered three-dimensional transformation coefficients. The first inverse transform and aggregation unit is used to perform an inverse separable transform on the threshold-filtered three-dimensional transform coefficients to obtain a denoised block group, and to superimpose each image block in the denoised block group back to the corresponding position of the original SAR image to obtain the output image of the first inverse transform and aggregation unit. The second SSIM block matching unit is used to divide the output image of the first inverse transform and aggregation unit into multiple image blocks of the same size, and to classify the current image blocks using the SSIM method to form a second three-dimensional array; The coordinated filtering unit is used to perform separable transformations on the second three-dimensional array in two spatial dimensions and one grouping dimension to obtain transform domain coefficients, and to perform Wiener filtering on the transform domain coefficients to obtain Wiener-filtered transform domain coefficients. The second inverse transform and aggregation unit is used to perform an inverse separable transform on the transform domain coefficients after Wiener filtering to obtain a denoised block group. Each image block in the current denoised block group is then superimposed back to the corresponding position of the original SAR image through a weighted average to obtain the output image of the second inverse transform and aggregation unit.

[0008] In one embodiment of the present invention, the first SSIM block matching unit is specifically used for: The input SAR image is divided into N×N image blocks of the same size, with each image block serving as a reference block. y Searching for the reference block in the SAR image y The K image blocks with the largest SSIM values ​​are used as the current reference blocks. y The similar blocks, where the SSIM values ​​of two image blocks are expressed as: , , , , in, Indicates reference block y With image blocks x SSIM value, Indicates reference block y With image blocks x The brightness similarity term, Indicates reference block y With image blocks x Contrast similarity term, Indicates reference block y With image blocks x Structural similarity terms, Represents image blocks x The average pixel value, Representation and Reference Block y The average pixel value, Represents image blocks x The standard deviation of pixels, Indicates reference block y The standard deviation of pixels, Represents image blocks x With reference block y pixel covariance, , and It is a constant.

[0009] In one embodiment of the present invention, the thermodynamic constraint module is specifically used for: The horizontal and vertical gradients of the input multispectral image are calculated respectively to obtain the horizontal gradient features and vertical gradient features of the multispectral image. The horizontal gradient features and vertical gradient features of the multispectral image are then fused to obtain the fused gradient features of the multispectral image. The fused gradient features are collected and aggregated using a pyramid model to obtain the aggregated features; Modulation gating is used to process the aggregation features of the real multispectral image and the aggregation features of the predicted multispectral image to obtain constrained images of the real multispectral image and the predicted multispectral image, respectively.

[0010] In one embodiment of the present invention, the recognizer module is used to receive the constrained image of the real multispectral image and the constrained image of the predicted multispectral image, output the recognition result, and update the parameters of the generator network according to the recognition result.

[0011] In one embodiment of the present invention, modulation gating is used to process the aggregated features of the real multispectral image and the aggregated features of the predicted multispectral image to obtain constrained images of the real multispectral image and the predicted multispectral image, respectively, including: Generate modulation gating to suppress regions in the image that differ significantly from the original image:

[0012] in, For the Sigmoid function, This represents the aggregated features of a real multispectral image or the aggregated features of a predicted multispectral image. This indicates the gating control strength, used to modulate the constraint strength in structurally sensitive areas; By employing residual reprojection, the constrained aggregation features are added to the input image of the thermodynamic constraint module to obtain the constrained image: , in, For element-wise multiplication, The image is either a real multispectral image or a constrained version of a predicted multispectral image.

[0013] In one embodiment of the present invention, S2 includes: A preset number of SAR images and corresponding multispectral images are acquired to form a paired dataset. Each image in the paired dataset is preprocessed to form a training dataset. The SAR images in the training dataset are input into the generator network, which outputs predicted multispectral images. The predicted multispectral image and the real multispectral image corresponding to the SAR image are input into the recognizer network to obtain the recognition result and update the parameters of the generator network according to the recognition result; The generator network is iteratively trained using the training dataset, and after the predetermined training requirements are met, the trained generator network is obtained.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a method for translating high-resolution synthetic aperture radar (SAR) images into multispectral images. First, a TV-BM3D module is embedded in the generator network. Through total variational regularization and block-matching 3D filtering, speckle noise in ultra-high-resolution SAR images is suppressed while preserving edges and textures, solving the problems of artifacts and detail loss leading to low-quality generation in traditional methods. Second, a thermodynamic constraint module is introduced into the discriminator network. Through a spectral channel attention mechanism, the correlation and consistency of multispectral channels are explicitly modeled, solving the problems of incomplete spectral information, false color, and channel inconsistency when only SAR images are converted to multispectral images, thereby improving the spectral realism of the generated multispectral images. Third, a pairwise high-resolution dataset with a spatial resolution of 0.3m is constructed, along with a unified polarization false color, radiometric and geometric consistency, and high-precision registration process. This solves the problems of insufficient model generalization and unreliable evaluation caused by low resolution of publicly available data, scarcity of paired samples, and inconsistent preprocessing, supporting the stable application of the method in multiple scenarios.

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method for translating high-resolution synthetic aperture radar images into multispectral images, provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the processing procedure for a high-resolution synthetic aperture radar image to multispectral image translation method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the processing procedure of a TV-BM3D module provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the processing procedure of a thermodynamic constraint module provided in an embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of the translation method from high-resolution synthetic aperture radar image to multispectral image proposed according to the present invention is provided in conjunction with the accompanying drawings and specific embodiments.

[0018] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0019] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element.

[0020] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart of a method for translating high-resolution synthetic aperture radar images into multispectral images, provided by an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the processing steps of a high-resolution synthetic aperture radar image to multispectral image translation method provided in an embodiment of the present invention. The translation method of this embodiment includes the following steps: S1: Construct the generator network and the recognizer network.

[0021] The generator network in this embodiment includes a TV-BM3D module and a generator module. The TV-BM3D module is used to perform coupled denoising processing on the input SAR image to obtain a denoised SAR image. The generator module is used to obtain a corresponding predicted multispectral image using the denoised SAR image.

[0022] The TV-BM3D module in this embodiment specifically includes a variational regularization submodule, a 3D block matching filter submodule, and a Laplacian pyramid submodule. The variational regularization submodule is used to perform preliminary denoising on the input SAR image by constraining the minimization of the image gradient norm, thereby obtaining a preliminary denoised SAR image. The 3D block matching filter submodule is used to filter and denoise the input SAR image by utilizing the statistical correlation of a large number of similar blocks in the SAR image, thereby obtaining a filtered and denoised SAR image. The Laplacian pyramid module is used to perform image fusion on the preliminary denoised SAR image output by the variational regularization submodule and the filtered and denoised SAR image output by the 3D block matching filter submodule, thereby obtaining the final denoised SAR image.

[0023] Specifically, the variational regularization submodule reduces noise sensitivity by constraining the minimization of the gradient norm of the SAR image, thus smoothing the background texture while preserving the edge structure of the output SAR image. Its expression is: , in, This represents the input SAR image containing noise. This represents the SAR image after initial denoising by the variational regularization submodule. This represents total variational regularization, with parameters... This is the regularization coefficient, used to balance data fidelity and smoothing size. When When the value is large, the regularization effect is enhanced, and the noise reduction effect is significant, but details may be over-smoothed; when When a smaller value is used, the regularization effect weakens, and residual noise may increase, but image details can be better preserved.

[0024] like Figure 3 As shown, the three-dimensional block matching filter submodule of this embodiment includes a first SSIM block matching unit, an adaptive threshold transformation unit, a first inverse transformation and aggregation unit, a second SSIM block matching unit, a collaborative filtering unit, and a second inverse transformation and aggregation unit.

[0025] The first SSIM block matching unit is used to divide the input SAR image into multiple image blocks of the same size and classify the image blocks using the SSIM method to form a first three-dimensional array. Specifically, while the SAR image to be processed is input to the variational regularization submodule, the SAR image is also input to the first SSIM block matching unit in the three-dimensional block matched filtering submodule. The first SSIM block matching unit divides the input SAR image into N×N image blocks of the same size, where N represents that each pixel block contains N×N pixels, such as each image block containing 8×8 or 16×16 pixels.

[0026] Next, each image block is used as a reference block. y Searching for reference blocks in SAR images y The K image patches with the highest SSIM (Structural Similarity Index) values ​​are selected as the current reference patches. y In a specific embodiment, when searching for similar blocks, the current reference block is used. y Centered on the image, a search is performed within a 39×39 grid in the image. The SSIM values ​​of two image patches are represented as follows: , , , , in, Indicates reference block y With image blocks x SSIM value, Indicates reference block y With image blocks x The brightness similarity term, Indicates reference block y With image blocks x Contrast similarity term, Indicates reference block y With image blocks x Structural similarity terms, Represents image blocks x The average pixel value, Representation and Reference Block y The average pixel value, Represents image blocks x The standard deviation of pixels, Indicates reference block y The standard deviation of pixels, Represents image blocks x With reference block y pixel covariance, , and It is a constant. It is a very small constant (to prevent the denominator from being zero), if the pixel mean values ​​of the two image patches are close ( ≈ ),but ≈1 indicates similar brightness; conversely, a large difference in brightness indicates a large difference in brightness. It is a very small constant, if the pixel standard deviations of two image patches are close to ( ≈ ),but ≈1 indicates similar contrast; conversely, a value of ≈1 indicates significant differences in the depth and brightness of the texture. It is a very small constant. If the pixel covariance of two image patches... (Right now ≈1) indicates that their pixel fluctuation patterns are consistent (such as the same edge position and similar texture direction), and their structures are highly similar; otherwise, their structures are very different (such as one is a smooth region and the other is an edge region).

[0027] Following the method described above, each image block is sequentially used as a reference block for traversal, thereby obtaining similar blocks for each reference block. Assume that K similar blocks have been found for each reference block through block matching (K is typically 16-64, depending on noise intensity and image content). These similar blocks have different spatial locations in the image but are highly similar in content. Next, the image blocks, grouped together, are stacked along the third dimension to form an N×N×K first three-dimensional block group.

[0028] Furthermore, the adaptive threshold transformation unit performs separable transformations on the first three-dimensional array in two spatial dimensions and one grouping dimension to obtain three-dimensional transformation coefficients, and uses a set adaptive threshold to perform threshold filtering on the three-dimensional transformation coefficients to obtain threshold-filtered three-dimensional transformation coefficients.

[0029] Specifically, the core of separable transformation is to perform one-dimensional transformations on the spatial dimension and the grouping dimension step by step, thereby reducing computational complexity. The specific operations are as follows: For each "slice" (i.e., a fixed third-dimensional index) of the first three-dimensional array described above k For an N×N two-dimensional block, first perform a one-dimensional transformation (such as DCT, Haar wavelet, or 5 / 3 lifting wavelet) along the row direction (first spatial dimension) to obtain an intermediate result; then perform a one-dimensional transformation along the column direction (second spatial dimension) to obtain the two-dimensional transformed result T. 2D ( x , y , k Essentially, this is equivalent to directly performing a two-dimensional transformation on each two-dimensional block, but it is easier to implement and can be parallelized through step-by-step one-dimensional transformations.

[0030] After completing the two-dimensional transformation of the spatial dimension, the transformed result T is... 2D ( x , y , k Grouping dimension (third dimension) k Perform a one-dimensional transformation (such as one-dimensional DCT or one-dimensional wavelet). At this point, each position of the three-dimensional block group ( x , y T corresponds to a one-dimensional sequence of length K, which, after transformation, yields T. 3D ( x , y , k The final result of the separable transformation is the three-dimensional transformation coefficients T. 3D Each of its elements T 3D ( x , y , k ) indicates the original block group at position ( x , y , k The transform domain response at ().

[0031] Finally, the portions of the obtained 3D transform coefficients smaller than a threshold are set to zero, while those larger than the threshold are retained. Specifically, in the transform domain, signal energy is concentrated in a few coefficients (e.g., large coefficients correspond to image structures), while noise is evenly distributed across all coefficients (small coefficients are mostly noise). Therefore, a threshold function is used to suppress small coefficients and retain large coefficients.

[0032] For example, the threshold is set to λ Typically related to the noise standard deviation σ Related, if | T 3D ( x , y , k )∣> λ ,reserve T 3D ( x , y , k Otherwise, set to zero. In application, an adaptive threshold adjustment mechanism is introduced. This mechanism adjusts the threshold for each spatial location based on the local noise estimation results. x , y ) and grouping dimensions k The threshold function is applied independently, meaning the set threshold is an adaptive threshold. For example, a larger threshold of 0.5 is used for regions with high noise intensity to achieve stronger denoising; a smaller threshold of 0.3 is used for regions with weak noise or complex details to retain more detail information, ultimately yielding the filtered transform coefficients. T ' 3D ( x , y , k ).

[0033] The first inverse transform and aggregation unit in this embodiment performs an inverse separable transform on the threshold-filtered 3D transform coefficients to obtain a denoised block group. Each image block in the denoised block group is then superimposed back to its corresponding position in the original SAR image to obtain the output image of the first inverse transform and aggregation unit. Specifically, the filtered transform coefficients... T ' 3D ( x , y , k The inverse transformation is performed, and then the weighted average of each block is superimposed back to the position of the original SAR image to form the output image of the first inverse transformation and aggregation unit.

[0034] The inverse transformation performed by the first inverse transformation and the inverse transformation of the aggregation unit is the reverse process of the forward transformation, and is executed step by step: (1) Inverse one-dimensional transformation of the grouping dimension For each spatial location ( x , y ),right T ' 3D ( x , y , k Along the grouping dimension k Performing an inverse one-dimensional transformation (such as inverse DCT) yields... T ' 2D ( x, y , k (coefficients after approximate two-dimensional transformation).

[0035] (2) Inverse one-dimensional transformation of spatial dimension right T ' 2D ( x , y , k First, perform an inverse one-dimensional transformation along the column direction, then perform an inverse one-dimensional transformation along the row direction to obtain the three-dimensional block group after the inverse transformation. B '( x , y , k (each slice) B '( , , k (This is a denoised 2D block).

[0036] Finally, block stacking is performed (each image block is stacked back to the corresponding position in the original SAR image) to obtain the output image of the first inverse transform and aggregation unit.

[0037] Furthermore, the second SSIM block matching unit is used to divide the output image of the inverse transform and aggregation unit into multiple image blocks of the same size, and to classify the current image blocks using the SSIM method to form a second three-dimensional array.

[0038] The structure and operation of the second SSIM block matching unit in this embodiment are the same as those of the first SSIM block matching unit described above. The only difference is that the input of the second SSIM block matching unit is the output image of the inverse transform and aggregation unit. For the specific operation process, please refer to the first SSIM block matching unit described above, which will not be repeated here.

[0039] The coordinating filtering unit performs separable transformations on the second three-dimensional array in two spatial dimensions and one grouping dimension to obtain transform domain coefficients. Wiener filtering is then applied to these coefficients to obtain Wiener-filtered transform domain coefficients. The second inverse transform and aggregation unit performs an inverse separable transformation on the Wiener-filtered transform domain coefficients to obtain denoised block groups. Each image block in the current denoised block group is then weighted and superimposed back to its corresponding position in the original SAR image to obtain the output image of the second inverse transform and aggregation unit.

[0040] Subsequently, the Laplacian pyramid submodule is used to fuse the initially denoised SAR image output from the variational regularization submodule and the filtered denoised SAR image output from the 3D block matching filter submodule to obtain the final denoised SAR image. This final denoised SAR image is then input into the trained generator module to obtain the predicted multispectral image.

[0041] As described above, the recognizer network in this embodiment includes a thermodynamic constraint module and a recognizer module. The thermodynamic constraint module is used to perform thermodynamic constraint processing on the real multispectral image and the predicted multispectral image during the generator network training process to obtain the constrained real multispectral image and the predicted multispectral image. The recognizer module is used to obtain the recognition result using the constrained real multispectral image and the predicted multispectral image and update the parameters of the generator network module.

[0042] Specifically, please see Figure 4 First, the thermodynamic constraint module calculates the horizontal and vertical gradients of the input multispectral image to obtain the horizontal and vertical gradient features of the multispectral image. The horizontal and vertical gradient features of the multispectral image are then fused to obtain the fused gradient features of the multispectral image. The pyramid model is then used to collect and aggregate the fused gradient features to obtain the aggregated features.

[0043] It should be noted that, in practice, the recognizer module is used to train the generator module in the generator network. Therefore, the input to the recognizer module is the constrained image of the real multispectral image and the constrained image of the predicted multispectral image. For this reason, the input to the thermodynamic constraint module is the predicted multispectral image output by the recognizer network and the corresponding real multispectral image from the training dataset.

[0044] Therefore, the thermodynamic constraint module in this embodiment is specifically used for: The horizontal and vertical gradients of the input real multispectral image are calculated separately to obtain the horizontal and vertical gradient features of the real multispectral image. The horizontal and vertical gradient features of the real multispectral image are then fused to obtain the fused gradient features of the real multispectral image. At the same time, the horizontal and vertical gradients of the predicted multispectral image are calculated separately to obtain the horizontal and vertical gradient features of the predicted multispectral image. The horizontal and vertical gradient features of the predicted multispectral image are then fused to obtain the fused gradient features of the predicted multispectral image. This predicted multispectral image is obtained by inputting the real SAR image corresponding to the current real multispectral image into the generator network.

[0045] Specifically, firstly, the horizontal and vertical gradients are calculated for the multispectral image (real or predicted multispectral image) input to the thermodynamic constraint module to capture edge and structural details. This step is performed in parallel by two independent convolutional branches: , , in, This is the input image for the thermodynamic constraint module. For convolution, These represent the gradient kernel in the horizontal direction and the gradient kernel in the vertical direction, respectively. Here, the gradient kernel can be either fixed or learnable. Represents the gradient characteristics in the horizontal direction. This represents the gradient characteristics in the vertical direction.

[0046] The two gradient features are then concatenated along the channel dimension and fused using a convolutional layer to form a comprehensive gradient representation. A pyramid model is then constructed for sampling aggregation (multi-scale aggregation) to serve as the basis for subsequent consistency constraints. , , in, Indicates channel-dimensional splicing. This represents a 1×1 convolution operation. This represents the features after a 1×1 convolution operation. and They represent the first Downsampling and upsampling operators at each scale, i.e., for Multi-scale downsampling and upsampling are performed and then aggregated to obtain aggregated features. That is, the aggregated features of real multispectral images or the aggregated features of predicted multispectral images.

[0047] Furthermore, the thermodynamic constraint module is specifically used to: process the aggregation features of the real multispectral image and the aggregation features of the predicted multispectral image using modulation gating, to obtain constrained images of the real multispectral image and the predicted multispectral image, respectively.

[0048] Specifically, a modulation gate is first generated to suppress regions in the image that differ significantly from the original image: , in, For the Sigmoid function, This represents the gating control strength, used to modulate the constraint intensity in structurally sensitive areas. Finally, residual reprojection is used to add the constrained aggregate features to the input image of the thermodynamic constraint module, preserving key information while smoothing noise and unrealistic deformations to obtain the constrained image: , in, For element-wise multiplication, Outputs either a real multispectral image or a constrained version of a predicted multispectral image.

[0049] As shown above, the input to the thermodynamic constraint module is the predicted multispectral image output by the recognizer network and the corresponding real multispectral image in the training dataset. The final output is the constrained image of the real multispectral image and the constrained image of the predicted multispectral image.

[0050] Next, the constrained real and predicted multispectral images are input into the generator module, which outputs a recognition result. This result determines the difference between the predicted and real multispectral images generated by the generator module and updates the parameters of the generator module in the generator network. The thermodynamic constraint module preprocesses the real and predicted multispectral images, making the discriminator module focus more on structural consistency rather than just local texture. This makes it more sensitive to and suppresses artifacts and spectral channel information biases in cross-modal mapping, ensuring that the energy budget of the generator domain and the information transfer of the source domain remain statistically consistent.

[0051] S2: Obtain the training dataset and use the training dataset and the recognizer network to train the generator network to obtain the trained recognizer network.

[0052] Specifically, a large number of SAR images and their corresponding multispectral images are acquired to form a paired dataset. Each image in the paired dataset is preprocessed to form a training dataset. The SAR images in the training dataset are input into the generator network, which outputs predicted multispectral images. The predicted multispectral images and the corresponding real multispectral images of the SAR images are input into the recognizer network to obtain the recognition results and update the parameters of the generator network based on the recognition results. The generator network is iteratively trained using the training dataset. After the predetermined training requirements are met, the trained generator network is obtained.

[0053] In practice, a large number of SAR images and their corresponding multispectral images are first paired into a dataset. This dataset covers typical land cover types, including urban buildings, roads, water bodies, vegetation, and bare land. SAR images and multispectral images of the same scene form an image pair. After acquiring the image data, each image undergoes geometric registration, radiometric correction, and cropping normalization to ensure consistency in spatial and spectral dimensions. All images have a spatial resolution of 0.3m, and after georegistration, they form one-to-one paired samples. Preprocessing involves radiometric calibration and speckle removal for the SAR images, followed by atmospheric correction and spectral normalization for the multispectral (MS) images. Subpixel-level registration and cropping are then performed on the paired images to form the training dataset. Next, the generator module in the generator network and the discriminator module in the discriminator network are initialized. The parameters of both the generator and discriminator modules are initialized using Xavier, and the learning rate is set to 1×10⁻⁶. -4 The optimizer is Adam.

[0054] Subsequently, the SAR images from the training dataset are input into the generator network. After image denoising by the TV-BM3D module, they are input into the generator module, which outputs a predicted multispectral image corresponding to the SAR image. Then, the real multispectral images paired with the SAR image from the training dataset and the predicted multispectral image output by the generator network are input into the recognizer network. The recognizer network outputs a recognition result, characterizing the accuracy of the current generator module, and updates the generator module's parameters using the generator network's output. Through multiple iterations, the accuracy of the generator module in image translation is continuously improved. When the preset number of training iterations or accuracy requirements are reached, training ends, and the trained generator module is obtained. At this point, the generator module can output high-resolution, multispectral consistent conversion result images, realizing the translation and generation of SAR to multispectral images. It should be noted that during training, only the parameters of the generator module are updated, not the parameters of the TV-BM3D module.

[0055] S3: Input the original high-resolution SAR image to be processed into the trained recognizer network to obtain the corresponding multispectral image.

[0056] Specifically, the original SAR image to be processed is output to the trained recognizer network. The original SAR image is denoised by the TV-BM3D module, and the denoised SAR image is output. Then, the denoised SAR image is translated into the corresponding multispectral image by the trained recognizer module.

[0057] To verify the effectiveness of the proposed high-resolution synthetic aperture radar image to multispectral image translation method, quantitative evaluations were performed on the multispectral images generated by the method of this invention and various existing methods based on existing networks. The SSIM, PSNR (Peak Signal-to-Noise Ratio), LPIPS (Learned Perceptual Image Patch Similarity), MSSSIM (Multi-Scale Structural Similarity Index Measure), and SAM (Spectral Angle Mapper) indices were obtained to assess the structural similarity and spectral consistency of the generated images. The results are shown in Table 1. Table 1 shows that the method of this invention achieves the highest PSNR and MSSSIM scores, indicating the best realism of the generated multispectral images. Meanwhile, the LPIPS and SAM indices are the lowest, demonstrating the highest spectral realism of the generated multispectral images, consistent with human visual observation.

[0058] Table 1. Evaluation results of multispectral images generated by the method of the present invention and existing methods

[0059] The proposed high-resolution synthetic aperture radar (SAR) image to multispectral image translation method has three innovations. The first innovation is the introduction of a TV-BM3D (Total Variation - Block-Matching 3D Filtering) module into the generator network. This combines total variational regularization with improved 3D block-matching filtering to suppress noise and preserve structure in SAR images within the feature layer. The second innovation involves incorporating an attention-based discrimination method for spectral channels. This multi-channel attention-based discrimination mechanism adaptively assigns weights to each band using global averaging and max pooling, a shared multilayer perceptron, and a sigmoid activation function. This dynamically strengthens key spectral channels, significantly enhancing the discriminator's ability to detect spectral distortion and cross-band inconsistencies, thus providing more effective spectral constraints on the generated results. The third innovation lies in the establishment of a paired dataset of high-resolution SAR images and multispectral images. Compared with publicly available datasets, the resolution is significantly improved, and the datasets undergo various preprocessing steps to provide the generator model with more sufficient learning priors and more reliable evaluation benchmarks, making them suitable for downstream tasks of SAR image and multispectral image translation.

[0060] In summary, the translation method of this invention addresses several issues. First, by embedding a TV-BM3D module in the generator network, it uses total variational regularization and block-matching 3D filtering to suppress speckle noise in ultra-high resolution SAR images while preserving edges and textures. This solves the problems of artifacts and detail loss leading to low generation quality in traditional methods. Second, by introducing a thermodynamic constraint module in the discriminator network, it explicitly models multispectral channel correlation and spectral consistency through a spectral channel attention mechanism. This solves the problems of incomplete spectral information, false color, and channel inconsistency when only converting SAR images to multispectral images, thereby improving the spectral realism of the generated multispectral images. Third, by constructing a pairwise high-resolution dataset with a spatial resolution of 0.3m and providing a unified polarization false color, radiometric and geometric consistency, and high-precision registration process, it solves the problems of insufficient model generalization and unreliable evaluation caused by low resolution of publicly available data, scarcity of paired samples, and inconsistent preprocessing. This supports the stable application of the method in multiple scenarios.

[0061] Another embodiment of the present invention provides a storage medium storing a computer program for executing the steps of the high-resolution synthetic aperture radar image-to-multispectral image translation method described in the above embodiments. A further aspect of the present invention provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the high-resolution synthetic aperture radar image-to-multispectral image translation method described in the above embodiments. Specifically, the integrated modules implemented as software functional modules can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0062] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for translating high-resolution synthetic aperture radar images into multispectral images, characterized in that, include: S1: Construct a generator network and a recognizer network. The generator network includes a TV-BM3D module and a generator module. The TV-BM3D module performs coupled denoising processing on the input SAR image to obtain a denoised SAR image. The generator module uses the denoised SAR image to obtain a corresponding predicted multispectral image. The recognizer network includes a thermodynamic constraint module and a recognizer module. The thermodynamic constraint module performs thermodynamic constraint processing on the real multispectral image and the predicted multispectral image during the generator network training process to obtain constrained real multispectral images and predicted multispectral images. The recognizer module uses the constrained real multispectral images and predicted multispectral images to obtain recognition results and update the parameters of the generator network. S2: Obtain a training dataset and train the generator network using the training dataset and the recognizer network to obtain the trained recognizer network; S3: Input the original SAR image to be processed into the trained recognizer network to obtain the corresponding multispectral image.

2. The method for translating high-resolution synthetic aperture radar images into multispectral images according to claim 1, characterized in that, The TV-BM3D module specifically includes a variational regularization submodule, a 3D block matching filter submodule, and a Laplacian pyramid submodule, wherein... The variational regularization submodule is used to perform preliminary denoising on the input SAR image by constraining the minimization of the image gradient norm, thereby obtaining a preliminary denoised SAR image. The three-dimensional block matching filtering submodule is used to filter and denoise the input SAR image by utilizing the statistical correlation of a large number of similar blocks in the SAR image, and obtain a filtered and denoised SAR image. The Laplacian pyramid submodule is used to perform image fusion on the initially denoised SAR image and the filtered denoised SAR image to obtain the final denoised SAR image.

3. The method for translating high-resolution synthetic aperture radar images into multispectral images according to claim 2, characterized in that, The processing expression of the variational regularization submodule is: ， in, This represents the input SAR image containing noise. This represents the SAR image after initial denoising by the variational regularization submodule. This represents total variational regularization, with parameters... This is the regularization coefficient.

4. The method for translating high-resolution synthetic aperture radar images into multispectral images according to claim 2, characterized in that, The three-dimensional block-matched filtering submodule includes a first SSIM block matching unit, an adaptive threshold transformation unit, a first inverse transformation and aggregation unit, a second SSIM block matching unit, a collaborative filtering unit, and a second inverse transformation and aggregation unit, wherein... The first SSIM block matching unit is used to divide the input SAR image into multiple image blocks of the same size, and to classify the image blocks using the SSIM method to form a first three-dimensional array; The adaptive threshold transformation unit is used to perform separable transformations on the first three-dimensional array in two spatial dimensions and one grouping dimension to obtain three-dimensional transformation coefficients, and to perform threshold filtering on the three-dimensional transformation coefficients using a set adaptive threshold to obtain threshold-filtered three-dimensional transformation coefficients. The first inverse transform and aggregation unit is used to perform an inverse separable transform on the threshold-filtered three-dimensional transform coefficients to obtain a denoised block group, and to superimpose each image block in the denoised block group back to the corresponding position of the original SAR image to obtain the output image of the first inverse transform and aggregation unit. The second SSIM block matching unit is used to divide the output image of the first inverse transform and aggregation unit into multiple image blocks of the same size, and to classify the current image blocks using the SSIM method to form a second three-dimensional array; The coordinated filtering unit is used to perform separable transformations on the second three-dimensional array in two spatial dimensions and one grouping dimension to obtain transform domain coefficients, and to perform Wiener filtering on the transform domain coefficients to obtain Wiener-filtered transform domain coefficients. The second inverse transform and aggregation unit is used to perform an inverse separable transform on the transform domain coefficients after Wiener filtering to obtain a denoised block group. Each image block in the current denoised block group is then superimposed back to the corresponding position of the original SAR image through a weighted average to obtain the output image of the second inverse transform and aggregation unit.

5. The method for translating high-resolution synthetic aperture radar images into multispectral images according to claim 4, characterized in that, The first SSIM block matching unit is specifically used for: The input SAR image is divided into N×N image blocks of the same size, with each image block serving as a reference block. y Searching for the reference block in the SAR image y The K image blocks with the largest SSIM values ​​are used as the current reference blocks. y The similar blocks, where the SSIM values ​​of two image blocks are expressed as: ， ， ， ， in, Indicates reference block y With image blocks x SSIM value, Indicates reference block y With image blocks x The brightness similarity term, Indicates reference block y With image blocks x Contrast similarity term, Indicates reference block y With image blocks x Structural similarity terms, Represents image blocks x The average pixel value, Representation and Reference Block y The average pixel value, Represents image blocks x The standard deviation of pixels, Indicates reference block y The standard deviation of pixels, Represents image blocks x With reference block y pixel covariance, , and It is a constant.

6. The method for translating high-resolution synthetic aperture radar images into multispectral images according to claim 1, characterized in that, The thermodynamic constraint module is specifically used for: The horizontal and vertical gradients of the input multispectral image are calculated respectively to obtain the horizontal gradient features and vertical gradient features of the multispectral image. The horizontal gradient features and vertical gradient features of the multispectral image are then fused to obtain the fused gradient features of the multispectral image. The fused gradient features are collected and aggregated using a pyramid model to obtain the aggregated features; Modulation gating is used to process the aggregation features of the real multispectral image and the aggregation features of the predicted multispectral image to obtain constrained images of the real multispectral image and the predicted multispectral image, respectively.

7. The method for translating high-resolution synthetic aperture radar images into multispectral images according to claim 6, characterized in that, The recognizer module is used to receive the constrained image of the real multispectral image and the constrained image of the predicted multispectral image, output the recognition result, and update the parameters of the generator network according to the recognition result.

8. The method for translating high-resolution synthetic aperture radar images into multispectral images according to claim 6, characterized in that, Modulation gating is used to process the aggregated features of the real multispectral image and the aggregated features of the predicted multispectral image to obtain constrained images of the real multispectral image and the predicted multispectral image, respectively, including: Generate modulation gating to suppress regions in the image that differ significantly from the original image: in, For the Sigmoid function, This represents the aggregated features of a real multispectral image or the aggregated features of a predicted multispectral image. This indicates the gating control strength, used to modulate the constraint strength in structurally sensitive areas; By employing residual reprojection, the constrained aggregation features are added to the input image of the thermodynamic constraint module to obtain the constrained image: ， in, For element-wise multiplication, The image is either a real multispectral image or a constrained version of a predicted multispectral image.

9. The method for translating high-resolution synthetic aperture radar images into multispectral images according to claim 6, characterized in that, S2 includes: A preset number of SAR images and corresponding multispectral images are acquired to form a paired dataset. Each image in the paired dataset is preprocessed to form a training dataset. The SAR images in the training dataset are input into the generator network, which outputs predicted multispectral images. The predicted multispectral image and the real multispectral image corresponding to the SAR image are input into the recognizer network to obtain the recognition result and update the parameters of the generator network according to the recognition result; The generator network is iteratively trained using the training dataset, and the trained generator network is obtained after the predetermined training requirements are met.

Images