Method for the computer-aided generation of a data-driven model for the computer-aided processing of a single-look complex (SLC) image

Abstract

The invention relates to a method for the computer-aided generation of a data-driven model for the computer-aided processing of a single-look complex (SLC) image (10) acquired with a SAR system configured to obtain radar reflection, wherein the SLC image (10) comprises a real part and an imaginary part, and wherein the data-driven model is configured to jointly remove speckle from and compress the SLC image (10). The generation of the data-driven model comprises the following steps: - Provision of a training data set (TDS) comprising a plurality of SLC images (10), each of which includes a real part and an imaginary part; - Training a neural network (NN) with the training data set (TDS), wherein the neural network (NN) comprises an input for inputting either the real part or the imaginary part of a respective SLC image (10) of the plurality of SLC images (10) as input information, wherein a latency space (y) resulting from the encoding of the input information with an encoder (100) is concatenated and compressed to form a compressed bitstream (MBS, SBS), and an output for outputting an estimated intensity (Î), wherein the estimated intensity (1) results from the splitting and decoding of the compressed bitstream (MBS, SBS), wherein the trained neural network (NN) represents the generated data-driven model.

Description

[0001] The present invention relates to a method and apparatus for the computer-aided generation of a data-based model for the computer-aided processing of a single-look complex (SLC) image acquired with a synthetic aperture radar (SAR) system configured to obtain radar reflectance. Furthermore, the invention relates to a method for the computer-aided processing of an SLC image acquired with a SAR system configured to obtain radar reflectance, as well as to a computer program product.

[0002] Over the past decade, space-based synthetic aperture radar (SAR) images have proven extremely useful for a wide range of civilian and military applications, from deforestation monitoring to dark vessel detection. However, capturing, transmitting, and processing these images presents challenges in many ways.

[0003] To transmit the data collected by a satellite to the ground, a downlink is established by pointing a ground station antenna toward the passing satellite. The maximum available downlink duration limits the amount of data that can be transmitted to the ground. Furthermore, downlink operation is costly, as it is a technical task requiring high-end hardware, and few companies offer it as a pay-per-use service. For these two reasons, it is desirable to compress the transmitted data as much as possible to maximize the amount of data transmitted while minimizing costs.

[0004] Recent work on end-to-end compression of optical images has shown that deep learning can outperform some of the best classical methods, such as JPEG2000, a compression standard based on wavelet transforms. However, the usability of SAR images is limited due to their severe perturbation by a multiplicative and spatially correlated pseudo-noise effect called speckle. Removing speckle while preserving resolution and image features is one of the most critical tasks in SAR imaging and hinders the potential interpretation of SAR results. In recent decades, the task of speckle removal has been addressed using various methods, such as filtering in a transformed space, the so-called homomorphic approach, as well as local and nonlocal methods.Despeckling has also shown some promising results using deep learning-based approaches. However, the lack of noise-free SAR images, i.e., ground truth, for training leads to more complex training approaches.

[0005] Recently, a training method independent of a deep learning architecture has demonstrated state-of-the-art despeckling performance while being simpler to implement than other supervised learning methods: A novel self-supervised training strategy was developed that assumes that the real and imaginary parts of an SLC image—that is, a focused complex image—contain independent realizations of speckle noise. This makes it possible to train a deep learning model with one realization as input and the other as reference. Since the network treats speckle as random noise, the model would be unable to reproduce it, resulting in a despeckled network.

[0006] In the area of ​​SAR image compression, deep learning approaches have recently been proposed. These methods focus solely on compression, without performing despeckling.

[0007] The training of a neural network for denoising (or estimating reflectivity) SAR images can be formalized as the selection of an architecture (also called a model) trained on a pair of (input / reference) data. Some training strategies may involve several steps of data preprocessing, several training steps with pairs of different types, and a specific use of the trained model to make the desired predictions. An example of such a multi-step approach can be found in the so-called SAR2SAR architecture of reference [1], where the proposed algorithm is trained by minimizing a difference between the input and output images to obtain a speckle-free version of the image.

[0008] Furthermore, self-supervised training strategies are known, which are particularly attractive because they have the advantage of requiring neither ground truth nor synthetic images or synthetic noise. However, supervised training strategies also show good results in the literature. For example, one can perform temporal multi-looking to generate a ground truth image with up to 24 looks (as described in Reference [2]). This data can then be used to train a supervised network, where the (input / reference) pairs each consist of a single-look image and the multi-look image. However, this method requires a large amount of data from the same area. In addition, all these images must be co-registered, the temporal changes must be detected, and a way to deal with them (i.e., masking the detected areas that have changed), all of which is not a trivial task.

[0009] Recently, a novel self-supervised training approach was developed and published in [3], using the so-called "MERLIN" strategy, which is based on the idea that the real and imaginary parts of a SLC are two independent, identically distributed realizations of the same signal. The fact that the real and imaginary parts are two realizations of the same signal implies, in particular, two realizations of the noise signals. Based on this, one can use the squared real part and the squared imaginary part as (input / reference) pairs to train a neural network. However, these pairs exhibit only half the performance (found in each real or imaginary part) compared to training a network using the total performance of an intensity image. Regarding self-supervised training strategies, it can be noted that they have advantages over other supervised methods.Thus, it is much easier to create a dataset, avoid a range shift between the type of training and the application of the network (as shown in [4]), and they are easier to implement compared to other training strategies.

[0010] Regarding image compression, variational auto-encoders, a type of machine learning (ML) architecture, have achieved good results in the literature, offering even better compression rates compared to classical methods (see, e.g., references [5] and [6]). SAR image compression using variational auto-encoders has also achieved some success in recent years. However, such architectures operate directly on detected images rather than directly on SLC images (see, e.g., [7] and [8]).

[0011] Finally, combined decompression and denoising approaches for optical images have also been investigated in the literature with good performance using ML approaches, as described in references [9] and

[10] . However, they are not suitable for SAR images, since only work on compression and despeckling has been performed using classical methods, as outlined in references

[11] and

[12] , and thus no state-of-the-art ML-based compression methods have been used.

[0012] The above-mentioned algorithms for compressing SAR images do not take into account that despeckling the signal before compression leads to higher compression rates and more efficient data transmission. Current machine learning-based deep speckling algorithms do not consider the compression task in a comprehensive manner.

[0013] It would be desirable to have an algorithm that performs despeckling and compression jointly. To properly address this problem, the data used for training would need to be the same data that is also compressed / filtered, thus avoiding synthetic data. Furthermore, both tasks—despeckling and compression—would have to be solved simultaneously and optimized accordingly. Known algorithms that combine compression and despeckling are based on optical satellite imagery, not SAR data, and are therefore unsuitable for processing SAR data.

[0014] It is therefore an object of the present invention to provide a method and apparatus for processing single-look complex (SLC) images acquired with a SAR system configured to obtain radar reflectance and capable of jointly compressing and speckle-removing the SLC images.

[0015] This object is achieved by a method for computer-aided generation of a data-based model for computer-aided processing of the SLC image according to claim 1, a method for computer-aided processing of an SLC image according to claim 10, a device for computer-aided processing of an SLC image according to claim 14 and a computer program product according to claim 15.

[0016] Preferred embodiments are specified in the dependent claims.

[0017] According to a first aspect, a method is provided for the computer-aided generation of a data-driven model for the computer-aided processing of a single-look complex (SLC) image acquired with a SAR system configured to obtain radar reflectance. The SLC image comprises a real part and an imaginary part. The data-driven model is configured to jointly de-speckle and compress the SLC image.

[0018] The generation of the data-driven model comprises the following steps: according to a first step, a training dataset is provided comprising a plurality of SLC images, each of which comprises a real part and an imaginary part; according to a second step, a neural network is trained with the training dataset, wherein the neural network comprises an input for inputting either the real part or the imaginary part of a respective SLC image of the plurality of SLC images as output information, wherein a latent space resulting from the encoding of the input information with an encoder is concatenated and compressed into a compressed bitstream, and an output for outputting an estimated intensity. The estimated intensity results from the division and decoding of the compressed bitstream, wherein the trained neural network represents the generated data-driven model.

[0019] The combined approach of compression and despeckling has the advantage that encoded and despeckle-free images result in better compression performance, as the overall entropy of the signal decreases after despeckling. Furthermore, the resulting reflectivity can be estimated and compressed.

[0020] The inventive method is based on the concept of self-supervision for SAR despeckling in conjunction with the latest compression methods based on deep network architectures. The method relies on a previously unpaired architecture and training strategy. The method for training a neural network is based on pairs of real / imaginary intensity images (SLC images). During training, the resulting latent space is concatenated to train the network to compress two latent spaces during actual application of the network. Concatenating the real part of the imaginary part of an SLC image means doubling the number of channels to a desired number.

[0021] According to a preferred embodiment, the total reflectivity of the pixels in the SLC image is obtained, particularly above a predetermined threshold corresponding to point scatterers, and the total reflectivity is divided into two halves by assigning it to its imaginary and real parts, respectively. This ensures that the neural networks can account for the deterministic phases present in the real data. The predetermined threshold can be set to 9 dB, for example, which corresponds to point scatterers.

[0022] According to a further preferred embodiment, the trained neural network is based on a variational autoencoder with hyperpriority to process compression and despeckling simultaneously.

[0023] As such, the inventive method can use the self-monitoring approach of noise2noise, MERLIN, etc., which are known from the prior art, e.g., according to references [1] and [3].

[0024] According to another preferred embodiment, the concatenated latent space is transmitted to a first quantizer and a first arithmetic encoder to output a de-speckled and compressed main bitstream, and to a hyper-encoder, the output of which is transmitted to a second quantizer and a second arithmetic encoder to output a de-speckled and compressed side bitstream. The arithmetically decoded side bitstream is transmitted to a hyper-decoder to estimate scales encoded in the main bitstream for further compression.

[0025] According to a further preferred embodiment, the squared real part or the squared imaginary part of a respective SLC image is input into a neural network.

[0026] Preferably, the training of the neural network comprises the calculation of a loss function that takes into account both the compression and the reflectivity estimation, with the possibility of emphasizing one or the other by including a parameter λ. Preferably, the loss corresponds to a relaxed rate-distortion optimization problem according to L(I^,Im)=R+λD(I^k,Imk) where (L) the loss function, Î the estimated intensity (output) for the neural network, Re 2 or in 2 the squared real or imaginary part (input),

[0027] D is the joint despeckling / distortion loss, R is the entropy of the latents ŷ and ẑ, and λ is a Lagrange multiplier.

[0028] The loss function L is used to train the weights of the neural network.

[0029] According to a second aspect of the present invention, a method is provided for computer-aided processing of an SLC image acquired with a SAR system configured to obtain radar reflectance. The SLC image comprises a real part and an imaginary part. Both are input into a data-driven model. The estimated intensity image of the SLC image is generated using a data-driven model generated using a method according to one or more embodiments of the invention.

[0030] Preferably, the real part and the imaginary part of the SLC image are encoded and concatenated.

[0031] It is further preferred that the real part and the imaginary part of the SLC image are encoded with the same encoder.

[0032] According to a further preferred embodiment, the concatenated latent space is split into two latent spaces on the decoder side and then decoded into their respective estimates, the estimated intensity image being obtained after adding the two estimates.

[0033] In contrast to training, two different but very similar latent spaces are obtained, which are concatenated and then compressed. On the decoder side, the concatenated latent space is first split into two parts and then decoded into the respective estimates of the intensity image. The final intensity image is then obtained by summing the two estimates. This provides a very efficient method for jointly despecking and compressing SLC images.

[0034] According to a third aspect, an apparatus is provided for computer-aided processing of an SLC image acquired with a SAR system configured to obtain radar reflectance, wherein the SLC image comprises a real part and an imaginary part, both of which are input to a data-driven model. An estimated intensity image of the SLC image is generated, the apparatus comprising a data-driven model generated by a method according to one or more embodiments of the present invention.

[0035] According to a fourth aspect of the present invention, a computer program product is provided comprising program code stored on a machine-readable medium for performing a method according to one or more embodiments of the present invention when the program code is executed on a computer.

[0036] The invention will be described in more detail with reference to the accompanying drawings. Fig. 1 shows a schematic representation of a neural network architecture with hyperprior, which is used to train the neural network according to the invention. Fig. Figure 2 shows a schematic representation of the architecture of the neural network used for processing SLC images to obtain the radar reflectance. Fig. Figure 3 shows a graph illustrating rate-distortion performance based on a peak signal-to-nose ratio (PSNR) metric and the resulting bits per pixel (BPP).

[0037] The present invention relates to a self-monitoring concept for SAR despeckling, along with compression methods based on deep network architectures. The data-driven model, which is a trained neural network (NN) described below, simultaneously despeckles and compresses.

[0038] The proposed method relies on a previously unpaired architecture and training strategy. That is, training and prediction are performed in two different ways to optimize the compression task.

[0039] Fig.Figure 1 shows a proposed network architecture using Hyperprior to generate the neural network for the computational processing of an SLC image 10 acquired with a SAR system (not shown) configured to obtain the radar reflectance. As is known to those skilled in the art, the SLC image 10 includes a real part and an imaginary part. The neural network is trained using either the squared real part Re 2 or the squared imaginary part 2 of a corresponding SLC image 10. The neural network NN is described in more detail below.

[0040] The neural network NN comprises the components of a transmitter 100 and a receiver 200. The transmitter 100 consists of an encoder 110, a concatenation unit 130, a quantizer 131, an arithmetic encoder 132, and a hyper-encoder 140, followed by a quantizer 151 and an arithmetic encoder 152.

[0041] The receiver 200 consists of a hyper-decoder 240, an arithmetic decoder 250, a splitter 251, and a decoder 210 that provides an estimated intensity Î of an intensity image 20. In general, the neural network NN is based on the well-known self-supervised approach as described in references [1] and [3]. In addition to these self-supervised approaches, the concatenation unit 130 and the splitter 251 are added.

[0042] The encoder 110 comprises - in the following order - a down-sampling unit 111, an activation function (GDN) 112, a residual block 113, a down-sampling unit 114, an activation function (GDN) 115, a residual block 116, a down-sampling unit 117, an activation function (GDN) 118, a residual block 119 and a down-sampling unit 120. An input of the encoder 110 receives the squared real part Re 2 or the squared imaginary part2 of the SLC image 10 and outputs a latent space y resulting from the coding of the input information (Re 2 or in 2 ) with the encoder 110. The concatenation unit 130 concatenates the latent space y during training, ie, the number of channels is doubled. The concatenated latent space y c is passed to both the quantizer 131 and the hyper encoder 140.

[0043] The hyper-encoder 140 consists of an absolute value unit 141, a down-sampling unit 142, an activation function (ReLU) 143, a down-sampling unit 144, an activation function (ReLU) 145 and a down-sampling unit 146. The coded concatenated latent space y cis passed to the quantizer 151 and then to the arithmetic encoder 152 to provide a side bitstream SBS as the output of the transmitter 100. The bitstream SBS, received as input information by the receiver 200, is decoded by the arithmetic decoder 251 and passed to the hyper decoder 240 of the receiver 200.

[0044] The hyper-decoder 240 consists of an up-sampling unit 241, an activation function (ReLU) 242, an up-sampling unit 243, an activation function (ReLU) 244, and an up-sampling unit 245. The hyper-decoder 240 provides an estimated scaling σ̂, which is forwarded to the arithmetic decoder 250 of the receiver 200 and the arithmetic encoder 132 of the transmitter 100. The arithmetic encoder 132 uses the estimated scaling σ̂ to generate the quantized concatenated latent space ŷ cinto a bit symbol and makes it available as the main bit stream MBS (another output of the transmitter 100). The main bit stream MBS is also forwarded to the arithmetic decoder 250 of the receiver 200. This decoder decodes the main bit stream MBS using the estimated scaling σ̂ obtained from the hyper decoder. The output of the arithmetic decoder 250 is forwarded to the splitter 251, which provides a quantized latent space ŷ.

[0045] The quantized latent space ŷ represents the input information of the decoder 210. The decoder 210 consists of an up-sampling unit 211, an activation function (IGDN) 212, a residual block 213, an up-sampling unit 214, an activation function (IGDN) 215, a residual block 216, an up-sampling unit 217, an activation function (IGDN) 218, a residual block 219, and an up-sampling unit 220. The output of the decoder represents the output of the neural network NN and outputs the estimated intensity Î, where the estimated intensity Î results from the division and decoding of the compressed bit stream. 20 represents the denoised intensity image, which is amplified with the squared real part Re 2 or the squared imaginary part 2 of the SLC image 10.

[0046] Fig. Figure 2 shows the neural network used to predict the estimated intensity Î of an SLC image.

[0047] The complex SLC image 10 is divided into a first part, which contains the real part 10 Re and a second part containing the imaginary part 10 Im comprises, divided, whereby both the squared real part Re 2 as well as the squared imaginary part 2 of the SLC image 10 (10 Re , 110 Im ) are used as input for the neural network NN. Both the squared real part Re 2 as well as the squared imaginary part 2 of the SLC image 10 are obtained using the neural network NN, as shown in Fig. 1 described in Latent Dreams y Re and y Im coded to be concatenated by the concatenation unit 130. It should be noted that, in contrast to training, the square of the real part and the imaginary part is 10 re and 10 Imare input to the encoder 110. The same encoder 110 with common weights is used to encode the respective input information. In contrast to training, two different but very similar latent dreams y are obtained. Re and y Im , which are concatenated and then compressed.

[0048] On the decoder side, the concatenated latent space is first split into two parts using the splitting unit 252 (from which ŷ Re and ŷ Im result) and then into their respective intensity estimates Î Re and Î Im decoded, resulting in the respective intensity images 15 Re and 15 Im The intensity images 15 Re and 15 Im are then combined by addition, resulting in the combined intensity image 20.

[0049] In Fig.2, the components circled with a solid line represent the components of the transmitter 100, while the components circled with a dashed line represent the components of the receiver 200. The hyper-decoder 240 appears in both the transmitter 100 and the receiver 200 because the estimated scalings σ̂ at the time of compression are known to the encoder 110 and the decoder 210.

[0050] This results in a self-supervised machine learning method for estimating and compressing the reflectivity of SAR images using neural networks, in particular convolutional neural networks. Despeckling SAR images with CNNs is known from the prior art, in particular from [3]. The approach proposed in [3] allows extending the original training strategy by using a variational autoencoder to focus on the task of compression and simultaneous despeckling. Furthermore, the radiometric preservation of the resulting speckle-free images is improved. The present invention goes beyond this proposal.

[0051] It is known that point scatterers exhibit a deterministic phase that appears different in the real and imaginary parts of the SLC image. This effect leads to inaccurate modeling of such scatterers, which affects the radiometric accuracy during training and subsequently in reflectivity estimation. It is therefore proposed to improve the self-supervised training described in [3] by determining the total reflectivity of pixels above 9 dB (corresponding to the mentioned scatterers) and splitting it in half during the training phase, respectively assigning it to the imaginary and real parts. This ensures that the neural network can account for the deterministic phases present in the real data.

[0052] Compressing SAR images with CNNs is already known from the state of the art. Deep learning-based image compression methods have several advantages over classical compression methods such as JPEG and JPEG2000. In particular, deep learning-based methods can learn important features in SAR images and avoid compressing redundant information, enabling better compression performance in terms of rate-distortion ratios and visual clarity. The methods mentioned in references

[13] and

[14] attempt to compress the already estimated (noisy) reflectivity derived from the detection of SLC images, further compressing redundant information in the form of speckle, but ignoring this information does not improve the compression rate.In contrast, the neural network proposed and described above uses both the variational auto-encoder scheme with hyperprior, but trains the network in such a way that the noise component is not estimated, thus increasing the compression rate.

[0053] To be able to jointly compress and de-speckle SLC images, the proposed neural network relies on a previously unpaired architecture and training strategy. Furthermore, the training and prediction steps are unique—for example, the network is used differently when performing an inference task compared to training—and are enhanced to best address the joint compression and de-speckle challenges.

[0054] Therefore, a special loss function is used to train the neural network (NN), which considers both compression and reflectivity estimation, while allowing for the possibility of emphasizing one or the other using a parameter, λ. The loss is formulated as a relaxed rate-distortion optimization problem (R function for estimation and D for distortion estimation) with the introduction of the Lagrange multiplier λ. The distortion term serves not only to measure the difference between the input image, but also between a pair of noisy images to estimate the noise-free reflectivity, while the R term serves to minimize the resulting bit rate.

[0055] The overall loss function is based on the rate-distortion optimization of two terms representing compression and distortion / despeckling, which are specified as follows: L(I^,Im)=R+λD(I^k,Imk)=R(y^)+R(z^)+ +λ∑k12log I^k+exp(2log|Imk|−log I^k) where D is the joint despeckling / distortion loss, R is the entropy of the latents ŷ and Ẑ. The input Re 2 (or in 2 ) and the output (estimated) intensity Î to the network are log-transformed and normalized within the network.

[0056] The considered loss function corresponds to a training strategy in which the input of the network Re 2 the initial intensity Î, while the squared imaginary part Im is used as a reference for performing a gradient descent. 2 To train the network with more examples, Re 2 and In 2 be randomly swapped, as in Fig. 1 is shown as the encoder input.

[0057] The rate R is the expected code length (bit rate) of the distortions of the latent spaces y and z, which are minimized during training with entropy models. The joint despeckling / distortion loss then corresponds to the sum over all pixels of the inverse of the lock likelihood of the marginal distributions (in logarithmic scale).

[0058] The inventors have demonstrated the success of the proposed methodology. Below, we describe some results of the neural network trained with various hyperparameters to explore the possibilities of an optimal trade-off between rate-distortion and classical compression metrics such as the peak signal-to-noise ratio (PSNR). In particular, the parameter λ serves as a trade-off parameter between the degree of distortion and compression, with lower values ​​favoring higher compression rates at the expense of a lower amount of detail and text information in the final reflectivity.

[0059] One of the challenges is the fact that there is no ground truth for noise-free reflectivities required for calculating a PSNR metric. To overcome this challenge and fully analyze the impact of despeckling on SAR image compression, a multi-temporal approach (as described in Reference

[15] ) was chosen to create a "noise-free" reflectivity image from 21 co-registered TerraSAR-X intensity images, which can serve as a ground truth reference for evaluating the distortion by the proposed joint despeckling and compression methods.

[0060] A resulting multi-temporal average image was obtained by processing acquisitions corresponding to ± 2 months with respect to the noisy acquisition, to avoid mixing different types of scattering mechanisms during the creation of the temporal image. The PSNR value measures the distortion (usually in decibels or dB) between the original image and the compressed image, with the higher the reconstruction, the higher the dB value. PSNR is useful for comparing different compression methods. It can be defined as follows: PSNR(A,A^)=10 log10max(A)MSE(A,A^) where A is the square root of the "noise-free" reflectivity and  is the square root of the estimated compressed reflectivity. MSE stands for the mean square error estimator, which measures the mean square error on a pixel-by-pixel basis. To allow comparison with the state of the art, two baselines were established: - MERLIN baseline (L1): A joint approach using the MERLIN approach to estimate reflectivities, followed by compression according to the JPEG2000 scheme. - Noisy Baseline (L2): A common approach without filtering the estimated (noisy) reflectivity, followed by compression with the JPEG2000 scheme. - SAR-BM3D

[16] : Although this filter does not use machine learning for the estimation of reflectivities, it was used because it provides excellent radiometric performance and can be used as a reference point for the estimated reflectivities.

[0061] The results of the rate-distortion performance based on the PSNR and the resulting bits-per-pixel (BPP) over the temporal image are shown in Fig.3. Based on these results, a value of the parameter λ of 5e-3 seems to be optimal, considering the resulting BPP of 0.3 and the increased PSNR value of about 1.5 dB compared to the MERLIN baseline (L1) and over 6 dB compared to the noisy baseline (L2). The poor performance of the L2 baseline can be clearly explained by using the “noise-free” reflectivity, obtained by averaging a large number of recordings, as a reference for calculating the PSNR. The results obtained with the proposed method (L3) and the comparison with the L2 baseline are also presented in Fig. 3 can be seen.

[0062] To detect distortions in the compressed reflectivities, several indicators can also be used: - Mean value of the image (MoI): should be preserved during filtering. - Ratio image: After filtering, the ratio image should contain only speckle. The presence of structures in the ratio image can be a sign of overfiltering. - Mean value of the ratio (MoR): The mean value of the ratio image should be as close to 1 as possible to indicate good radiometric preservation. - Variance of the Ratio (VoR): When the MoR value is close to 1, a VoR value less than 1 usually indicates undersmoothing and a value greater than 1 indicates oversmoothing.

[0063] Speckle reduction in homogeneous areas is usually measured in terms of the equivalent number of looks (ENL): ENL=μr2σr2 where µ r represents the mean value over a homogeneous area and σ ris the standard deviation calculated over the same area. These results are summarized in Table 1 and show the best performance for the proposed method.

[0064] The results for a homogeneous area can be found in Table 1 below: MoI MoR Before ENL Averaged image 0.178 1.12 1.11 10.31 Noisy image 0.209 - - 1 Our λ = 2.5e-3 0.178 1.04 0.96 288.26 Our λ = 5e-3 0.189 1.03 0.94 254.3 Our λ = 0.02 0.198 0.99 0.88 224.7 MERLIN 0.19 0.96 0.81 99.3 MERLIN + JPEG2000 0.17 0.97 0.85 406.03 SAR-BM3D 0.19 0.89 0.35 9.6

[0065] When corner reflectors are present in an image, two additional metrics can be used to verify the retention of the radiometric capabilities of the estimate by using corner reflectors, i.e., artificial radar reflectors commonly used for SAR calibration, in the image. These two metrics are the C nn , which is designed to measure the radiometric conservation of the strong scatterer, and the C bg, which attempts to measure the conservation of the ratio between the average intensity of the background compared to the strong scatterer, as specified in reference

[17] . These metrics can be formulated as follows: Cnn−10 log10rCFrNN Cbg−10 log10rCFrBG where r CF the intensity observed at the strong scatterer, r NN the average intensity of the 8 pixels surrounding the scatterer and r BG represents the average intensity of the background.

[0066] The results of this procedure are shown in Table 2, again with a proposed method that is closest to the original values ​​of the noisy image: C nn [dB] C bg [dB] Noisy image 26.75 42.54 Our λ = 2.5e-3 27.98 44.23 Our λ = 5e-3 28.11 44.47 Our λ = 0.02 27.29 43.29 MERLIN 25.69 35.9 MERLIN + JPEG2000 24.56 32.86 SAR-BM3D 25.63 41.6

[0067] The present invention offers a variety of advantages for various applications.

[0068] Onboard image data reduction enables efficient data transmission and reduces the load on ground stations. The main bottleneck in Earth observation systems is that, while the number of satellites and the amount of data are increasing, the scale of ground stations cannot keep up with the number of sensing satellites. This is especially true for SAR systems, which require a sun-synchronous orbit, typically a low Earth orbit (LEO), so ground stations are primarily located at the poles, the location with the shortest replay time. At the poles, space for building ground stations is limited and far from scalable with the number of satellites and nanosatellites. This trend will become even more pronounced in the future, requiring new solutions such as onboard processing and compression.

[0069] It also enables efficient storage of SAR images. SAR data currently requires a lot of storage space. Long SAR image archives are a critical problem for mission ground segments. New SAR systems are achieving ever greater coverage and shorter repetition times, resulting in very high data throughput. The Sentinel-1 mission, for example, produces between 600 and 800 GB of data per day, and future missions will produce even more.

[0070] Finally, the proposed method simplifies the processing of time series. SAR time series applications are the most memory-intensive applications today, and very often the amount of data to be processed is limited due to memory constraints. By compressing the images, the intermediate data do not need to be stored in memory in an unaltered form, but rather in a compressed format that can be decompressed when the image is needed.

[0071] Other applications include satellite and airborne SAR downlinks, time-limited automatic / human SAR image interpretation for land surveillance, target detection, etc., and large-scale processing such as global product generation. Reference symbol 10 SLC image 10Re real part of the SLC image 10 10In the imaginary part of the SLC image 15Re intensity image of the real part of the SLC image 10 15In the intensity image of the imaginary part of the SLC image 20 (combined) intensity image 100 channels 110 encoders 111 Down-sampling unit 112 Activation Function (GDN) 113 Residual block 114 Down-sampling unit 115 Activation Function (GDN) 116 Residual block 117 Down-sampling unit 118 Activation Function (GDN) 119 Residual block 120 down-sampling unit 130 linking unit 131 Quantizer (Q) 132 Arithmetic Encoder (AE) 140 Hyper Encoders 141 Absolute value 142 Down-sampling unit 143 Activation function (ReLU) 144 Down-sampling unit 145 Activation function (ReLU) 146 Down-sampling unit 151 Quantizer (Q) 152 Arithmetic Encoder (AE) MBS main bitstream SBS page bitstream 200 recipients 210 decoders 211 Up-sampling unit 212 Activation Function (IGDN) 213 Residual block 214 Up-sampling unit 215 Activation Function (IGDN) 216 Residual block 217 Up-sampling unit 218 Activation Function (IGDN) 219 Residual block 220 Up-sampling unit 240 Hyper Decoder 241 Up-sampling unit 242 Activation function (ReLU) 243 Up-sampling unit 244 Activation function (ReLU) 245 Up-sampling unit 250 Arithmetic Decoder (AD) 251 Arithmetic Decoder (AD) 252 splinters NN neural network y Latent dream y c linked latent dream Î Re, Î I,m Estimates ŷ Latent (latent dream) ẑ Latent (latent dream) Loss function Î estimated intensity (output) re 2 squared real part (input) In the 2 Imaginary part (input) D joint despeckling / distortion loss R Entropy of the latent ŷ and ẑ λ parameter (Lagrange multiplier) References [1] Dalsasso, E., Denis, L., & Tupin, F. (2021). SAR2SAR: A Semi-Supervised Despeckling Algorithm for SAR Images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14, 4321-4329. [2] Mazza, A., Scarpa, G., Verdoliva, L., & Poggi, G. (2021). Impact of Training Set Design in CNN-Based SAR Image Despeckling. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS (pp. 415-418). [3] Dalsasso, E., Denis, L., & Tupin, F. (2022). As If by Magic: Self-Supervised Training of Deep Despeckling Networks With MERLIN. IEEE Transactions on Geoscience and Remote Sensing, 60, 1-13. [4] Dalsasso, E., Denis, L., Muzeau, M., & Tupin, F. (2022). Self-supervised training strategies for SAR image despeckling with deep neural networks. In 14th European Conference on Synthetic Aperture Radar (EUSAR). [5] Ballé, J., Laparra, V., & Simoncelli, E. (2017). End-to-end Optimized Image Compression. arXiv:1611.01704. [6] Ballé, J., Minnen, D., Singh, S., Hwang, S., & Johnston, N. (2018). Variational image compression with a scale hyperprior. arXiv:1802.01436. [7] Xu, Q., Xiang, Y., Di, Z., Fan, Y., Feng, Q., Wu, Q., & Shi, J. (2022). Synthetic Aperture Radar Image Compression Based on a Variational Autoencoder. IEEE Geoscience and Remote Sensing Letters, 19, 1-5. [8] Di, Z., Chen, X., Wu, Q., Shi, J., Feng, Q., & Fan, Y. (2022). Learned Compression Framework With Pyramidal Features and Quality Enhancement for SAR Images. IEEE Geoscience and Remote Sensing Letters, 19, 1-5. [9] Cheng, K., Xie, Y., & Chen, Q. (2022). Optimizing image compression via joint learning with denoising. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIX (pp. 56-73).

[10] Oliveira, V., Chabert, M., Oberlin, T., Poulliat, C., Bruno, M., Latry, C., Carlavan, M., Henrot, S., Falzon, F., & Camarero, R. (2022). Satellite Image Compression and Denoising With Neural Networks. IEEE Geoscience and Remote Sensing Letters, 19, 1-5.

[11] Zhaohui Zeng, & Cumming, I. (2001). SAR image data compression using a tree-structured wavelet transform. IEEE Transactions on Geoscience and Remote Sensing, 39(3), 546-552.

[12] Mercier, G. (2001). Despeckle-based SAR image compression. In IGARSS 2001. Scanning the Present and Resolving the Future. Proceedings. IEEE 2001 International Geoscience and Remote Sensing Symposium (Cat. No.01CH37217) (pp. 100-102 vol. 1).

[13] Xu, Q., Xiang, Y., Di, Z., Fan, Y., Feng, Q., Wu, Q., & Shi, J. (2022). Synthetic Aperture Radar Image Compression Based on a Variational Autoencoder. IEEE Geoscience and Remote Sensing Letters, 19, 1-5.

[14] Di, Z., Chen, X., Wu, Q., Shi, J., Feng, Q., & Fan, Y. (2022). Learned Compression Framework With Pyramidal Features and Quality Enhancement for SAR Images. IEEE Geoscience and Remote Sensing Letters, 19, 1-5.

[15] Vitale, S., Ferraioli, G., & Pascazio, V. (2021). Analysis on the building of training dataset for deep learning SAR despeckling. IEEE Geoscience and Remote Sensing Letters, 19, 1-5.

[16] Parrilli, S., Poderico, M., Angelino, C., & Verdoliva, L. (2012). A Nonlocal SAR Image Denoising Algorithm Based on LLMMSE Wavelet Shrinkage. IEEE Transactions on Geoscience and Remote Sensing, 50(2), 606-616.

[17] Di Martino, G., Poderico, M., Poggi, G., Riccio, D., & Verdoliva, L. (2014). Benchmarking Framework for SAR Despeckling. IEEE Transactions on Geoscience and Remote Sensing, 52(3), 1596-1615. ZITATE ENTHALTEN IN DER BESCHREIBUNG

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Dalsasso, E., Denis, L., & Tupin, F. (2021). SAR2SAR: A Semi-Supervised Despeckling Algorithm for SAR Images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14, 4321-4329

[0071] Mazza, A., Scarpa, G., Verdoliva, L., & Poggi, G. (2021). Impact of Training Set Design in CNN-Based SAR Image Despeckling. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS (pp. 415-418

[0071] Dalsasso, E., Denis, L., & Tupin, F. (2022). As If by Magic: Self-Supervised Training of Deep Despeckling Networks With MERLIN. IEEE Transactions on Geoscience and Remote Sensing, 60, 1-13

[0071] Dalsasso, E., Denis, L., Muzeau, M., & Tupin, F. (2022). Self-supervised training strategies for SAR image despeckling with deep neural networks. In 14th European Conference on Synthetic Aperture Radar (EUSAR

[0071] Ballé, J., Laparra, V., & Simoncelli, E. (2017). End-to-end Optimized Image Compression. arXiv:1611.01704

[0071] Ballé, J., Minnen, D., Singh, S., Hwang, S., & Johnston, N. (2018). Variational image compression with a scale hyperprior. arXiv:1802.01436

[0071] Xu, Q., Xiang, Y., Di, Z., Fan, Y., Feng, Q., Wu, Q., & Shi, J. (2022). Synthetic Aperture Radar Image Compression Based on a Variational Autoencoder. IEEE Geoscience and Remote Sensing Letters, 19, 1-5

[0071] Di, Z., Chen, X., Wu, Q., Shi, J., Feng, Q., & Fan, Y. (2022). Learned Compression Framework With Pyramidal Features and Quality Enhancement for SAR Images. IEEE Geoscience and Remote Sensing Letters, 19, 1-5

[0071] Cheng, K., Xie, Y., & Chen, Q. (2022). Optimizing image compression via joint learning with denoising. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIX (pp. 56-73

[0071] Oliveira, V., Chabert, M., Oberlin, T., Poulliat, C., Bruno, M., Latry, C., Carlavan, M., Henrot, S., Falzon, F., & Camarero, R. (2022). Satellite Image Compression and Denoising With Neural Networks. IEEE Geoscience and Remote Sensing Letters, 19, 1-5

[0071] Zhaohui Zeng, & Cumming, I. (2001). SAR image data compression using a tree-structured wavelet transform. IEEE Transactions on Geoscience and Remote Sensing, 39(3), 546-552

[0071] Mercier, G. (2001). Despeckle-based SAR image compression. In IGARSS 2001. Scanning the Present and Resolving the Future. Proceedings. IEEE 2001 International Geoscience and Remote Sensing Symposium (Cat. No.01CH37217) (pp. 100-102 vol. 1

[0071] Vitale, S., Ferraioli, G., & Pascazio, V. (2021). Analysis on the building of training dataset for deep learning SAR despeckling. IEEE Geoscience and Remote Sensing Letters, 19, 1-5

[0071] Parrilli, S., Poderico, M., Angelino, C., & Verdoliva, L. (2012). A Nonlocal SAR Image Denoising Algorithm Based on LLMMSE Wavelet Shrinkage. IEEE Transactions on Geoscience and Remote Sensing, 50(2), 606-616

[0071] Di Martino, G., Poderico, M., Poggi, G., Riccio, D., & Verdoliva, L. (2014). Benchmarking Framework for SAR Despeckling. IEEE Transactions on Geoscience and Remote Sensing, 52(3), 1596-1615

[0071]

Claims

[1] A method for computer-aided generation of a data-based model for computer-aided processing of a single-look complex (SLC) image (10) acquired with a SAR system, wherein the SAR system is configured to obtain radar reflection, wherein the SLC image (10) comprises a real part and an imaginary part, and wherein the data-based model is configured to jointly remove speckle and compress the SLC image (10), wherein the generation of the data-based model comprises the following steps: - providing a training data set (TDS) comprising a plurality of SLC images (10), each of which comprises a real part and an imaginary part; - training a neural network (NN) with the training data set (TDS), wherein the neural network (NN) comprises an input for inputting either the real part or the imaginary part of a respective SLC image (10) of the plurality of SLC images (10) as input information, wherein a latent space (y) resulting from the encoding of the input information with an encoder (100) is concatenated and compressed into a compressed bit stream (MBS, SBS), and an output for outputting an estimated intensity (I), wherein the estimated intensity (Î) results from the splitting and decoding of the compressed bit stream (MBS, SBS), wherein the trained neural network (NN) represents the generated data-based model. [2] Method according to claim 1, wherein a total reflectivity of pixels in the SLC image (10), in particular above a predetermined threshold corresponding to point scatterers, is obtained and the total reflectivity is halved by assigning it to its imaginary or real part. [3] The method of claim 1 or 2, wherein the trained neural network (NN) is based on a variational autoencoder with hyperpriority to process compression and despeckling simultaneously. [4] Method according to one of the preceding claims, in which the concatenated latent space (y c) is transmitted to a first quantizer (131) and a first arithmetic encoder (132) to output a speckle-removed and compressed main bitstream (MBS), and to a hyper-encoder (140), the output of which is transmitted to a second quantizer (151) and a second arithmetic encoder (152) to output a speckle-removed and compressed side bitstream (SBS). [5] A method according to claim 4, wherein the (arithmetically decoded) side bit stream (SBS) is transmitted to a hyper decoder (240) to estimate scales (σ̂) which are encoded in the main bit stream (MBS) for further compression. [6] Method according to one of the preceding claims, wherein the squared real part or the squared imaginary part of a respective SLC image (10) is input into the neural network (NN). [7] Method according to one of the preceding claims, wherein the training of the neural network (NN) comprises the calculation of a loss function (L) which takes into account both compression and reflectivity estimation, with the possibility of emphasizing one or the other by including a parameter (λ). [8] The method of claim 7, wherein the loss corresponds to a relaxed rate distortion optimization problem according to: L(I^,Im)=R+λD(I^k,Imk) where (L) the loss function, Î the estimated intensity (output) for the neural network, Re 2 or in 2 the squared real or imaginary part (input), D the joint despeckling / distortion loss, R the entropy of the latents ŷ and ẑ, and λ, a Lagrange multiplier. [9] Method according to claim 7 or 8, wherein the loss function (L) is used to train weights of the neural network (NN). [10] A method for computer-aided processing of a single-look complex (SLC) image acquired with a SAR system configured to obtain radar reflectance, wherein the SLC image (10) comprises a real part and an imaginary part, both of which are input to a data-driven model, and wherein an estimated intensity image (Î) of the SLC image (10) is generated with a data-driven model generated by a method according to any one of claims 1 to 9. [11] A method according to claim 10, wherein both the real part and the imaginary part of the SLC image (10) are encoded and concatenated. [12] Method according to claim 11, wherein the real part and the imaginary part of the SLC image (10) are encoded with the same encoder. [13] Method according to one of claims 10 to 12, wherein on the decoder side the concatenated latent space (y c ) into two latent dreams (ŷ) and then into their respective estimates (Î Re , Î Im ) is decoded, whereby the estimated intensity image (I) after adding the two estimates (Î Re , Î Im ) is obtained. [14] Apparatus for computer-aided processing of a single-look complex (SLC) image acquired with a SAR system configured to obtain radar reflectance, the SLC image (10) comprising a real part and an imaginary part, both of which are input to a data-based model, and generating an estimated intensity image (1) of the SLC image (10), the apparatus comprising a data-based model generated by a method according to any one of claims 1 to 9. [15] A computer program product comprising program code stored on a machine-readable medium for carrying out a method according to any one of claims 1 to 9 when the program code is executed on a computer.

Images