Remote sensing image defogging method and system based on improved transformer model

Abstract

The application discloses a remote sensing image defogging method and system based on an improved Transform model, and relates to the technical field of image processing. The method comprises the following steps: preparing a remote sensing image dataset composed of a plurality of image pairs, and dividing a training set from the dataset; the image pair is composed of a haze-free remote sensing image and a haze remote sensing image corresponding to the haze-free remote sensing image; an existing feature extraction module in a Transform model is improved, and the improved feature extraction module is named a defogging module; the improvement comprises introducing a fast Fourier convolution (FFC) module; the improved Transform model is trained by using the training set, and an FFCFormer defogging model is obtained; and the FFCFormer defogging model is used to perform defogging processing on a haze remote sensing image input into the model. The system is used for transplanting and deploying the improved Transform model to an embedded computing device (NVIDIA Jetson Xavier NX), so that fast, real-time and visual remote sensing image defogging is realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method and system for dehazing remote sensing images based on an improved Transformer model. Background Technology

[0002] Remote sensing images are often affected by atmospheric scattering effects in practical applications. Dense clouds and heavy fog can cause problems such as blurred edges and color distortion, leading to decreased image quality, slow processing speeds, and hindering information acquisition and analysis. Remote sensing image dehazing is a challenging task because atmospheric phenomena such as fog and haze cause light scattering and absorption, resulting in blurred targets in remote sensing images. Removing these effects requires accurate scattering models and dehazing algorithms, which place high demands on computational and physical models. In practical applications, deep learning model-based dehazing methods have attracted widespread attention due to their excellent performance and generalization ability. The Transformer model, with its powerful feature extraction and modeling capabilities, has achieved significant results in fields such as natural language processing and has recently been applied to image processing tasks. For example, Transformer-based remote sensing image dehazing methods can handle daytime, nighttime, and remote sensing image dehazing tasks.

[0003] The main advantage of the Transformer model lies in its self-attention mechanism, which captures the dependencies between any two elements in a sequence, thus effectively handling global information and long-range dependencies in image dehazing. However, the Transformer model may face the problem of local information loss when processing hazy remote sensing images. Furthermore, processing large-scale remote sensing image data using the Transformer model typically requires significant computational resources and memory, resulting in high computational cost and slow processing speed. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method and system for dehazing remote sensing images based on an improved Transformer model.

[0005] The technical solution of this invention is as follows:

[0006] The first aspect of this invention provides a method for dehazing remote sensing images based on an improved Transformer model, the method comprising the following steps:

[0007] Step 1: Prepare a remote sensing image dataset consisting of multiple image pairs, and divide the dataset into a training set; the image pairs consist of fog-free remote sensing images and foggy remote sensing images corresponding to the fog-free remote sensing images;

[0008] Step 2: Improve the feature extraction module in the existing Transformer model and name the improved feature extraction module the dehazing module. The improvement includes introducing a Fast Fourier Convolution (FFC) module.

[0009] Step 3: Train the improved Transformer model using the training set to obtain the FFCFormer dehazing model;

[0010] Step 4: Use the FFCFormer dehazing model to dehaze the foggy remote sensing image input to the model.

[0011] The introduction of the Fast Fourier Convolution (FFC) module specifically involves introducing the FFC module before the scale normalization operation of the original feature extraction module.

[0012] The improvement of the feature extraction module in the existing Transformer model described in step 2 also includes replacing the reflection-filled shift window partitioning scheme with a copy-filled shift window partitioning scheme.

[0013] In step 4, the FFCFormer dehazing model is deployed on an embedded computing device to dehaze the input foggy remote sensing image.

[0014] Step 4 specifically includes the following steps:

[0015] Step 4.1: Save the FFCFormer dehazing model as a .pth file and name it best.pth;

[0016] Step 4.2: Prune and quantize best.pth, and name the simplified model dehazeformer.pth;

[0017] Step 4.3: Convert dehazeformer.pth to ONNX format and name it dehazeformer.onnx;

[0018] Step 4.4: Use the simplification tool onnxsim to perform graph transformation and constant folding operations on dehazeformer.onnx to eliminate redundant parameters and calculations in the model, and obtain the lightweight model dehazeformer-sim.onnx;

[0019] Step 4.5: Use the deep learning inference optimizer TensorRT to convert dehazeformer-sim.onnx into a callable engine file, dehaze.engine;

[0020] Step 4.6: Call the deep learning inference optimizer TensorRT on the embedded computing device. TensorRT calls the engine file dehaze.engine, which is then compiled to generate an executable dehazing program.

[0021] Step 4.7: Read the input hazy remote sensing image in the executable dehazing program to perform dehazing processing on the hazy remote sensing image.

[0022] The embedded computing device is an NVIDIA Jetson Xavier NX.

[0023] A second aspect of the present invention provides a remote sensing image dehazing system based on an improved Transformer model, established based on the method described above. The system is deployed on an embedded computing device, NVIDIA Jetson Xavier NX, and includes:

[0024] The image storage module is responsible for saving the collected foggy remote sensing images that need to be dehazed to the system album of the NVIDIA Jetson Xavier NX;

[0025] The image preprocessing module is responsible for selecting one or more foggy remote sensing images from the system album using the image selector provided by the NVIDIA Jetson XavierNX and preprocessing them, including scaling and normalization.

[0026] The image inference module is responsible for calling the deep learning inference optimizer TensorRT. TensorRT calls the engine file dehaze.engine, which is compiled to generate an executable dehazing program. The executable dehazing program reads the image processed by the image preprocessing module and performs the inference process to achieve image dehazing and obtain a dehazed remote sensing image.

[0027] The results processing and display module performs inverse normalization on the dehazed remote sensing images obtained by the image inference module, and then visualizes the processed dehazed remote sensing images on NVIDIA Jetson Xavier NX.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] 1. Remote sensing images are often affected by atmospheric conditions such as haze and clouds during the acquisition process, resulting in decreased image quality and blurred edges, which affects subsequent ground feature identification and analysis. To address this, this invention features an innovative design for preserving edge pixels. It replaces the existing reflection-filled shift window partitioning scheme in the feature extraction module of the Transformer model with a copy-filled shift window partitioning scheme. By copying the edge pixels of the image to the filling area, the edge information of the image can be preserved, resulting in high-quality dehazed remote sensing images.

[0030] 2. A Fast Fourier Convolution (FFC) module is introduced into the feature extraction module of the existing Transformer model. The FFC module has higher computational efficiency and better global perception capability. In image dehazing, the FFC module can help the Transformer model better understand the global structure and periodic features of the image, and can more effectively capture the global structure and periodic features of the image, reducing the loss of global information in the dehazing process. The dehazed image has improved clarity and color consistency.

[0031] 3. The improved Transformer model is optimized. First, pruning and quantization operations are performed to convert the improved Transformer model into ONNX format. Then, the simplification tool onnxsim is used to eliminate redundant parameters and calculations in the model, which greatly reduces the number of parameters and computational complexity of the model.

[0032] 4. The system of this invention deploys the optimized and improved Transformer model on an embedded computing device, NVIDIA Jetson Xavier NX. This allows for real-time model inference upon input of foggy remote sensing images, significantly reducing computational costs and improving system response speed. This is of great significance for real-time or near-real-time dehazing of remote sensing images. Attached Figure Description

[0033] Figure 1 This is a flowchart of the remote sensing image dehazing method based on the improved Transformer model in this embodiment;

[0034] Figure 2 This is a structural diagram of the Transformer model in this implementation method;

[0035] Figure 3 The diagrams show a comparison of the structure of the feature extraction module in the Transformer model before and after the improvement in this embodiment. (a) is the structure diagram of the feature extraction module before the improvement; (b) is the structure diagram of the feature extraction module after the improvement.

[0036] Figure 4 This is a flowchart of step 4 of this embodiment;

[0037] Figure 5 This is a block diagram of the remote sensing image dehazing system based on the improved Transformer model in this embodiment;

[0038] Figure 6 A comparison chart of the dehazing results of different image dehazing methods. Detailed Implementation

[0039] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0040] This invention provides a remote sensing image dehazing scheme based on an improved Transformer model. The core idea is as follows: Addressing the issue of global information loss in existing Transformer-based remote sensing image dehazing methods, this invention introduces an FFC module into the Transformer model to extract and fuse features from both local and global perspectives. To address the edge blurring problem in existing Transformer-based remote sensing image dehazing methods, a shift window partitioning scheme based on copy-fill is used instead of reflection-fill, selectively preserving edge pixels. The fused features undergo scale normalization and copy-fill, and spatial attention weights are calculated and dynamically aggregated to improve the quality and fidelity of the dehazed image. Furthermore, to address the issues of large computational parameters and long computation time in existing Transformer-based remote sensing image dehazing methods, this invention further optimizes the improved Transformer model by performing pruning, quantization, and redundant parameter elimination to reduce the number of parameters and computational complexity, thereby improving computational speed. This invention also provides a remote sensing image dehazing system based on an improved Transformer model. The optimized improved Transformer model is ported and deployed on an embedded computing device, NVIDIA Jetson Xavier NX, to achieve fast, real-time, and visualized remote sensing image dehazing.

[0041] The overall flowchart of the remote sensing image dehazing method based on the improved Transformer model in this implementation is as follows: Figure 1 As shown, it includes the following steps:

[0042] Step 1: Prepare a remote sensing image dataset consisting of multiple image pairs, and divide the dataset into a training set; the image pairs consist of fog-free remote sensing images and foggy remote sensing images corresponding to the fog-free remote sensing images;

[0043] The publicly available RS-Haze dataset contains 6,000 fog-free remote sensing images with various terrain features and 1,500 cirrus cloud channels with distributions similar to natural haze. In this embodiment, nine synthetic fog images are artificially generated for each fog-free remote sensing image in the RS-Haze dataset using cirrus cloud channels, including 18,000 fog images each with three different fog densities: light fog, medium fog, and thick fog. This embodiment selects 18,000 fog-free remote sensing images and their corresponding 18,000 thick fog images to construct the remote sensing image dataset of this embodiment. Each fog-free remote sensing image and its corresponding thick fog image constitute one image pair. Therefore, the remote sensing image dataset of this embodiment has a total of 18,000 image pairs. From this embodiment, 14,400 pairs are used for pre-training training, and 1,800 pairs are used for testing.

[0044] Step 2: Improve the feature extraction module in the existing Transformer model and name the improved feature extraction module the dehazing module. The improvement includes: introducing a Fast Fourier Convolution (FFC) module and using a shift window partitioning scheme based on copy padding instead of a shift window partitioning scheme based on reflection padding.

[0045] This implementation improves the feature extraction module in the existing Transformer model by adding a Fast Fourier Convolution (FFC) module before the scale normalization operation in the original feature extraction module to compensate for the loss of global information in the Transformer model, which leads to color distortion and pixel blurring in the dehazing result. After the scale normalization operation in the feature extraction module, a shift window partitioning scheme based on copy filling is used instead of the reflection filling scheme to selectively preserve edge pixels, thereby improving the quality and fidelity of the dehazed image. The improved feature extraction module is named the dehazing module, and the overall structure of the improved Transformer model is the same as described above. Figure 2 The overall structure of the Transformer model before improvement is shown.

[0046] The dehazing module processes the input image through the following steps:

[0047] Step 2.1: The image obtained after performing a 3×3 convolution on the foggy remote sensing image I. The input to the FFC module is for I. r Frequency domain features are extracted from both the global and local branches, and then fused to obtain a fused feature map containing frequency domain contextual relationships.

[0048] like Figure 2 and Figure 3 As shown, the image obtained after processing the foggy remote sensing image I, which is input into the Transformer model, through a 3×3 convolution is... Enter the FFC module, where H represents height, W represents width, and C represents the number of channels. Figure 3 The diagram shows a comparison of the feature extraction module before and after the improvement. Figure (a) is the original, unimproved feature extraction module structure diagram, and Figure (b) is the improved dehazing module structure diagram of this embodiment. The dashed box contains the Fast Fourier Convolution module, i.e., the FFC module.

[0049] The FFC module consists of a global branch and local branches. The global branch has a global receptive field and is responsible for transforming the spatial features of the input data to the frequency domain and extracting the spectral data, and then transforming it back to the spatial format. The local branch is specifically used to extract local information from the global information. In this embodiment, the specific operation of the Fast Fourier Convolution branch is to first transform the image I along the dimension of the feature channel. r Data I divided into local branches r l and data sent to the global branch Right now Data fed into the local branch The data responsible for learning from the local neighborhood and feeding it into the global branch. Aimed at capturing remote context, where α in ∈[0,1] represents the percentage of feature channels allocated to the global branch, and in this embodiment, α in The value is set to 0.25. The local branch contains two parallel 3×3 convolutional layers, referred to as the local-to-local branches Y. l→l and the local-to-global branch Y l→g The result of the data fed into the local branch after passing through two parallel convolutional layers is denoted as Y. l The global branch contains a signal transformer, i.e., the global-to-global branch Y. g→g A parallel 3×3 convolutional layer, i.e., the global-to-local branch Y. g→l The result of the data fed into the global branch passing through the parallel signal generator and the convolutional layer is denoted as Y. g .

[0050] The update process of the Fast Fourier Convolution branch is represented as follows:

[0051]

[0052] Where Y l Y g→l f g f l→g These represent operations on the input data of different branches, f l f g→lf l→g For a normal convolution operation, f g This is a signal transformation operation.

[0053] In a signal converter, the data I sent to the global branch is first... r g As input, a 1×1 convolution-ReLU is used to reduce the number of data channels. Then, a Fast Fourier Transform (FFT)-1×1-ReLU-Inverse Fast Fourier Transform (IFFT) operation is performed sequentially. For real tensors, the FFT produces conjugate-symmetric results. Considering data availability, only half of the conjugate-symmetric result is retained, and the other half is reconstructed using a simple conjugate method. The real and imaginary parts of these results are concatenated along the channel dimension, and then processed through 1×1 convolution, normalization, and ReLU activation. Finally, the result is decomposed into real and imaginary parts and converted to an inverse number format. This method leverages the FFT's ability to process information in the frequency domain and capture local features within global information through local operations. Channel fusion is performed in the final stage of processing, using another 1×1 convolution operation to restore the number of channels, reconstructing the channel dimension of the features, and obtaining a fused feature map containing frequency domain contextual relationships.

[0054] Step 2.2: Perform shift window partitioning and dynamic spatial attention aggregation based on copy-filling;

[0055] After processing by the FFC module, the fused feature map output by the FFC module is... Upon entering the dehazing module, the model first undergoes scale normalization, followed by shift window partitioning based on copy-fill. Existing Transformer models often employ shift window partitioning based on reflection fill, which can sometimes introduce unnatural reflections at edges, potentially negatively impacting the dehazing task. Copy-fill, however, avoids generating overly complex boundary effects. This method offers high consistency in edge processing; since edge pixels are copied, it avoids artifacts or false information that might result from reflection fill. The specific operation of shift window partitioning based on copy-fill is as follows: based on the scale-normalized fused feature map... Size: In this implementation, a fixed-size window of 3x3 pixels is selected to scan the feature map to be fused. from Starting from one corner, slide the window line by line to cover the entire area. When the window is merging feature maps When swiping up, especially when the window is close to When the window is at the edge or corner, a portion of the window may extend beyond the fused feature map. The actual size of the window will result in some blank areas. These blank areas contain no actual pixel data, so a padding technique is needed to handle them. Specifically, this involves padding the area within the window... Pixel data near the boundary is copied and used to fill the blank areas within the window. The resulting feature map after copying and filling is denoted as...

[0056] Next, we will look at the filled feature image. Flattened into a one-dimensional vector, and further flattened into a one-dimensional sequence according to spatial dimensions H×W, where each sequence is a C-dimensional feature vector. A linear layer projects the flattened feature vectors onto multi-head attention factors Q, K, and V. The global and local information obtained from the projection are sampled and enhanced through spatial information aggregation. The spatial information aggregation weights are dynamic, but always positive, thus requiring an additional convolution on V. In the multi-head attention module, Q and K are channel-fused, and V undergoes a 1×1 convolution operation. The specific scheme for spatial information aggregation is as follows:

[0057]

[0058] Where d is the dimension of the vector, and S is the relative position bias term. V represents the channel before partitioning, and Conv(·) represents a 1×1 convolution. The output obtained by aggregating spatial information is fed into a linear layer to obtain the linear output J. liner The linear output J liner After channel clipping, the data is fed into the affine layer. J is then linearly output in the affine layer. liner and the filled feature image Linear weighted fusion is performed to obtain two feature maps, denoted as p1 and p2. These feature maps p1 and p2 are then fed into a multilayer perceptron (MLP). The MLP consists of a linear layer-ReLU-linear layer residual structure. The first linear layer f(·) is used to project p1 onto... The fusion weights are obtained using global average pooling (GAP), the softmax function, and the split operation:

[0059]

[0060] The fusion result q is obtained by using the fusion weights {a1, a2}: That is, guided by the fusion result q, the improved Transformer model can complete the dynamic aggregation of spatial attention and achieve higher quality dehazing results in the dehazing inference process.

[0061] Step 3: Train the improved Transformer model using the training set to obtain the FFCFormer dehazing model;

[0062] In the dehazing inference process of the Transformer model, current image dehazing networks generally predict fog-free reconstructed images. Since the degradation model is approximate, it is beneficial to introduce priors in the absence of strong constraints. Therefore, the defogging inference process can be derived from:

[0063]

[0064] Rewritten as:

[0065]

[0066] Where K(x) = 1 / t(x)⁻¹, B(x) = -(1 / t(x)⁻¹)A, where I is the captured hazy fog image, A is the global atmospheric light, and t is the medium transmission map. The transfer quantity can be expressed as: t(x) = e -βd(x) In the formula, β is the atmospheric scattering coefficient and d is the depth of field.

[0067] The dehazing task of the Transformer model is to predict scattering. Scattering O can be further decomposed into and Therefore, using the fusion result q from step 2.2, the Transformer model applies soft constraints to the relationship between K(x) and B(x), namely:

[0068] O(x)=qK(x)+(1-q)B(x) (6)

[0069] Therefore, by deriving the scattering O(x), the Transformer model can recover the haze-free reconstructed image. This system implements a remote sensing image dehazing function. The overall network is a 5-layer UNet structure, with downsampling and upsampling representing the downsampling and upsampling layers, respectively. Two downsampling and two upsampling operations are performed after each dehazing module. Finally, a ReLU layer and a 3×3 convolutional layer are used to reconstruct a haze-free image with the same size as the input image.

[0070] This implementation trains the improved Transformer model on the PyTorch platform using an NVIDIA V100 graphics card. Before training, all images in the training set are resized to 400×400. The improved Transformer model is trained using L1 loss for 150 epochs, and the learning rate is adjusted according to the model's minimum batch size. L1 loss is used to measure the mean absolute error between the model's predictions and the true labels. Specifically, for a sample set of size N, the L1 loss is defined as follows:

[0071]

[0072] Among them, l i This represents the true label of the sample. This represents the predicted label of the sample by the improved Transformer model. The L1 loss is obtained by averaging the absolute errors of each sample. The training process uses an Adam optimizer with a batch size of 8 and a patch size of 128×128 for 480K iterations, where the decay rate β1 for the first moment estimation is 0.9 and the decay rate β2 for the second moment estimation is 0.999. The initial learning rate is set to 2×10⁻⁶. -4 And a cosine annealing decay strategy is used as the training strategy to gradually reduce the learning rate to 10. -6 In this implementation, the optimal model obtained during training is referred to as the FFCFormer dehazing model.

[0073] Step 4: Use the FFCFormer dehazing model to dehaze the input foggy remote sensing image.

[0074] Furthermore, in practical applications, the Transformer model typically requires significant computational resources and memory to process large-scale remote sensing image data, resulting in high computational costs and slow processing speeds. To address this issue, in step 4 of this invention, the FFCFormer dehazing model obtained in step 3 is first optimized. Then, the optimized FFCFormer dehazing model is deployed on an embedded computing device, the NVIDIA Jetson Xavier NX, where the input foggy remote sensing image is dehazed. Figure 4 Step 4 shown specifically includes the following steps:

[0075] Step 4.1: Save the FFCFormer dehazing model as a .pth file and name it best.pth;

[0076] Step 4.2: Prune and quantize best.pth, and name the simplified model dehazeformer.pth;

[0077] Step 4.3: Convert dehazeformer.pth to ONNX format and name it dehazeformer.onnx;

[0078] This implementation uses the torch.onnx conversion tool provided by PyTorch to convert dehazeformer.pth into an ONNX (Open Neural Network Exchange) format file and name it dehazeformer.onnx. ONNX has cross-platform capabilities; therefore, converting the model to ONNX format enables cross-platform deployment.

[0079] Step 4.4: Use the simplification tool onnxsim to perform graph transformation and constant folding operations on dehazeformer.onnx to eliminate redundant parameters and calculations in the model, and obtain the lightweight model dehazeformer-sim.onnx.

[0080] In some cases, models converted to ONNX format may contain a large number of redundant parameters and complex calculations, making them difficult to understand and optimize. Therefore, this implementation uses the simplification tool onnxsim to perform a series of graph transformations and constant folding operations on the ONNX format model to eliminate redundant parameters and calculations, resulting in a lightweight model named dehazeformer-sim.onnx. These operations can significantly reduce the complexity and size of the model while maintaining its performance and accuracy.

[0081] Step 4.5: Use the deep learning inference optimizer to convert dehazeformer-sim.onnx into a callable engine file, dehaze.engine;

[0082] This implementation uses the high-performance deep learning inference optimizer TensorRT to perform low-latency, high-throughput model conversion on the lightweight model dehazeformer-sim.onnx, generating the C++ callable engine file dehaze.engine, further improving the model's running speed on the device.

[0083] Step 4.6: Call the deep learning inference optimizer TensorRT on the embedded computing device, call dehaze.engine through the deep learning inference optimizer TensorRT, and generate an executable dehazing program after compilation;

[0084] The embedded computing device used in this embodiment is an NVIDIA Jetson Xavier NX.

[0085] Step 4.7: Read the input hazy remote sensing image in the executable dehazing program to perform dehazing processing on the hazy remote sensing image.

[0086] The executable dehazing program reads foggy remote sensing images from the test set as input to the FFCFormer dehazing model. These foggy images are then fed into the FFCFormer model for dehazing inference. During the image dehazing inference process, this implementation utilizes the GPU acceleration technology of the embedded computing device NVIDIA Jetson Xavier NX to accelerate the inference of the FFCFormer dehazing model, thus achieving dehazing processing of foggy remote sensing images. The NVIDIA Jetson Xavier NX, with its 384-core NVIDIA Volta GPU featuring 48 Tensor Cores and a 6-core NVIDIA Carmel ARM v8.2 64-bit CPU, provides powerful computing capabilities, significantly reducing inference time and contributing to high-precision, high-speed dehazing of remote sensing images.

[0087] The dehazing experimental results on the RS-Haze test set are shown in Table 1. Remote sensing images have more monotonous scenes than natural scenes, making it easier for this invention to estimate the latent color and brightness of the image, thereby improving the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). PSNR represents the ratio of the maximum possible power of the signal to the power of destructive noise affecting its representation accuracy; a higher ratio indicates higher image quality. Structural Similarity measures the similarity and realism of the image before and after distortion; a higher value indicates higher similarity between the images before and after dehazing. The uneven haze concentration in the RS-Haze dataset destroys high-frequency information in the image. The FFC module can capture global information and long-distance dependencies, avoiding the loss of global information in the results. This allows the FFCFormer dehazing model to better capture the global structure and periodic features of the image, reducing the loss of global information during the dehazing process and improving the clarity and color consistency of the dehazed image, thus verifying the effectiveness of the method proposed in this invention. This invention also achieves significant advantages in terms of parameter count (#Param) and cumulative multiply-accumulate operations (MACs). Parameter count refers to the total number of trainable parameters in a deep learning model. These parameters include weights and biases, typically present in each layer of the network. A higher parameter count leads to greater storage requirements, longer training time, and higher computational costs for training and inference. The cumulative multiply-accumulate operations refer to the total number of multiplication and addition operations performed during a single forward propagation or inference. A higher number of MACs typically requires more computational resources and time to complete a single forward propagation. The FFCFormer dehazing model achieves a lower parameter count while simultaneously having fewer cumulative multiply-accumulate operations, demonstrating faster network convergence and inference speeds. When processing large-scale remote sensing image data, it can significantly save computational resources and memory.

[0088] Table 1. Comparison of objective evaluation, parameter count, and multiply-accumulate operations of different defogging methods on the RS-Haze dataset.

[0089] DCP 17.86 0.734 - - AOD-Net 24.90 0.830 0.002M 0.115G MSBDN 38.57 0.965 31.35M 41.54G FFA-Net 39.39 0.969 4.456M 287.8G AECR-Net 35.69 0.959 2.611M 52.50G Method of the present invention 39.95 0.975 2.965M 29.36G

[0090] This invention also provides a remote sensing image dehazing system based on an improved Transformer model, the structural block diagram of which is shown below. Figure 5 As shown, the FFCFormer dehazing model is deployed on the embedded computing device NVIDIA Jetson XavierNX to achieve a low-latency, high-throughput dehazing inference process and real-time visualization of dehazing results. The system includes: an image storage module, an image preprocessing module, an image inference module, and a result processing and display module.

[0091] The image storage module is responsible for storing the collected foggy remote sensing images that require dehazing. These images can be synthetic remote sensing fog images similar to the RS-Haze dataset, or remote sensing images of real foggy scenes captured by a mobile device's camera. The collected synthetic fog images or captured foggy remote sensing images are saved to the system album of the NVIDIA Jetson Xavier NX device. When the number of foggy remote sensing images to be processed is large, the collected images are numbered and stored, including assigning unique identifiers and writing index fields to speed up the retrieval process, and the numbered foggy remote sensing images are then sent to the image storage module.

[0092] The image preprocessing module is responsible for selecting saved hazy remote sensing images from the system's image album using the device's built-in image selector and performing preprocessing operations on them to ensure that the selected hazy remote sensing images meet the input requirements of the FFCFormer dehazing model. During preprocessing, the image is first scaled to adjust its size to be compatible with the input layer of the FFCFormer dehazing model. This scaling ensures that the image has a uniform size before being input into the FFCFormer dehazing model, thereby improving its stability and efficiency. Secondly, the image is normalized, adjusting the pixel value range to between 0 and 1, resulting in a more uniform pixel value distribution. Normalization not only facilitates accurate feature extraction by the FFCFormer dehazing model but also improves its adaptability and generalization ability to input images.

[0093] The image inference module is responsible for calling the deep learning inference optimizer TensorRT, which in turn calls dehaze.engine to generate an executable dehazing program after compilation. The executable dehazing program reads the image processed by the image preprocessing module and performs the inference process to achieve image dehazing and obtain a dehazed remote sensing image.

[0094] The result processing and display module performs inverse normalization on the dehazed remote sensing image obtained by the image inference module to map the image back to the original pixel value range. Subsequently, the processed dehazed remote sensing image is visualized on the embedded computing device NVIDIA Jetson XavierNX, allowing users to view and compare the images before and after dehazing in real time. Figure 6Comparison of image dehazing results from different dehazing methods is shown in the following figures: (a) original foggy remote sensing image; (b) labeled fog-free remote sensing image; (c) dehazing result using the DCP method; (d) dehazing result using the AOD-Net method; (e) dehazing result using the MSBDN method; (f) dehazing result using the FFA-Net method; (g) dehazing result using the AECR-Net method; and (h) dehazing result using the method of this invention. Figure 6 As can be seen above, the ACER-Net method can effectively remove haze but causes severe color distortion and artifacts. Other methods also perform poorly, such as DCP, AOD-Net, MSBDN, and FFA-Net, which cannot effectively remove haze under non-uniform conditions. The FFCFormer dehazing model proposed in this invention can recover more structural information and preserve the edge information of the image. It can more effectively capture the global structure and periodic features of the image, reduce the loss of global information during the dehazing process, and improve the clarity, color consistency, and edge preservation of the dehazed image.

[0095] It should be understood that, inspired by the technical concept of this invention, those skilled in the art can make various improvements or modifications based on the above content without departing from the scope of this invention, and these modifications still fall within the protection scope of this invention.

Claims

1. A method for dehazing remote sensing images based on an improved Transformer model, characterized in that, The method includes the following steps: Step 1: Prepare a remote sensing image dataset consisting of multiple image pairs, and divide the dataset into a training set; the image pairs consist of fog-free remote sensing images and foggy remote sensing images corresponding to the fog-free remote sensing images; Step 2: Improve the feature extraction module in the existing Transformer model and use the improved feature extraction module. Named the dehazing module, the improvements include the introduction of a Fast Fourier Convolution (FFC) module and the adoption of a shift window partitioning scheme based on copy padding. Step 3: Train the improved Transformer model using the training set to obtain the FFCFormer dehazing model; Step 4: Use the FFCFormer dehazing model to dehaze the foggy remote sensing image input to the model.

2. The method as described in claim 1, characterized in that, The introduction of the Fast Fourier Convolution (FFC) module specifically involves introducing the FFC module before the scale normalization operation of the feature extraction module.

3. The method as described in claim 1, characterized in that, In step 4, the FFCFormer dehazing model is deployed on an embedded computing device to dehaze the input foggy remote sensing image.

4. The method as described in claim 3, characterized in that, Step 4 specifically includes the following steps: Step 4.1: Save the FFCFormer dehazing model as a .pth file and name it best.pth; Step 4.2: Prune and quantize best.pth, and name the simplified model dehazeformer.pth; Step 4.3: Convert dehazeformer.pth to ONNX format and name it dehazeformer.onnx; Step 4.4: Use the simplification tool onnxsim to perform graph transformation and constant folding operations on dehazeformer.onnx to eliminate redundant parameters and calculations in the model, and obtain the lightweight model dehazeformer-sim.onnx; Step 4.5: Use the deep learning inference optimizer TensorRT to convert dehazeformer-sim.onnx into a callable engine file, dehaze.engine; Step 4.6: Call the deep learning inference optimizer TensorRT on the embedded computing device. TensorRT calls the engine file dehaze.engine, which is then compiled to generate an executable dehazing program. Step 4.7: Read the input hazy remote sensing image in the executable dehazing program to perform dehazing processing on the hazy remote sensing image.

5. The method as described in claim 4, characterized in that, The embedded computing device is an NVIDIA Jetson Xavier NX.

6. A remote sensing image dehazing system based on an improved Transformer model, established based on the method described in claim 5, wherein the system is installed on an embedded computing device NVIDIA Jetson Xavier NX, characterized in that, The system includes: The image storage module is responsible for saving the collected foggy remote sensing images that need to be dehazed to the system album of the NVIDIA Jetson Xavier NX; The image preprocessing module is responsible for selecting one or more foggy remote sensing images from the system album using the image selector provided by the NVIDIA Jetson Xavier NX and preprocessing them, including scaling and normalization. The image inference module is responsible for calling the deep learning inference optimizer TensorRT. Through TensorRT, the deep learning inference optimizer calls the engine file dehaze.engine described in claim 5, and after compilation, generates an executable dehazing program. The executable dehazing program reads the image processed by the image preprocessing module and performs the inference process to achieve image dehazing and obtain a dehazed remote sensing image. The results processing and display module performs inverse normalization on the dehazed remote sensing images obtained by the image inference module, and then visualizes the processed dehazed remote sensing images on the NVIDIA Jetson Xavier NX.

Images