Remote sensing image defogging method and system based on improved Transform model

Abstract

The invention discloses a remote sensing image defogging method and system based on an improved Transform model, and relates to the technical field of image processing. The method comprises the following steps: preparing a remote sensing image data set composed of a plurality of image pairs, and dividing a training set from the data set; the image pair is composed of a fogless remote sensing image and a foggy remote sensing image corresponding to the fogless remote sensing image; a feature extraction module in an existing Transform model is improved, and the improved feature extraction module is named as a defogging module, wherein the improvement comprises introduction of a fast Fourier convolution (FFC) module; the training set is used to train the improved Transform model, and an FFCform defogging model is obtained; and carrying out defogging processing on the foggy remote sensing image input into the model by using the FFCFormer defogging model. According to the remote sensing image defogging system, an improved Transform model is transplanted and deployed to an embedded computing device NVIDIA Jetson Xavier NX, so that quick, real-time and visual remote sensing image defogging is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a remote sensing image defogging method and system based on an improved Transformer model. Background Art

[0002] Remote sensing images are often affected by atmospheric scattering effects in practical applications. When the clouds are dense and the haze is heavy, problems such as blurred edges and color distortion may occur, resulting in reduced image quality, slow processing speed, and affecting information acquisition and analysis. Remote sensing image defogging is a challenging task because atmospheric phenomena such as haze can cause light scattering and absorption, resulting in blurred target objects in remote sensing images. Removing these effects requires accurate scattering models and defogging algorithms, which places high demands on computing and physical models. In practical applications, deep learning model defogging methods have attracted widespread attention due to their excellent performance and generalization capabilities. The Transformer model has achieved remarkable results in fields such as natural language processing with its powerful feature extraction and modeling capabilities. In recent years, it has also been applied to image processing tasks. For example, remote sensing image defogging methods based on the Transformer model can handle daytime, nighttime and remote sensing image defogging tasks.

[0003] The main advantage of the Transformer model is its self-attention mechanism, which can capture the dependencies between any two elements in a sequence, so that it can effectively process global information and long-distance dependencies in image dehazing. However, the Transformer model may face the problem of local information loss when processing foggy remote sensing images. In addition, using the Transformer model to process large-scale remote sensing image data usually requires a lot of computing resources and memory, resulting in high computational cost and slow computing speed. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a remote sensing image defogging method and system based on an improved Transformer model.

[0005] The technical solution of the present invention is:

[0006] A first aspect of the present invention provides a remote sensing image defogging method based on an improved Transformer model, the method comprising the following steps:

[0007] Step 1: prepare a remote sensing image data set consisting of a plurality of image pairs, and divide a training set from the data set; the image pairs consist of a fog-free remote sensing image and a foggy remote sensing image corresponding to the fog-free remote sensing image;

[0008] Step 2: Improve the feature extraction module in the existing Transformer model and name the improved feature extraction module as a defogging module, wherein the improvement includes introducing a fast Fourier convolution (FFC) module;

[0009] Step 3: Use the training set to train the improved Transformer model to obtain the FFCFormer dehazing model;

[0010] Step 4: Use the FFCFormer defogging model to defog the foggy remote sensing image input into the model.

[0011] The introducing of the fast Fourier convolution (FFC) module specifically refers to introducing the FFC module before the scale normalization operation of the original feature extraction module.

[0012] The improvement of the feature extraction module in the existing Transformer model described in step 2 also includes adopting a shift window division scheme based on copy filling to replace the shift window division scheme of reflection filling.

[0013] In step 4, the FFCFormer defogging model is deployed on an embedded computing device to perform defogging on the input foggy remote sensing image.

[0014] The step 4 specifically includes the following steps:

[0015] Step 4.1: Save the FFCFormer dehazing model as a .pth file and name it best.pth;

[0016] Step 4.2: Prune and quantize best.pth, and the simplified model is named dehazeformer.pth;

[0017] Step 4.3: Convert dehazeformer.pth to an ONNX file and name it dehazeformer.onnx;

[0018] Step 4.4: Use the simplification tool onnxsim to perform graph transformation and constant folding operations on dehazeformer.onnx to eliminate redundant parameters and calculations in the model and obtain the lightweight model dehazeformer-sim.onnx;

[0019] Step 4.5: Use the deep learning inference optimizer TensorRT to convert dehazeformer-sim.onnx into a callable engine file dehaze.engine;

[0020] Step 4.6: Call the deep learning inference optimizer TensorRT on the embedded computing device, call the engine file dehaze.engine through the deep learning inference optimizer TensorRT, and generate an executable dehazing program after compilation;

[0021] Step 4.7: Read the input foggy remote sensing image in the executable defogging program to implement defogging of the foggy remote sensing image.

[0022] The embedded computing device is NVIDIA Jetson Xavier NX.

[0023] A second aspect of the present invention provides a remote sensing image defogging system based on an improved Transformer model, which is established based on the method described above. The system is arranged on an embedded computing device NVIDIA Jetson Xavier NX, and the system comprises:

[0024] The image storage module is responsible for saving the collected foggy remote sensing images that need to be defogged to the system album of NVIDIA Jetson Xavier NX;

[0025] The image preprocessing module is responsible for selecting one or more foggy remote sensing images from the system album through the image selector of NVIDIA Jetson XavierNX and preprocessing them, including scaling and normalization.

[0026] The image inference module is responsible for calling the deep learning inference optimizer TensorRT, calling the engine file dehaze.engine through the deep learning inference optimizer TensorRT, and generating an executable dehazing program after compilation; the image processed by the image preprocessing module is read in the executable dehazing program and the inference process is executed to realize image dehazing and obtain a dehazed remote sensing image;

[0027] The result processing and display module performs denormalization on the defogging remote sensing image obtained by the image reasoning module, and visualizes the processed defogging remote sensing image on the NVIDIA Jetson Xavier NX.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] 1. Remote sensing images are often affected by atmospheric conditions such as haze and clouds during the acquisition process, resulting in reduced image quality and blurred edges, which affects subsequent object recognition and analysis. To this end, the present invention has made an innovative design for retaining edge pixels, replacing the shift window division scheme based on reflection filling in the feature extraction module of the existing Transformer model with a shift window division scheme based on copying and filling, and copying the edge pixels of the image to the filling area, which can maintain the edge information of the image and obtain high-quality dehazed remote sensing images.

[0030] 2. The fast Fourier convolution (FFC) module is introduced into the feature extraction module of the existing Transformer model. The FFC module has higher computational efficiency and better global perception ability. In image dehazing processing, the FFC module can help the Transformer model better understand the global structure and periodic characteristics of the image, and can more effectively capture the global structure and periodic characteristics of the image, reduce the global information loss in the dehazing process, and improve the clarity and color consistency of the dehazed image.

[0031] 3. Optimize the improved Transformer model. First, perform pruning and quantization operations to convert the improved Transformer model into ONNX format, and then use the simplification tool onnxsim to eliminate redundant parameters and calculations in the model, greatly reducing the number of parameters and computational complexity of the model.

[0032] 4. The system of the present invention deploys the optimized improved Transformer model on the embedded computing device NVIDIA Jetson Xavier NX, which can perform model inference immediately after inputting foggy remote sensing images, greatly reducing computing costs and improving system response speed. This is of great significance for real-time or near real-time remote sensing image defogging. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the remote sensing image defogging method based on the improved Transformer model in this embodiment;

[0034] Figure 2 This is a structural diagram of the Transformer model in this implementation mode;

[0035] Figure 3 1 is a structural comparison diagram of the feature extraction module before and after the improvement in the Transformer model of this embodiment, wherein (a) is the structural diagram of the feature extraction module before the improvement; (b) is the structural diagram of the feature extraction module after the improvement;

[0036] Figure 4 This is a flow chart of step 4 of this implementation mode;

[0037] Figure 5 This is a structural block diagram of the remote sensing image defogging system based on the improved Transformer model in this embodiment;

[0038] Figure 6 A comparison chart of the visual dehazing results of different image dehazing methods. DETAILED DESCRIPTION

[0039] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present application are given in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thoroughly and comprehensively understood.

[0040] The present invention provides a remote sensing image defogging solution based on an improved Transformer model, and its core ideas are as follows: In view of the fact that the existing remote sensing image defogging method based on the Transformer model still has the problem of global information loss, the present invention introduces the FFC module in the Transformer model to extract features from local and global perspectives and fuse them respectively; in view of the problem of edge blur in the existing remote sensing image defogging method based on the Transformer model, a shift window division scheme based on copy filling is used to replace the shift window division scheme based on reflection filling, and the edge pixels are retained in a targeted manner; the fused features are scaled and replicated, and the spatial attention weights are calculated and dynamically aggregated to improve the quality and fidelity of the defogging image. In addition, in view of the problems that the existing remote sensing image defogging method based on the Transformer model has a large number of calculation parameters and a long time, the present invention further performs optimization operations such as pruning, quantization, and elimination of redundant parameters on the Transformer model after the above-mentioned improved operation to reduce the number of parameters and calculation complexity and improve the calculation speed. The present invention also provides a remote sensing image defogging system based on the improved Transformer model, which transplants and deploys the optimized improved Transformer model to the embedded computing device NVIDIA Jetson Xavier NX to achieve fast, real-time and visual remote sensing image defogging.

[0041] The overall flow chart of the remote sensing image defogging method based on the improved Transformer model in this embodiment is as follows: Figure 1 As shown, the following steps are included:

[0042] Step 1: prepare a remote sensing image data set consisting of a plurality of image pairs, and divide a training set from the data set; the image pairs consist of a fog-free remote sensing image and a foggy remote sensing image corresponding to the fog-free remote sensing image;

[0043] The public RS-Haze dataset contains 6,000 fog-free remote sensing images with various terrains and 1,500 cirrus cloud channels with distributions similar to natural haze. In this embodiment, nine synthetic fog images are artificially generated using cirrus cloud channels for each fog-free remote sensing image in the RS-Haze dataset, including 18,000 fog images of three different fog densities: light fog, medium fog, and thick fog. This embodiment selects 18,000 fog-free remote sensing images and 18,000 corresponding thick fog images to construct the remote sensing image dataset of this embodiment. Each fog-free remote sensing image in the dataset and the thick fog image corresponding to the fog-free remote sensing image constitute an image pair. Therefore, the remote sensing image dataset of this embodiment has a total of 18,000 image pairs, from which this embodiment divides 14,400 pairs of training sets for pre-training and 1,800 pairs of test sets for testing.

[0044] Step 2: improving the feature extraction module in the existing Transformer model and naming the improved feature extraction module as a defogging module, wherein the improvement includes: introducing a fast Fourier convolution (FFC) module and adopting a shift window partitioning scheme based on copy filling instead of a shift window partitioning scheme based on reflection filling;

[0045] The specific improvements of the feature extraction module in the existing Transformer model in this implementation are as follows: a fast Fourier convolution (FFC) module is added before the scale normalization operation of the original feature extraction module to compensate for the fact that the Transformer model will lose global information, resulting in color distortion, pixel blur and other deficiencies in the dehazing result; after the scale normalization operation of the feature extraction module, a shift window division scheme based on copy filling is used instead of a shift window division scheme based on reflection filling to specifically retain edge pixels to improve the quality and fidelity of the dehazing image. The improved feature extraction module is named the dehazing module, and the overall structure of the improved Transformer model is the same as that of Figure 2 The overall structure of the Transformer model before improvement is shown.

[0046] The specific processing process of the defogging module on the input image includes the following steps:

[0047] Step 2.1: The image obtained after a 3×3 convolution of the foggy remote sensing image I Input FFC module, FFC module for I r Frequency domain features are extracted from the global branch and the local branch respectively, and the extracted frequency domain features are fused to obtain a fused feature map containing frequency domain contextual connections.

[0048] like Figure 2 and Figure 3 As shown in the figure, the foggy remote sensing image I input to the Transformer model is obtained after a 3×3 convolution. Enter the FFC module, where H represents height, W represents width, and C represents the number of channels. Figure 3 What is shown is a structural comparison diagram of the feature extraction module before and after improvement. Figure (a) is the original unimproved feature extraction module structure diagram, and Figure (b) is the improved defogging module structure diagram of this embodiment, wherein the dotted box is the fast Fourier convolution module, namely the FFC module.

[0049] The FFC module consists of a global branch and a local branch. The global branch has a global receptive field and is responsible for converting the spatial features of the input data into the frequency domain and extracting the spectrum data, and then converting it back to the spatial format; the local branch is specifically used to extract local information from the global information. In this embodiment, the specific operation of the fast Fourier convolution branch is to first convert the image I along the dimension of the feature channel r Divided into data I sent to the local branch r l and data fed into the global branch Right now Data fed into local branches Responsible for learning from the local neighborhood and feeding data into the global branch Designed to capture long-range context, where α in ∈[0, 1] represents the percentage of feature channels assigned to the global branch. In this implementation, α in The local branch contains two parallel 3×3 convolutional layers, called local to local branch Y l→l and local to global branch Y l→g , the result of the data sent to the local branch after passing through two parallel convolutional layers is recorded as Y l The global branch contains a signal converter, namely the global to global branch Y g→g , and a parallel 3×3 convolutional layer, the global to local branch Y g→l The data sent to the global branch passes through the parallel signal generator and the convolution layer, and the result is recorded as Y g .

[0050] The update process of the fast Fourier convolution branch is expressed as:

[0051]

[0052] where Y l , Y g→l , f g , f l→g Respectively represent the operations on the input data of different branches, f l , f g→l, f l→g is a normal convolution operation, f g It is a signal transformation operation.

[0053] In the signal converter, the data I fed into the global branch is first r g As input, and use 1×1 convolution-RELU to reduce the number of data channels. Then the fast Fourier transform (FFT)-1×1-RELU-inverse fast Fourier transform (IFFT) operations are performed in sequence. For the processing of real tensors, the fast Fourier transform produces conjugate symmetric results. Considering the data availability, only half of the results with conjugate symmetry are retained, and the other half is reconstructed by a simple conjugation method. The real and imaginary parts of these results are concatenated along the channel dimension and then processed by 1x1 convolution, normalization, and RELU activation. Finally, the result is decomposed into real and imaginary parts and converted back to a logarithmic format. This method takes advantage of the ability of the fast Fourier transform to process information in the frequency domain and to capture local features in global information through local operations. Channel fusion is performed at the final stage of processing, and the number of channels is restored through another 1×1 convolution operation to reconstruct the channel dimension of the feature and obtain a fused feature map containing frequency domain contextual connections.

[0054] Step 2.2: Perform shift window division and spatial attention dynamic aggregation based on copy filling;

[0055] After the FFC module is processed, the fused feature map output by the FFC module Entering the dehazing module, it is first scale-normalized, and then the shifted window division based on copy filling is performed. Existing Transformer models mostly use shifted window division based on reflection filling. Reflection filling sometimes introduces unnatural reflection effects at the edges, which may have a negative impact on the dehazing task. Copy filling can avoid generating overly complex boundary effects. This method has a high consistency in edge processing. Since the edge pixels are copied, it will not introduce artifacts or false information that may be caused by reflection filling. The specific operation of the shifted window division based on copy filling is: according to the fused feature map after scale normalization Size, this implementation selects a fixed size window of 3x3 pixels to scan the feature map to be fused from Starting from one corner, slide the window row by row to cover the entire When the window is in the fusion feature map When sliding up, especially when the window is close to When the edge or corner of the window is The actual size of the window will result in some blank areas. There is no actual pixel data in these blank areas, so a filling technique is needed to process these blank areas. The pixel data near the boundary is copied and filled into the blank area in the window. The resulting feature map after copying and filling is recorded as

[0056] Next, the filled feature image Flatten into a one-dimensional vector, flatten into a one-dimensional sequence according to the spatial dimension H×W, where each sequence is a C-dimensional feature vector. The flattened feature vector is projected to the multi-head attention factors Q, K, and V through a linear layer, and the global and local information obtained by the projection is sampled and the local information is enhanced through spatial information aggregation. The spatial information aggregation weight is dynamic, but the weight is always positive, so additional convolution is required on V. In the multi-head attention module, Q and K are channel-fused, and V is subjected to a 1×1 convolution operation. The specific scheme of spatial information aggregation is:

[0057]

[0058] Where d is the dimension of the vector and S is the relative position offset. represents V before channel division, and Conv(·) represents 1×1 convolution. The output obtained by aggregating spatial information is sent to the linear layer to obtain the linear output J liner , the linear output J liner After channel clipping, it is sent to the affine layer. In the affine layer, the linear output J liner And the padded feature image Linear weighted fusion is performed to obtain two feature maps denoted as p1 and p2. Next, the feature maps p1 and p2 are sent to the multi-layer perceptron MLP. The multi-layer perceptron MLP consists of a residual structure of linear layer-RELU-linear layer. The first linear layer f(·) is used to project p1 to Use global average pooling GAP(·), softmax function and split operation to get the fusion weight:

[0059]

[0060] By using the fusion weights {a1, a2}, the fusion result q is: That is, under the guidance of the fusion result q, the improved Transformer model can complete the dynamic aggregation of spatial attention and achieve higher quality dehazing results in the dehazing reasoning process.

[0061] Step 3: Use the training set to train the improved Transformer model to obtain the FFCFormer dehazing model;

[0062] In the dehazing inference process of the Transformer model, current image dehazing networks generally predict a haze-free reconstructed image. Since the degradation model is approximate, it is beneficial to introduce priors without strong constraints, so the dehazing inference process can be obtained by:

[0063]

[0064] Rewritten as:

[0065]

[0066] Where K(x) = 1 / t(x)-1, B(x) = -(1 / t(x)-1)A, where I is the captured hazy fog image, A is the global atmospheric light, and t is the medium transmission map. The transmission amount can be expressed as: t(x) = e -βd(x) , where β is the atmospheric scattering coefficient and d is the depth of field.

[0067] The dehazing task of the Transformer model is to predict the scattered The scattering O can be further decomposed into and Therefore, using the fusion result q in step 2.2, the Transformer model imposes soft constraints on the relationship between K(x) and B(x), that is,

[0068] O(x)=qK(x)+(1-q)B(x) (6)

[0069] Therefore, by deriving the scattering O(x), the Transformer model can restore the haze-free reconstructed image The network is a 5-layer Unet structure. Downsampling and upsampling represent the downsampling layer and upsampling layer respectively. Two downsampling and two upsampling are required after each defogging module. Finally, a RELU layer and a 3×3 convolution layer are used to restore the fog-free reconstructed image with the same size as the input.

[0070] This implementation uses the NVIDIA V100 graphics card on the Pytorch platform to train the improved Transformer model. Before training, all images in the training set are resized to 400×400. The improved Transformer model is trained using 150 rounds of L1 loss, and the learning rate is adjusted according to the model's minimum batch size. L1 loss is used to measure the mean absolute error between the model's prediction results and the true label. Specifically, for a sample set of size N, L1 loss is defined as follows:

[0071]

[0072] Among them, l i represents the true label of the sample, The improved Transformer model predicts the label of the sample. The absolute error of each sample is averaged to obtain the L1 loss. The training process uses the Adam optimizer with a batch size of 8 and a patch size of 128×128 for 480K iterations, with a decay rate of β1=0.9 for the first-order moment estimate and a decay rate of β2=0.999 for the second-order moment estimate. The initial learning rate is set to 2×10 -4 , and use the cosine annealing decay strategy as the training strategy to gradually reduce the learning rate to 10 -6 In this embodiment, the optimal model obtained during the training process is called the FFCFormer defogging model.

[0073] Step 4: Use the FFCFormer defogging model to defog the input foggy remote sensing image.

[0074] Furthermore, in practical applications, the Transformer model usually requires a large amount of computing resources and memory when processing large-scale remote sensing image data, and the computing cost is high and the computing speed is slow. To solve this problem, in step 4, the present invention first optimizes the FFCFormer defogging model obtained in step 3, and then deploys the optimized FFCFormer defogging model on the embedded computing device NVIDIAJetson Xavier NX, and defogs the input foggy remote sensing image on the device, such as Figure 4 Step 4 shown specifically includes the following steps:

[0075] Step 4.1: Save the FFCFormer dehazing model as a .pth file and name it best.pth;

[0076] Step 4.2: Prune and quantize best.pth, and the simplified model is named dehazeformer.pth;

[0077] Step 4.3: Convert dehazeformer.pth to an ONNX file and name it dehazeformer.onnx;

[0078] In this implementation, the torch.onnx conversion tool provided by PyTorch converts dehazeformer.pth into a file in ONNX (Open Neural Network Exchange) format and names it dehazeformer.onnx. ONNX has its cross-platform characteristics, so converting the model to ONNX format can realize cross-platform deployment of the model.

[0079] Step 4.4: Use the simplification tool onnxsim to perform graph transformation and constant folding operations on dehazeformer.onnx to eliminate redundant parameters and calculations in the model and obtain the lightweight model dehazeformer-sim.onnx.

[0080] In some cases, the model converted to ONNX format may contain a large number of redundant parameters and complex calculations, which makes the model difficult to understand and optimize. Therefore, this implementation method uses the simplification tool onnxsim to perform a series of graph transformations and constant folding operations on the ONNX format model to eliminate redundant parameters and calculations in the model, and obtains a lightweight model named dehazeformer-sim.onnx. These operations can significantly reduce the complexity and size of the model while maintaining the performance and accuracy of the model;

[0081] Step 4.5: Use the deep learning inference optimizer to convert dehazeformer-sim.onnx into a callable engine file dehaze.engine;

[0082] This implementation uses the high-performance deep learning inference optimizer TensorRT to perform low-latency, high-throughput model conversion on the lightweight model dehazeformer-sim.onnx, and generates a C++ callable engine file dehaze.engine, further improving the running speed of the model on the device;

[0083] Step 4.6: Call the deep learning inference optimizer TensorRT on the embedded computing device, call dehaze.engine through the deep learning inference optimizer TensorRT, and generate an executable dehazing program after compilation;

[0084] The embedded computing device of this embodiment adopts NVIDIA Jetson Xavier NX.

[0085] Step 4.7: Read the input foggy remote sensing image in the executable defogging program to implement defogging of the foggy remote sensing image.

[0086] In the executable defogging program, the foggy remote sensing image in the test set is read as the input of the FFCFormer defogging model, and the foggy remote sensing image is sent to the FFCFormer defogging model for defogging reasoning. In the image defogging reasoning process, this embodiment uses the GPU acceleration technology of the embedded computing device NVIDIA Jetson Xavier NX to accelerate the reasoning of the FFCFormer defogging model and realize the defogging of foggy remote sensing images. NVIDIA Jetson Xavier NX uses a 384-core NVIDIA Volta GPU equipped with 48 Tensor Cores and a 6-core NVIDIA Carmel ARM v8.2 64-bit CPU, providing powerful computing power, which can significantly reduce the reasoning time and help achieve high-precision and high-speed remote sensing image defogging.

[0087] The results of the dehazing experiment on the RS-Haze test set are shown in Table 1. The scenes of remote sensing images are more monotonous than natural scenes, so the present invention is easier to estimate the potential color and brightness of the image, thereby improving the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), where the peak signal-to-noise ratio represents the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy. The larger the ratio, the higher the image quality; structural similarity is used to measure the similarity and authenticity of the image before and after distortion. The larger the value, the higher the image similarity before and after dehazing. The haze concentration of the RS-Haze dataset is uneven, which destroys the high-frequency information of the image. The FFC module can capture global information and long-distance dependencies, avoiding the situation where global information is lost in the result. This enables the FFCFormer dehazing model to better capture the global structure and periodic characteristics of the image, reduce the global information loss in the dehazing process, and improve the clarity and color consistency of the dehazed image, thereby verifying the effectiveness of the method of the present invention. At the same time, the present invention also achieves obvious advantages in parameter quantity (#Param) and multiplication-addition accumulation number (MACs), where parameter quantity refers to the total number of all trainable parameters in the deep learning model. These parameters include weights and biases, which are usually present in each layer of the network. The more parameters there are, the longer the storage requirements and training time of the model, and the higher the computational cost of training and reasoning; the multiplication-addition accumulation number refers to the total number of all multiplication and addition operations in the model when performing a forward propagation or an inference. The more MACs there are, the more computing resources and time the model usually requires to complete a forward propagation. The FFCFormer dehazing model can have fewer multiplication-addition accumulation operations while having a lower parameter quantity, proving that the present invention is faster in network convergence speed and reasoning speed, and can greatly save computing resources and memory when processing large-scale remote sensing image data.

[0088] Table 1 Objective evaluation, parameter quantity, and multiplication-addition-accumulation operation comparison results of different dehazing methods on RS-Haze dataset

[0089] Dehazing Method PSNR↑ SSIM↑ #Param↓ MACs↓ DCP 17.86 0.734 - - AOD-Net 24.90 0.830 0.002M 0.115G MSBDN 38.57 0.965 31.35M 41.54G FFA-Net 39.39 0.969 4.456M 287.8G AECR-Net 35.69 0.959 2.611M 52.50G Method of the present invention 39.95 0.975 2.965M 29.36G

[0090] The present invention also provides a remote sensing image defogging system based on an improved Transformer model, and its structural block diagram is as follows: Figure 5 As shown in the figure, the FFCFormer defogging model is deployed to the embedded computing device NVIDIA Jetson Xavier NX to achieve low-latency, high-throughput defogging reasoning process and real-time defogging result visualization. The system includes: image storage module, image preprocessing module, image reasoning module, result processing and display module.

[0091] The image storage module is responsible for storing the collected foggy remote sensing images that need to be defogged. These images can be synthetic remote sensing fog images similar to the RS-Haze dataset, or remote sensing images of real haze scenes can be taken using the camera of a mobile device. The collected synthetic fog images or the taken foggy remote sensing images are saved in the system album of the device NVIDIA Jetson Xavier NX. When the number of foggy remote sensing images to be processed is huge, the collected images are subjected to image numbering and storage operations, including giving unique identifiers, writing index fields, etc., to speed up the retrieval speed, and the numbered foggy remote sensing images are sent to the image storage module.

[0092] The image preprocessing module is responsible for selecting the saved foggy remote sensing image from the system album through the image selector of the device itself and performing preprocessing operations on it to ensure that the selected foggy remote sensing image meets the input requirements of the FFCFormer defogging model. During the preprocessing process, the image is first scaled to adjust the size of the image so that it is compatible with the input layer of the FFCFormer defogging model. Through the scaling operation, it can be ensured that the image has a uniform size before being input into the FFCFormer defogging model, thereby improving the stability and efficiency of the FFCFormer defogging model. Secondly, the image is normalized, that is, the pixel value range of the image is adjusted, the pixel value is reduced to between 0 and 1, and the pixel value distribution of the image is more uniform. Normalization is not only conducive to the accurate extraction of image features by the FFCFormer defogging model, but also can improve the adaptability and generalization ability of the FFCFormer defogging model to the input image.

[0093] The image reasoning module is responsible for calling the deep learning reasoning optimizer TensorRT, calling dehaze.engine through the deep learning reasoning optimizer TensorRT, and generating an executable dehazing program after compilation; in the executable dehazing program, the image processed by the image preprocessing module is read and the reasoning process is executed to realize image dehazing and obtain a dehazed remote sensing image.

[0094] The result processing and display module performs a denormalization process on the defogging remote sensing image obtained by the image reasoning module to map the image back to the original pixel value range. Subsequently, the processed defogging remote sensing image is visualized on the embedded computing device NVIDIA Jetson Xavier NX so that users can view and compare the images before and after defogging in real time. Figure 6The following are comparisons of the image defogging visualization results of different defogging methods, where (a) is the original remote sensing image with fog, (b) is the labeled remote sensing image without fog, (c) is the defogging result of the DCP method, (d) is the defogging result of the AOD-Net method, (e) is the defogging result of the MSBDN method, (f) is the defogging result of the FFA-Net method, (g) is the defogging result of the AECR-Net method, and (h) is the defogging result of the method of the present invention. Figure 6 It can be seen from the above that the ACER-Net method can effectively remove haze but will cause serious color distortion and artifacts. Other methods also performed poorly, such as DCP, AOD-Net, MSBDN and FFA-Net, which cannot effectively remove haze under non-uniform conditions. The FFCFormer dehazing model proposed in the present invention can restore more structural information and maintain the edge information of the image, can more effectively capture the global structure and periodic features of the image, reduce the loss of global information in the dehazing process, and improve the clarity, color consistency and edge retention of the dehazed image.

[0095] It should be understood that, inspired by the technical concept of the present invention, those skilled in the art may make various improvements or changes based on the above content without departing from the content of the present invention, which still fall within the protection scope of the present invention.

Claims

1. A remote sensing image defogging method based on an improved Transformer model, characterized in that: The method comprises the following steps: Step 1: prepare a remote sensing image data set consisting of a plurality of image pairs, and divide a training set from the data set; the image pairs consist of a fog-free remote sensing image and a foggy remote sensing image corresponding to the fog-free remote sensing image; Step 2: Improve the feature extraction module in the existing Transformer model and name the improved feature extraction module as a defogging module, wherein the improvement includes introducing a fast Fourier convolution (FFC) module; Step 3: Use the training set to train the improved Transformer model to obtain the FFCFormer dehazing model; Step 4: Use the FFCFormer defogging model to defog the foggy remote sensing image input into the model.

2. The method according to claim 1, characterized in that The introducing of the fast Fourier convolution (FFC) module specifically refers to introducing the FFC module before the scale normalization operation of the original feature extraction module.

3. The method according to claim 1, characterized in that The improvement of the feature extraction module in the existing Transformer model described in step 2 also includes adopting a shift window division scheme based on copy filling to replace the shift window division scheme of reflection filling.

4. The method according to claim 1, characterized in that In step 4, the FFCFormer defogging model is deployed on an embedded computing device to perform defogging on the input foggy remote sensing image.

5. The method according to claim 4, characterized in that The step 4 specifically includes the following steps: Step 4.1: Save the FFCFormer dehazing model as a .pth file and name it best.pth; Step 4.2: Prune and quantize best.pth, and the simplified model is named dehazeformer.pth; Step 4.3: Convert dehazeformer.pth to an ONNX file and name it dehazeformer.onnx; Step 4.4: Use the simplification tool onnxsim to perform graph transformation and constant folding operations on dehazeformer.onnx to eliminate redundant parameters and calculations in the model and obtain the lightweight model dehazeformer-sim.onnx; Step 4.5: Use the deep learning inference optimizer TensorRT to convert dehazeformer-sim.onnx into a callable engine file dehaze.engine; Step 4.6: Call the deep learning inference optimizer TensorRT on the embedded computing device, call the engine file dehaze.engine through the deep learning inference optimizer TensorRT, and generate an executable dehazing program after compilation; Step 4.7: Read the input foggy remote sensing image in the executable defogging program to implement defogging of the foggy remote sensing image.

6. The method according to claim 5, characterized in that The embedded computing device is NVIDIA Jetson Xavier NX.

7. A remote sensing image defogging system based on an improved Transformer model, established based on the method of claim 5, the system being arranged on an embedded computing device NVIDIA Jetson Xavier NX, characterized in that: The system includes: The image storage module is responsible for saving the collected foggy remote sensing images that need to be defogged to the system album of NVIDIA Jetson Xavier NX; The image preprocessing module is responsible for selecting one or more foggy remote sensing images from the system album through the image selector of NVIDIA Jetson XavierNX and preprocessing them, including scaling and normalization. An image reasoning module is responsible for calling the deep learning reasoning optimizer TensorRT, calling the engine file dehaze.engine described in claim 5 through the deep learning reasoning optimizer TensorRT, and generating an executable dehazing program after compilation; The image processed by the image preprocessing module is read in the executable defogging program and the reasoning process is performed to achieve image defogging to obtain a defogging remote sensing image; The result processing and display module performs denormalization on the defogging remote sensing image obtained by the image reasoning module, and visualizes the processed defogging remote sensing image on the NVIDIA Jetson Xavier NX.

Images