An image dehazing transformer based on Legendre polynomial approximation and error compensation module

Abstract

The present invention discloses an image defogging transformer based on Legendre polynomial approximation and error compensation modules, comprising a multi-scale feature extraction and embedding module, a Legendre polynomial self-attention module, a calculation error compensation module, and a sampling error compensation module. The multi-scale feature extraction and embedding module is used to perform multi-scale feature extraction; the Legendre polynomial self-attention module approximates the exponential function in the softmax function by using Legendre polynomials to reduce computational complexity; and the calculation error compensation module and the sampling error compensation module are used to reduce errors introduced in the calculation and sampling processes, thereby enhancing the image defogging effect.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to an image defogging method based on a Transformer architecture, which is applicable to various visual measurement systems including autonomous driving, monitoring systems, and satellite image processing. Background Art

[0002] Images captured under adverse weather conditions such as haze are often affected by absorption and scattering from atmospheric particles, resulting in reduced image contrast, blurred details, and color distortion. These degradation phenomena severely impact the performance of visual measurement systems, such as those used in autonomous driving, surveillance, and satellite imagery. Traditional image dehazing methods based on convolutional neural networks (CNNs) can restore image quality to a certain extent. However, due to CNN's limitations in capturing long-range pixel relationships and multi-scale semantic information, dehazed images can still retain residual fog and unsatisfactory detail restoration.

[0003] In recent years, the Transformer model has been widely used in computer vision and has demonstrated strong performance in high-level visual tasks. However, directly applying the Transformer model to low-level pixel-level tasks (such as image dehazing) suffers from high computational complexity and insufficient ability to extract local information. Therefore, how to reduce the computational complexity of the Transformer model while maintaining effective dehazing and improving its ability to recover detail remains a pressing challenge in existing technologies. Summary of the Invention

[0004] The present invention aims to overcome the shortcomings of the prior art by providing an image dehazing transformer based on Legendre polynomial approximation and error compensation, addressing the high computational complexity and insufficient detail restoration issues of the prior art. This invention introduces Legendre polynomial approximation to reduce the computational complexity of the self-attention mechanism and uses an error compensation module to correct errors in the approximate calculation and sampling processes, thereby achieving efficient and high-quality image dehazing.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] An image dehazing transformer based on Legendre polynomial approximation and error compensation module. In the self-attention calculation, the transformer calculates the dot product between the query, key, and value and converts it into attention weight through the softmax function. The computational complexity of the above operation is Where: O represents the complexity, L represents the length of the input sequence, d kIndicates the dimensional size of the key feature; the image dehazing transformer includes a multi-scale feature extraction and embedding module, a Legendre polynomial self-attention module, a computational error compensation module, and a sampling error compensation module. The multi-scale feature extraction and embedding module is used for multi-scale feature extraction; the Legendre polynomial self-attention module approximates the exponential function in the softmax function through Legendre polynomials to reduce the computational complexity; at the same time, the computational error compensation module and the sampling error compensation module are used to reduce the errors introduced in the computation and sampling processes, thereby enhancing the image dehazing effect.

[0007] Furthermore, the Legendre polynomial self-attention module uses Legendre polynomials to adjust the exponential function e in the softmax function. x Approximation is performed to reduce the computational complexity to And use the associativity of matrix multiplication to optimize self-attention calculation.

[0008] Furthermore, the processing flow of the image dehazing transformer is as follows:

[0009] (1) Input preprocessing: Perform a convolution operation on the input haze image I and extract the shallow visual features f0 through a 3×3 convolution layer:

[0010] f0=Conv(I)

[0011] Among them, Conv() represents the convolution operation;

[0012] (2) Multi-scale feature embedding: The shallow visual feature f0 is input into the multi-scale feature extraction and embedding module to generate multi-scale tokens; the query, key and value matrices are generated using the depthwise separable convolution method;

[0013] (3) In the first layer of encoding, the shallow visual feature f0 is calculated through the Legendre polynomial self-attention module to output the feature E1:

[0014] Q1,K1,V1=Patch embedding(f0)

[0015] E1=L-Former(Q1,K1,V1)

[0016] Among them, Patch embedding() represents the multi-scale feature extraction embedding module, L-Former(·) represents the Legendre polynomial self-attention module, which uses the feedforward network FFN to implement nonlinear transformation;

[0017] The i-th layer of the subsequent encoding obtains multi-scale features through downsampling operation:

[0018] E i =L-Former(Patch embedding(↓Ei-1 )),i=2,3

[0019] Among them, ↓ represents the downsampling operation;

[0020] (4) At the jth layer of decoding, an upsampling operation is first performed, and then the features output by each layer in the encoding process are directly transferred and fused into the decoder in combination with the selective kernel feature fusion module:

[0021] D j =L-Former(Patchembedding(SKFF(E i ,↑D j+1 ))),j=2,1;i=2,1;

[0022] Among them, SKFF() represents the selective kernel feature fusion module, and ↑ represents the upsampling operation. In each upsampling operation, the sampling error compensation module is used to perform error compensation on the decoded output feature map to restore the structure and texture of the dehazed image. j Represents the output feature map of layer j in the corresponding decoding process;

[0023] (5) Generate dehazed image: Generate the residual image between the input haze image I and the haze-free image through a 3×3 convolutional layer; add the residual image to the original input haze image I to obtain the dehazed image J:

[0024] J = I + Conv(D1);

[0025] During the approximation calculation process of the Legendre polynomial self-attention module, the computational error compensation module reduces the error introduced in the approximation calculation process by compensating weights, thereby enhancing the dehazing performance of the image dehazing transformer.

[0026] Furthermore, in the Legendre polynomial self-attention module, Legendre polynomials are used to approximate e x ; Legendre polynomial P n (x) is an orthogonal polynomial defined on the interval [-1,1], satisfying the recurrence relation:

[0027]

[0028] n represents the order of the corresponding Legendre polynomial, and x is the independent variable;

[0029] The best approximation interval of Legendre polynomial is selected as [-0.5, 0.5] to improve the calculation accuracy; the Legendre polynomial is used to approximate e x The approximate expression is:

[0030]

[0031] N is the total order of Legendre polynomials;

[0032] Among them, a n are the coefficients to be determined; the values ​​of the coefficients are determined by minimizing the approximation error function E(a0,a1); the calculation formula is as follows:

[0033]

[0034] Among them, take N = 2, n = 0, 1; select a0 = 1.04218639, a1 = 1 for self-attention calculation;

[0035] Finally, after using Legendre polynomial approximation, the calculation of self-attention is expressed as:

[0036]

[0037] Among them, A L is the attention score obtained by Legendre approximation; represents the query matrix, L is the length of the input sequence, d k It is the key feature dimension; represents the bond matrix; Represents a value matrix, where d v is the dimension of the value; T is the matrix transpose operation, M is the number of dimensions of the matrix column of K, and m is the dimension sequence number.

[0038] Furthermore, the sampling error compensation module SECM estimates the sampling error e by the following method: s And make compensation:

[0039] e s =Conv(F-down(up(F)))

[0040] Among them, F is the feature map output by the Legendre polynomial self-attention module, down() and up() represent downsampling and upsampling operations respectively; the feature map F after error compensation UP for:

[0041] F UP =up(F)+up(e s )

[0042] The computational error compensation module CECM generates the compensation weight matrix E by learning the nonlinear mapping relationship between Q, K, and V c :

[0043] E c =ReLU(Conv(concat(Q,K,V)))

[0044] Among them, concat() represents feature concatenation, Conv() represents convolution operation; the self-attention output after compensation is:

[0045]

[0046] in, Represents element-wise multiplication.

[0047] Preferably, the present invention also provides a training method based on the above-mentioned image defogging transformer, comprising:

[0048] S1. Extract the images to be trained from the RESIDE dataset;

[0049] S2. Perform data augmentation on the training images;

[0050] S3. Input the enhanced image to be trained into the image dehazing transformer for forward propagation and calculate the L1 loss;

[0051] S4. Optimize the parameters of the image dehazing transformer by backpropagation and update the weights of the image dehazing transformer using the Adam optimizer.

[0052] S5. Repeat steps S2-S4 until the training is completed;

[0053] S6. The image dehazing transformer performance is evaluated on the SOTS test set and the NH-HAZE dataset and compared with existing methods.

[0054] Preferably, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that when the processor executes the program, the function of an image dehazing transformer based on Legendre polynomial approximation and error compensation module is realized.

[0055] Preferably, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the functions of an image defogging transformer based on Legendre polynomial approximation and error compensation modules.

[0056] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0057] 1. Improved Computational Efficiency: By introducing the Legendre polynomial self-attention module, the computational complexity of the self-attention mechanism is significantly reduced. This innovation makes the Transformer model, or image dehazing transformer, more efficient when processing high-resolution images, reducing computing resource consumption and opening up the possibility of real-time applications.

[0058] 2. High-quality dehazing: Experimental results demonstrate that the proposed technology significantly improves the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) compared to traditional dehazing methods on multiple public datasets (such as SOTS, O-Haze, and Dense-Haze). Specifically, it achieves a PSNR of 37.32dB and an SSIM close to 1.0 on the SOTS-outdoor dataset, demonstrating excellent image quality restoration capabilities.

[0059] 3. Detail Preservation and Color Restoration: This paper utilizes an error compensation module to effectively reduce errors in the calculation and sampling processes, thereby better preserving image detail and color information while removing haze. Especially when processing complex scenes, the model can reduce artifacts and enhance image contrast and clarity.

[0060] 4. Wide Application Scenarios: The image dehazing transformer, or Transformer model, is applicable to a variety of fields, including autonomous driving, surveillance systems, and satellite image processing. Its reduced computational complexity enables efficient image dehazing in resource-constrained environments, expanding its practical application range.

[0061] 5. Model parameter optimization: The Transformer model parameters of this invention are only 2.80M, which is much lower than many existing dehazing methods. This reduces storage and deployment costs, while also speeding up model training and inference.

[0062] In summary, the defogging method of the present invention improves computational efficiency while ensuring the quality of the defogging effect and the restoration of details, and has good practical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a schematic diagram of the overall framework of the image defogging converter and its core modules of the present invention;

[0064] Figure 2 is a training flow chart of the image defogging transformer of the present invention;

[0065] Figure 3 Comparison of visual effects of different dehazing methods on the O-Haze dataset;

[0066] Figure 4 Comparison of visual effects of different dehazing methods on the NH-Haze dataset. DETAILED DESCRIPTION

[0067] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] This embodiment proposes an image dehazing transformer based on Legendre polynomial approximation and an error compensation module. The core idea is to reduce the computational complexity of the self-attention mechanism in traditional transformers by introducing Legendre polynomial approximation. Furthermore, the computation error compensation module (CECM) and sampling error compensation module (SECM) are used to reduce errors introduced during computation and sampling, thereby enhancing the image dehazing effect.

[0069] In the self-attention calculation, the traditional transformer needs to calculate the dot product between the query, key, and value and convert it into attention weight through the softmax function. The computational complexity of this operation is Where: O represents the complexity, L represents the length of the input sequence, d k Indicates the dimension size of the key feature;

[0070] In order to reduce the computational complexity, this embodiment uses the Legendre polynomial self-attention module to calculate the exponential function e in the softmax function. x Approximation is performed, thus reducing the computational complexity to This method approximates the exponential function in softmax through Legendre polynomials and uses the associativity of matrix multiplication to optimize the self-attention calculation. Specifically:

[0071] The Transformer model of this embodiment, i.e., the image dehazing transformer, is based on a hierarchical cascaded encoding-decoding structure, adopts the U-net network as the basic framework, and introduces the Legendre polynomial approximation self-attention module and the error compensation module. The overall structure is as follows Figure 1 As shown in Figure 1, it mainly consists of the following three core modules: Legendre Polynomial Self-Attention Module (Legendre-Former), Computational Error Compensation Module (CECM), and Sampling Error Compensation Module (SECM). The overall structure and processing flow of the model are as follows:

[0072] (1) Input preprocessing: First, a convolution operation is performed on the input haze image I, and a 3×3 convolution layer is used to extract the shallow visual features f0:

[0073] f0=Conv(I)

[0074] Among them, Conv() represents the convolution operation.

[0075] (2) Multi-scale feature embedding: The shallow visual feature f0 is input into the multi-scale feature extraction and embedding module to generate tokens with multi-scale receptive fields. The depth-wise separable convolution method is used to generate three matrices: query, key, and value.

[0076] (3) Coding part:

[0077] In the first layer of encoding, the shallow visual feature f0 is calculated through the Legendre-Former self-attention module to output the feature E1:

[0078] Q1,K1,V1=Patch embedding(f0)

[0079] E1=L-Former(Q1,K1,V1)

[0080] Among them, Patch embedding() represents the multi-scale feature extraction embedding module, L-Former(·) represents the Legendre polynomial self-attention module, which uses the feedforward network (FFN) proposed in the literature [1] to implement nonlinear transformation.

[0081] The i-th layer of the subsequent encoding obtains multi-scale features through downsampling operation:

[0082] E i =L-Former(Patch embedding(↓E i-1 )),i=2,3

[0083] Among them, ↓ represents the downsampling operation;

[0084] (4) Decoding part:

[0085] At the jth layer of decoding, an upsampling operation is first performed, and then the features output by each layer during the encoding process are directly transferred and fused into the decoder in combination with the Selective Kernel Feature Fusion Module (SKFF) proposed in the literature [2]:

[0086] D j =L-Former(Patchembedding(SKFF(E i ,↑D j+1 ))),j=2,1;i=2,1;

[0087] Among them, SKFF() represents the selective kernel feature fusion module, and ↑ represents the upsampling operation. In each upsampling operation, the sampling error compensation module is used to perform error compensation on the decoded output feature map to restore the structure and texture of the dehazed image. j Represents the output feature map of layer j in the corresponding decoding process;

[0088] (5) Generate dehazed image: Generate the residual image between the input haze image I and the haze-free image through a 3×3 convolutional layer; add the residual image to the original input haze image I to obtain the dehazed image J:

[0089] J=I+Conv(D1)

[0090] The application of the error compensation module is as follows:

[0091] Sampling Error Compensation Module (SECM): In each upsampling operation, the SECM module performs error compensation on the decoded visual feature map to restore the fine structure and texture of the image.

[0092] Computational Error Compensation Module (CECM): During the approximation calculation process of the Legendre polynomial self-attention module, the CECM module reduces the error introduced by the Legendre polynomial approximation calculation by compensating weights, thereby enhancing the dehazing performance of the image dehazing transformer.

[0093] Through the collaboration of these three core modules, the dehazing network of this embodiment can effectively remove haze from images and restore clear images. This network architecture not only improves dehazing performance but also reduces computational complexity, making it suitable for low-level vision tasks in practical applications.

[0094] Specifically, the Legendre polynomial approximation self-attention calculation is as follows:

[0095] In the traditional self-attention mechanism, the attention score A is calculated as follows:

[0096]

[0097] in:

[0098] represents the query matrix, L is the length of the input sequence, d k is the characteristic dimension;

[0099] represents the bond matrix;

[0100] Represents a value matrix, where d v is the dimension of the value;

[0101] It is a scaling factor used to prevent the dot product result from being too large, causing the softmax function gradient to disappear.

[0102] The softmax function converts the input vector into a probability distribution, which is defined as:

[0103]

[0104] Among them, x i and x j are the elements in vector x, and N is the dimension of the vector.

[0105] In order to optimize the calculation, this embodiment uses Legendre polynomials to approximate e x Legendre polynomial P n (x) is an orthogonal polynomial defined on the interval [-1,1], satisfying the recurrence relation:

[0106]

[0107] n represents the order of the corresponding Legendre polynomial, and x is the independent variable;

[0108] In this embodiment, the best approximation interval of the Legendre polynomial is selected as [-0.5, 0.5] to improve the calculation accuracy. x The approximate expression is:

[0109]

[0110] N is the total order of Legendre polynomials;

[0111] Among them, a n are the coefficients to be determined. By minimizing the approximation error function E(a0,a1), the values ​​of the coefficients are determined. The calculation formula is as follows:

[0112]

[0113] Among them, take N=2, n=0, 1; select a0=1.04218639, a1=1 for self-attention calculation.

[0114] Finally, after using Legendre polynomial approximation, the calculation of self-attention can be expressed as:

[0115]

[0116] Among them, A L is the attention score obtained by Legendre approximation; represents the query matrix, L is the length of the input sequence, d k It is the key feature dimension; represents the bond matrix; Represents a value matrix, where d v is the dimension of the value, T is the matrix transpose operation, M is the number of dimensions of the matrix column of K, and m is the dimension sequence number.

[0117] Specifically, the working process of the error compensation module is as follows:

[0118] Sampling Error Compensation Module (SECM): For multi-scale feature extraction, upsampling and downsampling operations are required in the encoder and decoder parts, which may cause loss of details. SECM estimates the sampling error e by s And make compensation:

[0119] e s =Conv(F-down(up(F)))

[0120] Where F is the feature map output by the Legendre polynomial self-attention module, down() and up() represent downsampling and upsampling operations respectively. The feature map F after error compensation UP for:

[0121] F UP =up(F)+up(e s )

[0122] Computational Error Compensation Module (CECM): In order to further reduce the computational error caused by Legendre polynomial approximation, CECM generates a compensation weight matrix E by learning the nonlinear mapping relationship between Q, K, and V. c :

[0123] E c =ReLU(Conv(concat(Q,K,V)))

[0124] Among them, concat() represents feature concatenation and Conv() represents convolution operation. The self-attention output after compensation is:

[0125]

[0126] in, Represents element-wise multiplication.

[0127] Preferably, the present invention also provides a training method based on the above-mentioned image defogging transformer, see Figure 2 , as follows:

[0128] Step 1: Extract training images from the RESIDE dataset.

[0129] Step 2: Perform data augmentation on the training images to increase the diversity of training.

[0130] Step 3: Input the enhanced image into the Transformer model, i.e., the image dehazing transformer, for forward propagation and calculate the L1 loss.

[0131] Step 4: Optimize the Transformer model parameters through backpropagation and update the Transformer model weights using the Adam optimizer.

[0132] Step 5: Repeat steps 2-4 until the training is completed.

[0133] Step 6: Evaluate the Transformer model performance on the SOTS test set and NH-HAZE dataset and compare with existing methods.

[0134] Specifically, this example uses the PyTorch framework for training in a Linux environment. The Adam optimizer is used for training, with an initial learning rate of 0.0002, which is gradually reduced to 1e-6 using a cosine annealing strategy. The batch size is set to 1, and training is repeated for 400 epochs. Data augmentation strategies include random cropping, rotation, scaling, and flipping to increase the diversity of the training data.

[0135] Specifically, this embodiment is trained and tested on multiple public datasets, including:

[0136] RESIDE [3] Dataset: Contains 13,990 synthetic indoor haze images, 500 indoor test images, 313,950 synthetic outdoor haze images, and 500 outdoor test images.

[0137] O-Haze[4] dataset: contains 45 pairs of haze images of real scenes.

[0138] Dense-Haze[5] dataset: contains 55 image pairs, suitable for dense haze scenes.

[0139] NH-Haze [6] Dataset: Contains 50 pairs of non-uniform haze images.

[0140] During training, the dataset samples were normalized, and the input images were resized to 256×256. The Adam optimizer was used for training, with an initial learning rate of 0.0001, a decay rate of 0.1, and a batch size of 16. To prevent overfitting, data augmentation methods were employed, including random flipping, rotation, and scaling. Training lasted for 100 epochs, and the best performance on the validation set determined the final model.

[0141] Specifically, the loss function is set as follows:

[0142] This example uses the loss function L1 reconstruction loss to optimize network performance. L1 is used to measure the difference between the dehazed image and the real image at the pixel level and is defined as:

[0143]

[0144] Among them, H, W represent the length and width of the image respectively; h, w represent the vertical and horizontal pixel position numbers of the image respectively, I dehazed Represents the dehazed image obtained by the model, I gt represents the original corresponding clear and haze-free image.

[0145] Experimental results demonstrate the superiority of this embodiment's dehazing method in both dehazing performance and computational efficiency. Comparative experiments with existing mainstream dehazing methods demonstrate that the method outperforms other methods on multiple datasets while significantly reducing the model's computational complexity and parameter count. The following are detailed experimental results and analysis.

[0146] 1. Quantitative Comparison

[0147] Table 1 lists the comparison between the proposed method and DehazeNet[7], AOD-Net[8], GFN[9], Grid-dehazenet

[10] , GCANet

[11] , FFA-Net

[12] 、MSBDN

[13] 、MDFEN

[14] AECR-Net

[15] ,MAXIM2S

[16] 、dehazeFormer

[17] and DeHamer

[18] Other methods in SOTS-outdoor [3] , O-Haze, Dense-Haze, and NH-Haze datasets. Quantitative comparison results of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) on the table. The table also includes the model parameter amount and computational complexity (MACs) of each method, demonstrating the advantages of the present invention in dehazing effect and computational efficiency. It can be seen from the table that the image dehazer, i.e., the method of the present invention, has achieved the highest PSNR and SSIM on multiple datasets, especially on the SOTS-outdoor dataset, achieving a PSNR of 37.32dB, and the SSIM indicators are all close to 1.0. In addition, the number of parameters of the model of the present invention is only 2.80M, and the computational complexity (MACs) is 32G, which is significantly lower than many mainstream methods.

[0148] 2. Qualitative Comparison

[0149] Figure 3 The paper demonstrates a visual comparison of the proposed method with other mainstream dehazing methods (such as GCANet, FFA-Net, and DeHamer) on the real-scene O-Haze dataset. The proposed method removes haze from real scenes while better maintaining image brightness and clarity.

[0150] Figure 4 The dehazing performance of the proposed method is demonstrated on the NH-Haze dataset, along with other mainstream dehazing methods (such as GCANet, FFA-Net, and DeHamer). The results show that the proposed method can significantly reduce artifacts and enhance image contrast when dealing with dense haze.

[0151] In summary, the method of the present invention can effectively remove haze from images and restore clear image details and color information, showing fewer artifacts and more natural visual effects compared with other methods.

[0152] Table 1 Quantitative comparison results of the proposed method and mainstream dehazing methods on multiple data sets

[0153]

[0154] Preferably, an embodiment of the present application further provides a specific implementation of an electronic device capable of realizing all functions of the image defogging converter based on Legendre polynomial approximation and error compensation module in the above embodiment, and the electronic device specifically includes the following contents:

[0155] Processor, memory, communications interface, and bus;

[0156] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0157] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all steps in the image defogging converter based on Legendre polynomial approximation and error compensation module in the above embodiment are implemented.

[0158] An embodiment of the present application also provides a computer-readable storage medium that can implement all the functions of the image dehazing transformer based on the Legendre polynomial approximation and error compensation module in the above-mentioned embodiment. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all the steps of the image dehazing transformer based on the Legendre polynomial approximation and error compensation module in the above-mentioned embodiment.

[0159] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the hardware + program embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0160] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0161] Although the present application provides method operation steps such as embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many steps and does not represent the only execution order. When an actual device or client product is executed, it can be executed in the order shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0162] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0165] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, those skilled in the art may make various specific modifications based on the teachings of the present invention, all of which fall within the scope of protection of the present invention.

[0166] References:

[0167] [1] SW Zamir, A. Arora, S. Khan, M. Hayat, FSKhan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp.5728–5739.

[0168] [2] SWZamir, A.Arora, S.Khan, M.Hayat, FSKhan, M.-H.Yang, and L.Shao, “Learning enriched features for real image restoration and enhancement,” inComputer Vision–ECCV 2020:16th European Conference,Glasgow,UK,August 23–28,2020,Proceedings,Part XXV 16, Springer, 2020, pp.492–511.

[0169] [3]B.Li,W.Ren,D.Fu,D.Tao,D.Feng,W.Zeng,and Z.Wang,“Benchmarkingsingle-image dehazing and beyond,”IEEE Transactions on Image Processing,vol.28,no.1,pp.492–505,2018.

[0170] [4]C.O.Ancuti,C.Ancuti,R.Timofte,and C.De Vleeschouwer,“O-HAZE:Adehazing benchmark with real hazy and haze-free outdoor images,”inProceedings of the IEEE Conference on Computer Vision and Pattern RecognitionWorkshops,2018,pp.754–762.

[0171] [5]C.O.Ancuti,C.Ancuti,M.Sbert,and R.Timofte,“Dense-HAZE:A benchmarkfor image dehazing with dense-haze and haze-free images,”in 2019IEEEInternational Conference on Image Processing(ICIP),IEEE,2019,pp.1014–1018.

[0172] [6]C.O.Ancuti,C.Ancuti,and R.Timofte,“NH-HAZE:An image dehazingbenchmark with non-homogeneous hazy and haze-free images,”in Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops,2020,pp.444–445.

[0173] [7]B.Cai,X.Xu,K.Jia,C.Qing,and D.Tao,“DehazeNet:An end-to-end systemfor single image haze removal,”IEEE Transactions on Image Processing,vol.25,no.11,pp.5187–5198,2016.

[0174] [8]B.Li,X.Peng,Z.Wang,J.Xu,and D.Feng,“AOD-Net:All-in-one dehazingnetwork,”in Proceedings of the IEEE International Conference on ComputerVision,2017,pp.4770–4778.

[0175] [9]W.Ren,L.Ma,J.Zhang,J.Pan,X.Cao,W.Liu,and M.-H.Yang,“Gated fusionnetwork for single image dehazing,”in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition,2018,pp.3253–3261.

[0176]

[10] X.Liu,Y.Ma,Z.Shi,and J.Chen,“GridDehazeNet:Attention-based multi-scale network for image dehazing,”in Proceedings of the IEEE / CVFInternational Conference on Computer Vision,2019,pp.7314–7323.

[0177]

[11] D.Chen,M.He,Q.Fan,J.Liao,L.Zhang,D.Hou,L.Yuan,and G.Hua,“Gatedcontext aggregation network for image dehazing and deraining,”in 2019 IEEEWinter Conference on Applications of Computer Vision(WACV),IEEE,2019,pp.1375–1383.

[0178]

[12] X.Qin,Z.Wang,Y.Bai,X.Xie,and H.Jia,“FFA-Net:Feature fusionattention network for single image dehazing,”in Proceedings of the AAAIConference on Artificial Intelligence,vol.34,no.07,2020,pp.11908–11915.

[0179]

[13] H.Dong,J.Pan,L.Xiang,Z.Hu,X.Zhang,F.Wang,and M.-H.Yang,“Multi-scale boosted dehazing network with dense feature fusion,”in Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition,2020,pp.2157–2167.

[0180]

[14] X.Yin,G.Tu,and Q.Chen,“Multiscale depth fusion with contextualhybrid enhancement network for image dehazing,”IEEE Transactions onInstrumentation and Measurement,vol.72,pp.1–12,2023.

[0181]

[15] H.Wu,Y.Qu,S.Lin,J.Zhou,R.Qiao,Z.Zhang,Y.Xie,and L.Ma,“Contrastivelearning for compact single image dehazing,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2021,pp.10551–10560.

[0182]

[16] Z.Tu,H.Talebi,H.Zhang,F.Yang,P.Milanfar,A.Bovik,and Y.Li,“MAXIM:Multi-axis MLP for image processing,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2022,pp.5769–5780.

[0183]

[17] Y.Song,Z.He,H.Qian,and X.Du,“Vision transformers for single imagedehazing,”IEEE Transactions on Image Processing,vol.32,pp.1927–1941,2023.

[0184]

[18] C.-L.Guo,Q.Yan,S.Anwar,R.Cong,W.Ren,and C.Li,“Image dehazingtransformer with transmission-aware 3D position embedding,”in Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition,2022,pp.5812–5820.

Claims

1. An image dehazing transformer based on Legendre polynomial approximation and error compensation module. In the self-attention calculation, the transformer calculates the dot product between the query, key, and value and converts it into attention weight through the softmax function. The computational complexity of the above operation is ,in: O represents the complexity, L represents the length of the input sequence, Indicates the dimensional size of the key feature; the image dehazing transformer includes a multi-scale feature extraction and embedding module, a Legendre polynomial self-attention module, a calculation error compensation module, and a sampling error compensation module. The multi-scale feature extraction and embedding module is used to perform multi-scale feature extraction; the Legendre polynomial self-attention module approximates the exponential function in the softmax function through Legendre polynomials to reduce the computational complexity; at the same time, the calculation error compensation module and the sampling error compensation module are used to reduce the errors introduced in the calculation and sampling processes, thereby enhancing the image dehazing effect; Approximation using Legendre polynomials ; Legendre polynomials It is defined in the interval Orthogonal polynomials on satisfy the recurrence relation: ； n represents the order of the corresponding Legendre polynomial, and x is the independent variable; The best approximation interval for the Legendre polynomial is chosen as , to improve the calculation accuracy; use Legendre polynomials to The approximate expression is: ； N is the total order of Legendre polynomials; in, is the coefficient to be determined; by minimizing the approximation error function , determine the value of each coefficient; the calculation formula is as follows: ； Among them, take N=2, n=0, 1; select , , used for self-attention calculation; Finally, after using Legendre polynomial approximation, the calculation of self-attention is expressed as: ； in, is the attention score obtained by Legendre approximation; represents the query matrix, is the length of the input sequence, It is the key feature dimension; represents the bond matrix; represents a value matrix, where is the dimension of the value; T is the matrix transpose operation, M is the number of dimensions of the matrix column of K, and m is the dimension sequence number.

2. The image dehazing transformer based on Legendre polynomial approximation and error compensation module according to claim 1, characterized in that: The Legendre polynomial self-attention module uses Legendre polynomials to adjust the exponential function in the softmax function. Approximation is performed to reduce the computational complexity to , and uses the associativity of matrix multiplication to optimize self-attention calculation.

3. The image defogging converter based on Legendre polynomial approximation and error compensation module according to claim 1, characterized in that: The processing flow of the image dehazing transformer is as follows: (1) Input preprocessing: input haze image Perform convolution operation and extract shallow visual features through a 3×3 convolution layer : ； in, Represents the convolution operation; (2) Multi-scale feature embedding: embedding shallow visual features Input the multi-scale feature extraction embedding module to generate multi-scale tokens; use the depth-separable convolution method to generate three matrices: query, key, and value; (3) In the first layer of encoding, shallow visual features The output features are calculated through the Legendre polynomial self-attention module : ； ； in, represents the multi-scale feature extraction embedding module, Represents the Legendre polynomial self-attention module, which uses the feedforward network FFN to implement nonlinear transformation; The i-th layer of the subsequent encoding obtains multi-scale features through downsampling operation: ； in, represents the downsampling operation; (4) At the jth layer of decoding, an upsampling operation is first performed, and then the features output by each layer during the encoding process are directly transferred and fused into the decoder in combination with the selective kernel feature fusion module: ； in, represents the selective kernel feature fusion module, represents an upsampling operation; wherein in each upsampling operation, the sampling error compensation module performs error compensation on the decoded output feature map to restore the structure and texture of the dehazed image; Represents the output feature map of layer j in the corresponding decoding process; (5) Generate dehazed image: Generate the input haze image through a 3×3 convolutional layer The residual image between the haze-free image and the original input haze image Add together to get the defogged image : ； During the approximation calculation process of the Legendre polynomial self-attention module, the computational error compensation module reduces the error introduced in the approximation calculation process by compensating weights, thereby enhancing the dehazing performance of the image dehazing transformer.

4. The image defogging converter based on Legendre polynomial approximation and error compensation module according to claim 1, characterized in that: The sampling error compensation module SECM estimates the sampling error by the following way And make compensation: ； in, is the feature map output by the Legendre polynomial self-attention module, and Represents downsampling and upsampling operations respectively; feature maps after error compensation for: ； The calculation error compensation module CECM is learned by 、 、 The nonlinear mapping relationship between them generates the compensation weight matrix : ； in, represents feature splicing, represents the convolution operation; the self-attention output after compensation is: ； in, Represents element-wise multiplication.

5. A training method for the image dehazing transformer according to any one of claims 1 to 4, characterized in that: include: S1. Extract the images to be trained from the RESIDE dataset; S2. Perform data augmentation on the training images; S3. Input the enhanced image to be trained into the image dehazing transformer for forward propagation and calculate the L1 loss; S4. Optimize the parameters of the image dehazing transformer by backpropagation and update the weights of the image dehazing transformer using the Adam optimizer. S5. Repeat steps S2-S4 until the training is completed; S6. The image dehazing transformer performance is evaluated on the SOTS test set and the NH-HAZE dataset and compared with existing methods.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the function of the image defogging converter based on Legendre polynomial approximation and error compensation module according to any one of claims 1 to 4 is realized.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the function of the image defogging transformer based on Legendre polynomial approximation and error compensation module as described in any one of claims 1 to 4 is realized.

Images