Image defogging converter based on Legendre polynomial approximation and error compensation module

Abstract

The invention discloses an image defogging converter based on a Legendre polynomial approximation and error compensation module, which comprises a multi-scale feature extraction embedding module, a Legendre polynomial self-attention module, a calculation error compensation module and a sampling error compensation module, and is characterized in that the multi-scale feature extraction embedding module is used for performing multi-scale feature extraction; the Legendre polynomial self-attention module approximates an exponential function in the softmax function through a Legendre polynomial so as to reduce the calculation complexity; meanwhile, errors introduced in the calculation and sampling processes are reduced through the calculation error compensation module and the sampling error compensation module, and the image defogging effect is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing, and in particular to an image defogging method based on a Transformer architecture, which is applicable to various visual measurement systems including autonomous driving, monitoring systems and satellite image processing. Background Art

[0002] Under adverse weather conditions such as haze, the captured images are often affected by the absorption and scattering of atmospheric suspended particles, resulting in reduced image contrast, blurred details and color distortion. These degradation phenomena seriously affect the performance of visual measurement systems, such as autonomous driving, monitoring, satellite imaging and other application fields. Traditional image dehazing methods based on convolutional neural networks (CNNs) can restore image quality to a certain extent, but due to the limitations of CNNs in capturing long-distance pixel relationships and multi-scale semantic information, the dehazed images will still have residual fog and the detail restoration is not ideal.

[0003] In recent years, the Transformer model has been widely used in the field of computer vision and has demonstrated strong performance in high-level visual tasks. However, when the Transformer model is directly applied to low-level tasks at the pixel level (such as image dehazing), there are problems with high computational complexity and insufficient local information extraction capabilities. Therefore, how to reduce the computational complexity of the Transformer model while ensuring the dehazing effect and improve the ability to restore details is a problem that needs to be solved urgently in the existing technology. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an image defogging transformer based on Legendre polynomial approximation and error compensation to solve the problems of high computational complexity and insufficient detail restoration in the prior art. The present invention reduces the computational complexity of the self-attention mechanism by introducing the Legendre polynomial approximation method, and corrects the errors in the approximate calculation and sampling process through the error compensation module, thereby achieving efficient and high-quality image defogging.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] An image dehazing transformer based on Legendre polynomial approximation and error compensation module. In the self-attention calculation, the transformer calculates the dot product between the query, key, and value and converts it into attention weight through the softmax function. The computational complexity of the above operation is Where: O represents the complexity, L represents the length of the input sequence, d kIndicates the dimensional size of the key feature; the image dehazing transformer includes a multi-scale feature extraction embedding module, a Legendre polynomial self-attention module, a calculation error compensation module and a sampling error compensation module. The multi-scale feature extraction embedding module is used for multi-scale feature extraction; the Legendre polynomial self-attention module approximates the exponential function in the softmax function through the Legendre polynomial to reduce the computational complexity; at the same time, the calculation error compensation module and the sampling error compensation module are used to reduce the errors introduced in the calculation and sampling processes, thereby enhancing the image dehazing effect.

[0007] Furthermore, the Legendre polynomial self-attention module uses Legendre polynomials to adjust the exponential function e in the softmax function. x Approximation is made to reduce the computational complexity to And use the associativity of matrix multiplication to optimize self-attention calculation.

[0008] Furthermore, the processing flow of the image dehazing transformer is as follows:

[0009] (1) Input preprocessing: Perform a convolution operation on the input haze image I and extract shallow visual features f through a 3×3 convolution layer 0 :

[0010] f 0 =Conv(I)

[0011] Among them, Conv() represents the convolution operation;

[0012] (2) Multi-scale feature embedding: embedding shallow visual features f 0 Input the multi-scale feature extraction embedding module to generate multi-scale tokens; use the deep separable convolution method to generate three matrices of query, key and value;

[0013] (3) In the first layer of encoding, the shallow visual feature f 0 The output feature E is calculated through the Legendre polynomial self-attention module 1 :

[0014] Q 1 ,K 1 ,V 1 =Patch embedding(f 0 )

[0015] E 1 =L-Former(Q 1 ,K 1 ,V 1 )

[0016] Among them, Patch embedding() represents the multi-scale feature extraction embedding module, L-Former(·) represents the Legendre polynomial self-attention module, in which the feedforward network FFN is used to realize nonlinear transformation;

[0017] The i-th layer of the subsequent encoding obtains multi-scale features through downsampling operations:

[0018] E i =L-Former(Patch embedding(↓E i-1 )),i=2,3

[0019] Among them, ↓ represents the downsampling operation;

[0020] (4) At the jth layer of decoding, an upsampling operation is first performed, and then combined with the selective kernel feature fusion module, the features output by each layer during the encoding process are directly transferred and fused to the decoder:

[0021] D j =L-Former(Patchembedding(SKFF(E i ,↑D j+1 ))),j=2,1;i=2,1;

[0022] Wherein, SKFF() represents the selective kernel feature fusion module, and ↑ represents the upsampling operation; in each upsampling operation, the error compensation module is used to compensate the decoded output feature map to restore the structure and texture of the dehazed image; D j Represents the output feature map of layer j in the corresponding decoding process;

[0023] (5) Generate a dehazed image: Generate a residual image between the input haze image I and the haze-free image through a 3×3 convolutional layer; add the residual image to the original input haze image I to obtain the dehazed image J:

[0024] J=I+Conv(D 1 );

[0025] In the approximate calculation process of the Legendre polynomial self-attention module, the calculation error compensation module reduces the error introduced in the approximate calculation process by compensating weights, thereby enhancing the defogging performance of the image defogging transformer.

[0026] Furthermore, in the Legendre polynomial self-attention module, Legendre polynomials are used to approximate e x ; Legendre polynomial P n (x) is an orthogonal polynomial defined on the interval [-1,1], satisfying the recurrence relation:

[0027]

[0028] n represents the order of the corresponding Legendre polynomial, and x is the independent variable;

[0029] The best approximation interval of the Legendre polynomial is selected as [-0.5, 0.5] to improve the calculation accuracy; the Legendre polynomial is used to approximate e x The approximate expression is:

[0030]

[0031] N is the total order of Legendre polynomials;

[0032] Among them, a n is the coefficient to be determined; by minimizing the approximation error function E(a 0 ,a 1 ), determine the value of each coefficient; the calculation formula is as follows:

[0033]

[0034] Among them, take N = 2, n = 0, 1; select a 0 =1.04218639, a 1 =1, used for self-attention calculation;

[0035] Finally, after using Legendre polynomial approximation, the calculation of self-attention is expressed as:

[0036]

[0037] Among them, A L is the attention score obtained by Legendre approximation; represents the query matrix, L is the length of the input sequence, d k It is the key feature dimension; represents the bond matrix; represents a value matrix, where d v is the dimension of the value; T is the matrix transpose operation, M is the number of dimensions of the matrix column of K, and m is the dimension sequence number.

[0038] Furthermore, the sampling error compensation module SECM estimates the sampling error e by: s And make compensation:

[0039] e s =Conv(F-down(up(F)))

[0040] Among them, F is the feature map output by the Legendre polynomial self-attention module, down() and up() represent downsampling and upsampling operations respectively; the feature map F after error compensation UP for:

[0041] F UP =up(F)+up(e s )

[0042] The computational error compensation module CECM generates the compensation weight matrix E by learning the nonlinear mapping relationship between Q, K, and V. c :

[0043] E c =ReLU(Conv(concat(Q,K,V)))

[0044] Among them, concat() represents feature concatenation, Conv() represents convolution operation; the self-attention output after compensation is:

[0045]

[0046] in, Represents element-wise multiplication.

[0047] Preferably, the present invention also provides a training method based on the above-mentioned image defogging transformer, comprising:

[0048] S1. Extract the images to be trained from the RESIDE dataset;

[0049] S2. Perform data enhancement processing on the training images;

[0050] S3. Input the enhanced image to be trained into the image dehazing transformer for forward propagation and calculate the L1 loss;

[0051] S4. Optimize the parameters of the image dehazing transformer by back propagation, and update the weights of the image dehazing transformer using the Adam optimizer;

[0052] S5. Repeat steps S2-S4 until the training is completed;

[0053] S6. The image dehazing transformer performance is evaluated on the SOTS test set and the NH-HAZE dataset and compared with existing methods.

[0054] Preferably, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the function of an image dehazing transformer based on Legendre polynomial approximation and error compensation module when executing the program.

[0055] Preferably, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the functions of an image defogging transformer based on Legendre polynomial approximation and an error compensation module.

[0056] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0057] 1. Improved computational efficiency: By introducing the Legendre polynomial self-attention module, the computational complexity of the self-attention mechanism is significantly reduced. This innovation makes the Transformer model, i.e., the image dehazing transformer, more efficient in processing high-resolution images, reduces the consumption of computational resources, and provides the possibility for real-time applications.

[0058] 2. High-quality dehazing effect: Experimental results show that the technical solution of the present invention can significantly improve the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) on multiple public data sets (such as SOTS, O-Haze, Dense-Haze, etc.) compared with traditional dehazing methods. Specifically, it achieves a PSNR of 37.32dB on the SOTS-outdoor data set, and the SSIM is close to 1.0, demonstrating excellent image quality restoration capabilities.

[0059] 3. Detail preservation and color restoration: The present invention effectively reduces the errors in the calculation and sampling process through the error compensation module, thereby better preserving the details and color information of the image while removing the fog. Especially when processing complex scenes, the model can reduce artifacts and enhance the contrast and clarity of the image.

[0060] 4. Applicable to a wide range of application scenarios: The image defogging transformer of the present invention, namely the Transformer model, is applicable to multiple fields such as autonomous driving, monitoring systems, and satellite image processing. Due to its reduced computational complexity, it can still achieve efficient image defogging processing in resource-constrained environments, thus expanding the scope of its practical application.

[0061] 5. Model parameter optimization: The Transformer model parameters of the present invention are only 2.80M, which is much lower than many existing dehazing methods, reducing storage and deployment costs, while also speeding up model training and reasoning.

[0062] In summary, the defogging method of the present invention improves the computational efficiency while ensuring the quality of the defogging effect and the restoration of details, and has good practical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the overall framework of the image defogging converter and each core module of the present invention;

[0064] Figure 2 is a training flow chart of the image defogging transformer of the present invention;

[0065] Figure 3It is a comparison of the visual effects of different dehazing methods on the O-Haze dataset;

[0066] Figure 4 It is a visual comparison of different dehazing methods on the NH-Haze dataset. DETAILED DESCRIPTION

[0067] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0068] This embodiment proposes an image dehazing transformer based on Legendre polynomial approximation and error compensation module. The core idea is to reduce the computational complexity of the self-attention mechanism in the traditional transformer by introducing Legendre polynomial approximation, and reduce the errors introduced by the calculation and sampling process by using the calculation error compensation module (CECM) and the sampling error compensation module (SECM), thereby enhancing the image dehazing effect.

[0069] In the self-attention calculation, the traditional transformer needs to calculate the dot product between the query, key, and value and convert it into attention weight through the softmax function. The computational complexity of this operation is Where: O represents the complexity, L represents the length of the input sequence, d k Indicates the dimension size of the key feature;

[0070] In order to reduce the computational complexity, this embodiment uses the Legendre polynomial self-attention module to calculate the exponential function e in the softmax function. x Approximation is made, thus reducing the computational complexity to This method approximates the exponential function in softmax through Legendre polynomials and uses the associativity of matrix multiplication to optimize the self-attention calculation. Specifically:

[0071] The Transformer model of this embodiment, i.e., the image defogging transformer, is based on a hierarchical cascaded encoding-decoding structure, uses a U-net network as the basic framework, and introduces a Legendre polynomial approximation self-attention module and an error compensation module. The overall structure is as follows Figure 1 As shown in the figure, it mainly consists of the following three core modules: Legendre-Former, CECM and SECM. The overall structure and processing flow of the model are as follows:

[0072] (1) Input preprocessing: First, a convolution operation is performed on the input haze image I, and a 3×3 convolution layer is used to extract shallow visual features f 0 :

[0073] f 0 =Conv(I)

[0074] Among them, Conv() represents the convolution operation.

[0075] (2) Multi-scale feature embedding: embedding shallow visual features f 0 Input the multi-scale feature extraction embedding module to generate tokens with multi-scale receptive fields. Use the depth-wise separable convolution method to generate three matrices: query, key, and value.

[0076] (3) Coding part:

[0077] In the first encoding layer, the shallow visual features f 0 The output feature E is calculated through the Legendre-Former self-attention module 1 :

[0078] Q 1 ,K 1 ,V 1 =Patch embedding(f 0 )

[0079] E 1 =L-Former(Q 1 ,K 1 ,V 1 )

[0080] Among them, Patch embedding() represents the multi-scale feature extraction embedding module, L-Former(·) represents the Legendre polynomial self-attention module, which uses the feed-forward network (FFN) proposed in the literature [1] to achieve nonlinear transformation.

[0081] The i-th layer of the subsequent encoding obtains multi-scale features through downsampling operations:

[0082] E i =L-Former(Patch embedding(↓E i-1 )),i=2,3

[0083] Among them, ↓ represents the downsampling operation;

[0084] (4) Decoding part:

[0085] At the jth layer of decoding, an upsampling operation is first performed, and then the features output by each layer in the encoding process are directly transferred and fused to the decoder in combination with the Selective Kernel Feature Fusion Module (SKFF) proposed in the literature [2]:

[0086] D j =L-Former(Patchembedding(SKFF(E i ,↑D j+1 ))),j=2,1;i=2,1;

[0087] Wherein, SKFF() represents the selective kernel feature fusion module, and ↑ represents the upsampling operation; in each upsampling operation, the error compensation module is used to compensate the decoded output feature map to restore the structure and texture of the dehazed image; D j Represents the output feature map of layer j in the corresponding decoding process;

[0088] (5) Generate a dehazed image: Generate a residual image between the input haze image I and the haze-free image through a 3×3 convolutional layer; add the residual image to the original input haze image I to obtain the dehazed image J:

[0089] J=I+Conv(D 1 )

[0090] The application of error compensation module is as follows:

[0091] Sampling Error Compensation Module (SECM): In each upsampling operation, the SECM module performs error compensation on the decoded visual feature map to restore the fine structure and texture of the image.

[0092] Computational Error Compensation Module (CECM): During the approximation calculation process of the Legendre polynomial self-attention module, the CECM module reduces the error introduced by the Legendre polynomial approximation calculation by compensating weights, thereby enhancing the dehazing performance of the image dehazing transformer.

[0093] Through the collaboration of these three core modules, the defogging network of this embodiment can effectively remove the haze in the image and restore a clear image result. This network architecture not only improves the defogging performance, but also reduces the computational complexity, making it suitable for low-level visual tasks in practical applications.

[0094] Specifically, the Legendre polynomial approximation self-attention calculation is as follows:

[0095] In the traditional self-attention mechanism, the attention score A is calculated as follows:

[0096]

[0097] in:

[0098] represents the query matrix, L is the length of the input sequence, d k is the characteristic dimension;

[0099] represents the bond matrix;

[0100] represents a value matrix, where d v is the dimension of the value;

[0101] It is a scaling factor used to prevent the dot product result from being too large, causing the softmax function gradient to disappear.

[0102] The softmax function converts the input vector into a probability distribution, which is defined as:

[0103]

[0104] Among them, x i and x j are the elements in vector x, and N is the dimension of the vector.

[0105] In order to optimize the calculation, this embodiment uses Legendre polynomials to approximate e x Legendre polynomial P n (x) is an orthogonal polynomial defined on the interval [-1,1], satisfying the recurrence relation:

[0106]

[0107] n represents the order of the corresponding Legendre polynomial, and x is the independent variable;

[0108] In this embodiment, the best approximation interval of the Legendre polynomial is selected as [-0.5, 0.5] to improve the calculation accuracy. x The approximate expression is:

[0109]

[0110] N is the total order of Legendre polynomials;

[0111] Among them, a n is the coefficient to be determined. By minimizing the approximation error function E(a 0 ,a 1 ), determine the value of each coefficient. The calculation formula is as follows:

[0112]

[0113] Among them, take N = 2, n = 0, 1; select a0 =1.04218639, a 1 =1, used for self-attention calculation.

[0114] Finally, after using Legendre polynomial approximation, the calculation of self-attention can be expressed as:

[0115]

[0116] Among them, A L is the attention score obtained by Legendre approximation; represents the query matrix, L is the length of the input sequence, d k It is the key feature dimension; represents the bond matrix; represents a value matrix, where d v is the dimension of the value, T is the matrix transpose operation, M is the number of dimensions of the matrix column of K, and m is the dimension sequence number.

[0117] Specifically, the working process of the error compensation module is as follows:

[0118] Sampling Error Compensation Module (SECM): For multi-scale feature extraction, upsampling and downsampling operations are required in the encoder and decoder parts, which may cause loss of details. SECM estimates the sampling error e by s And make compensation:

[0119] e s =Conv(F-down(up(F)))

[0120] Where F is the feature map output by the Legendre polynomial self-attention module, down() and up() represent downsampling and upsampling operations respectively. The feature map F after error compensation UP for:

[0121] F UP =up(F)+up(e s )

[0122] Computational Error Compensation Module (CECM): In order to further reduce the computational error caused by Legendre polynomial approximation, CECM generates a compensation weight matrix E by learning the nonlinear mapping relationship between Q, K, and V. c :

[0123] E c =ReLU(Conv(concat(Q,K,V)))

[0124] Among them, concat() represents feature concatenation, and Conv() represents convolution operation. The self-attention output after compensation is:

[0125]

[0126] in, Represents element-wise multiplication.

[0127] Preferably, the present invention also provides a training method based on the above-mentioned image defogging transformer, see Figure 2 , as follows:

[0128] Step 1: Extract training images from the RESIDE dataset.

[0129] Step 2: Perform data augmentation on the training images to increase the diversity of training.

[0130] Step 3: Input the enhanced image into the Transformer model, i.e., the image dehazing transformer, for forward propagation and calculate the L1 loss.

[0131] Step 4: Optimize the Transformer model parameters through back-propagation and update the Transformer model weights using the Adam optimizer.

[0132] Step 5: Repeat steps 2-4 until the training is completed.

[0133] Step 6: Evaluate the Transformer model performance on the SOTS test set and NH-HAZE dataset and compare it with existing methods.

[0134] Specifically, this embodiment uses the PyTorch framework for training in a Linux environment. The training uses the Adam optimizer, with an initial learning rate of 0.0002, which is gradually reduced to 1e-6 through a cosine annealing strategy. The batch size is set to 1, and the training iteration is 400 rounds. The data enhancement strategy includes random cropping, rotation, scaling, and flipping to increase the diversity of the training data.

[0135] Specifically, this embodiment is trained and tested on multiple public data sets, including:

[0136] RESIDE [3] Dataset: Contains 13,990 synthetic indoor haze images, 500 indoor test images, 313,950 synthetic outdoor haze images, and 500 outdoor test images.

[0137] O-Haze[4] dataset: contains 45 pairs of haze images of real scenes.

[0138] Dense-Haze[5] dataset: contains 55 image pairs and is suitable for dense haze scenes.

[0139] NH-Haze [6]Dataset: Contains 50 pairs of non-uniform haze images.

[0140] During the training process, the dataset samples used were standardized, and the size of the input images was unified to 256×256. The Adam optimizer was used in the training process, with an initial learning rate of 0.0001, a decay rate of 0.1, and a batch size of 16. In order to prevent overfitting, data enhancement methods were used, including random flipping, rotation, and scaling. The training lasted for 100 epochs, and the best performance on the validation set determined the final model.

[0141] Specifically, the loss function is set as follows:

[0142] This embodiment uses the loss function L 1 Reconstruction loss to optimize network performance, L 1 It is used to measure the difference between the dehazed image and the real image at the pixel level and is defined as:

[0143]

[0144] Where H, W represent the image length and width respectively; h, w represent the pixel position number in the vertical and horizontal directions of the image respectively. dehazed represents the dehazed image obtained by the model, I gt Represents the original corresponding clear and haze-free image.

[0145] The defogging method of this embodiment has been verified by experimental results to be superior in defogging performance and computational efficiency. Comparative experiments with existing mainstream defogging methods show that the defogging effect of the method of the present invention on multiple data sets is better than that of other methods, while significantly reducing the computational complexity and parameter amount of the model. The following are specific experimental results and analysis.

[0146] 1. Quantitative Comparison

[0147] Table 1 lists the performance of the proposed method compared with DehazeNet[7], AOD-Net[8], GFN[9], Grid-dehazenet

[10] , GCANet

[11] , and FFA-Net

[12] 、MSBDN

[13] 、MDFEN

[14] AECR-Net

[15] ,MAXIM2S

[16] 、dehazeFormer

[17] and DeHamer

[18] Other methods in SOTS-outdoor [3], O-Haze, Dense-Haze, and NH-Haze datasets. The table also includes the model parameter quantity and computational complexity (MACs) of each method, demonstrating the advantages of the present invention in dehazing effect and computational efficiency. It can be seen from the table that the image dehazer, i.e., the method of the present invention, has achieved the highest PSNR and SSIM on multiple datasets, especially on the SOTS-outdoor dataset, achieving a PSNR of 37.32 dB, and the SSIM index is close to 1.0. In addition, the number of parameters of the model of the present invention is only 2.80M, and the computational complexity (MACs) is 32G, which is significantly lower than many mainstream methods.

[0148] 2. Qualitative Comparison

[0149] Figure 3 The visual effect comparison between the proposed method and other mainstream dehazing methods (such as GCANet, FFA-Net, DeHamer, etc.) on the real scene O-Haze dataset is demonstrated. The proposed method can better maintain the brightness and clarity of the image while removing the haze in the real scene.

[0150] Figure 4 The dehazing effect of the method of the present invention and other mainstream dehazing methods (such as GCANet, FFA-Net, DeHamer, etc.) is demonstrated on the NH-Haze dataset. The results show that the method of the present invention can significantly reduce artifacts and enhance the contrast of images when dealing with dense haze.

[0151] In summary, the method of the present invention can effectively remove haze from images and restore clear image details and color information, showing fewer artifacts and more natural visual effects compared with other methods.

[0152] Table 1 Quantitative comparison results of the proposed method and mainstream dehazing methods on multiple data sets

[0153]

[0154] Preferably, the embodiment of the present application also provides a specific implementation of an electronic device capable of realizing all functions of the image defogging converter based on Legendre polynomial approximation and error compensation module in the above embodiment, and the electronic device specifically includes the following contents:

[0155] Processor, memory, communications interface and bus;

[0156] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0157] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the image defogging transformer based on Legendre polynomial approximation and error compensation module in the above embodiment are implemented.

[0158] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all functions of the image dehazing transformer based on Legendre polynomial approximation and error compensation module in the above-mentioned embodiment. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, all steps of the image dehazing transformer based on Legendre polynomial approximation and error compensation module in the above-mentioned embodiment are implemented.

[0159] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0161] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0162] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

[0166] References:

[0167] [1] SW Zamir, A. Arora, S. Khan, M. Hayat, FSKhan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp.5728–5739.

[0168] [2]S.W.Zamir,A.Arora,S.Khan,M.Hayat,F.S.Khan,M.-H.Yang,and L.Shao,“Learning enriched features for real image restoration and enhancement,”inComputer Vision–ECCV 2020:16th European Conference,Glasgow,UK,August 23–28,2020,Proceedings,Part XXV 16,Springer,2020,pp.492–511.

[0169] [3]B.Li,W.Ren,D.Fu,D.Tao,D.Feng,W.Zeng,and Z.Wang,“Benchmarkingsingle-image dehazing and beyond,”IEEE Transactions on Image Processing,vol.28,no.1,pp.492–505,2018.

[0170] [4]C.O.Ancuti,C.Ancuti,R.Timofte,and C.De Vleeschouwer,“O-HAZE:Adehazing benchmark with real hazy and haze-free outdoor images,”inProceedings of the IEEE Conference on Computer Vision and Pattern RecognitionWorkshops,2018,pp.754–762.

[0171] [5]C.O.Ancuti,C.Ancuti,M.Sbert,and R.Timofte,“Dense-HAZE:A benchmarkfor image dehazing with dense-haze and haze-free images,”in 2019IEEEInternational Conference on Image Processing(ICIP),IEEE,2019,pp.1014–1018.

[0172] [6]C.O.Ancuti,C.Ancuti,and R.Timofte,“NH-HAZE:An image dehazingbenchmark with non-homogeneous hazy and haze-free images,”in Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops,2020,pp.444–445.

[0173] [7]B.Cai,X.Xu,K.Jia,C.Qing,and D.Tao,“DehazeNet:An end-to-end systemfor single image haze removal,”IEEE Transactions on Image Processing,vol.25,no.11,pp.5187–5198,2016.

[0174] [8]B.Li,X.Peng,Z.Wang,J.Xu,and D.Feng,“AOD-Net:All-in-one dehazingnetwork,”in Proceedings of the IEEE International Conference on ComputerVision,2017,pp.4770–4778.

[0175] [9]W.Ren,L.Ma,J.Zhang,J.Pan,X.Cao,W.Liu,and M.-H.Yang,“Gated fusionnetwork for single image dehazing,”in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition,2018,pp.3253–3261.

[0176]

[10] X.Liu,Y.Ma,Z.Shi,and J.Chen,“GridDehazeNet:Attention-based multi-scale network for image dehazing,”in Proceedings of the IEEE / CVFInternational Conference on Computer Vision,2019,pp.7314–7323.

[0177]

[11] D.Chen,M.He,Q.Fan,J.Liao,L.Zhang,D.Hou,L.Yuan,and G.Hua,“Gatedcontext aggregation network for image dehazing and deraining,”in 2019 IEEEWinter Conference on Applications of Computer Vision(WACV),IEEE,2019,pp.1375–1383.

[0178]

[12] X.Qin,Z.Wang,Y.Bai,X.Xie,and H.Jia,“FFA-Net:Feature fusionattention network for single image dehazing,”in Proceedings of the AAAIConference on Artificial Intelligence,vol.34,no.07,2020,pp.11908–11915.

[0179]

[13] H.Dong,J.Pan,L.Xiang,Z.Hu,X.Zhang,F.Wang,and M.-H.Yang,“Multi-scale boosted dehazing network with dense feature fusion,”in Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition,2020,pp.2157–2167.

[0180]

[14] X.Yin,G.Tu,and Q.Chen,“Multiscale depth fusion with contextualhybrid enhancement network for image dehazing,”IEEE Transactions onInstrumentation and Measurement,vol.72,pp.1–12,2023.

[0181]

[15] H.Wu,Y.Qu,S.Lin,J.Zhou,R.Qiao,Z.Zhang,Y.Xie,and L.Ma,“Contrastivelearning for compact single image dehazing,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2021,pp.10551–10560.

[0182]

[16] Z.Tu,H.Talebi,H.Zhang,F.Yang,P.Milanfar,A.Bovik,and Y.Li,“MAXIM:Multi-axis MLP for image processing,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2022,pp.5769–5780.

[0183]

[17] Y.Song,Z.He,H.Qian,and X.Du,“Vision transformers for single imagedehazing,”IEEE Transactions on Image Processing,vol.32,pp.1927–1941,2023.

[0184]

[18] C.-L.Guo,Q.Yan,S.Anwar,R.Cong,W.Ren,and C.Li,“Image dehazingtransformer with transmission-aware 3D position embedding,”in Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition,2022,pp.5812–5820.

Claims

1. An image dehazing transformer based on Legendre polynomial approximation and error compensation module. In the self-attention calculation, the transformer calculates the dot product between the query, key, and value and converts it into attention weight through the softmax function. The computational complexity of the above operation is in: O represents the complexity, L represents the length of the input sequence, and d k Indicates the dimensional size of the key feature; it is characterized in that the image dehazing transformer includes a multi-scale feature extraction embedding module, a Legendre polynomial self-attention module, a calculation error compensation module and a sampling error compensation module, the multi-scale feature extraction embedding module is used for multi-scale feature extraction; the Legendre polynomial self-attention module approximates the exponential function in the softmax function through the Legendre polynomial to reduce the calculation complexity; at the same time, the calculation error compensation module and the sampling error compensation module are used to reduce the errors introduced by the calculation and sampling processes, thereby enhancing the image dehazing effect.

2. The image defogging transformer based on Legendre polynomial approximation and error compensation module according to claim 1, characterized in that: The Legendre polynomial self-attention module uses the Legendre polynomial to adjust the exponential function e in the softmax function. x Approximation is made to reduce the computational complexity to And use the associativity of matrix multiplication to optimize self-attention calculation.

3. The image defogging transformer based on Legendre polynomial approximation and error compensation module according to claim 1, characterized in that: The processing flow of the image dehazing transformer is as follows: (1) Input preprocessing: Perform a convolution operation on the input haze image I and extract shallow visual features f0 through a 3×3 convolution layer: f0=Conv(I) Among them, Conv() represents the convolution operation; (2) Multi-scale feature embedding: The shallow visual feature f0 is input into the multi-scale feature extraction and embedding module to generate multi-scale tokens; the deep separable convolution method is used to generate three matrices: query, key, and value; (3) In the first layer of encoding, the shallow visual feature f0 is calculated through the Legendre polynomial self-attention module to output the feature E1: Q1, K1, V1=Patch embedding(f0) E1=L-Former(Q1,K1,V1) Among them, Patch embedding() represents the multi-scale feature extraction embedding module, L-Former() represents the Legendre polynomial self-attention module, in which the feedforward network FFN is used to realize nonlinear transformation; The i-th layer of the subsequent encoding obtains multi-scale features through downsampling operations: E i =L-Former(Patch embedding(↓E i-1 )),i=2,3 Among them, ↓ represents the downsampling operation; (4) At the jth layer of decoding, an upsampling operation is first performed, and then combined with the selective kernel feature fusion module, the features output by each layer during the encoding process are directly transferred and fused to the decoder: D j =L-Former(Patch embedding(SKFF(E i ,↑D j+1 ))),j=2,1;i=2,1; Wherein, SKFF() represents the selective kernel feature fusion module, and ↑ represents the upsampling operation; in each upsampling operation, the error compensation module is used to compensate the decoded output feature map to restore the structure and texture of the dehazed image; D j Represents the output feature map of layer j in the corresponding decoding process; (5) Generate a dehazed image: Generate a residual image between the input haze image I and the haze-free image through a 3×3 convolutional layer; add the residual image to the original input haze image I to obtain the dehazed image J: J = I + Conv(D1); In the approximate calculation process of the Legendre polynomial self-attention module, the calculation error compensation module reduces the error introduced in the approximate calculation process by compensating weights, thereby enhancing the defogging performance of the image defogging transformer.

4. The image defogging transformer based on Legendre polynomial approximation and error compensation module according to claim 1, characterized in that: In the Legendre polynomial self-attention module, Legendre polynomials are used to approximate e x ; Legendre polynomial P n (x) is an orthogonal polynomial defined on the interval [-1,1], satisfying the recurrence relation: n represents the order of the corresponding Legendre polynomial, and x is the independent variable; The best approximation interval of the Legendre polynomial is selected as [-0.5, 0.5] to improve the calculation accuracy; the Legendre polynomial is used to approximate e x The approximate expression is: N is the total order of Legendre polynomials; Among them, a n is the coefficient to be determined; by minimizing the approximation error function E(a0,a1), the value of each coefficient is determined; the calculation formula is as follows: Among them, take N = 2, n = 0, 1; select a0 = 1.04218639, a1 = 1 for self-attention calculation; Finally, after using Legendre polynomial approximation, the calculation of self-attention is expressed as: Among them, A L is the attention score obtained by Legendre approximation; represents the query matrix, L is the length of the input sequence, d k It is the key feature dimension; represents the bond matrix; represents a value matrix, where d v is the dimension of the value; T is the matrix transpose operation, M is the number of dimensions of the matrix column of K, and m is the dimension sequence number.

5. The image defogging transformer based on Legendre polynomial approximation and error compensation module according to claim 3, characterized in that: The sampling error compensation module SECM estimates the sampling error e by the following method s And make compensation: e s =Conv(F-down(up(F))) Among them, F is the feature map output by the Legendre polynomial self-attention module, down() and up() represent downsampling and upsampling operations respectively; the feature map F after error compensation UP for: F UP =up(F)+up(e s ) The computational error compensation module CECM generates the compensation weight matrix E by learning the nonlinear mapping relationship between Q, K, and V. c : It is c =ReLU(Conv(concat(Q,K,V))) Among them, concat() represents feature concatenation, Conv() represents convolution operation; the self-attention output after compensation is: in, Represents element-wise multiplication.

6. A training method for the image defogging transformer according to any one of claims 2 to 5, characterized in that: include: S1. Extract the images to be trained from the RESIDE dataset; S2. Perform data enhancement processing on the training images; S3. Input the enhanced image to be trained into the image dehazing transformer for forward propagation and calculate the L1 loss; S4. Optimize the parameters of the image dehazing transformer by back propagation, and update the weights of the image dehazing transformer using the Adam optimizer; S5. Repeat steps S2-S4 until the training is completed; S6. The image dehazing transformer performance is evaluated on the SOTS test set and the NH-HAZE dataset and compared with existing methods.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the function of the image defogging transformer based on Legendre polynomial approximation and error compensation module as claimed in any one of claims 1 to 5 is realized.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the function of an image defogging transformer based on Legendre polynomial approximation and error compensation module is realized.

Images