Transform image defogging system with accurate color rendition effect

Abstract

The invention designs an image defogging system with an accurate color rendition effect based on a Transform network. The method comprises the following steps: designing a deep learning network comprising five levels of Transform modules; an end-to-end image defogging method is adopted, and the medium transmissivity t (x) and the atmospheric light value A in the atmospheric scattering model are estimated at the same time through an estimation parameter K (x) so as to reduce estimation errors; the network is trained on the RESIDE data set by using a Charbonnier loss function; carrying out block embedding on the fog-containing image to obtain a feature vector; transform comprises layer normalization, improved multi-head attention calculation with extra convolution on a Value vector, a nonlinear activation function, a linear layer and a full connection layer; and an output defogged image is obtained by reversely deducing an atmospheric scattering model and carrying out image reconstruction of feature fusion. The designed image defogging system is high in color rendition accuracy, good in defogging quantitative index and human eye observation effect and capable of being applied to advanced auxiliary driving and urban road condition monitoring, and effective image defogging is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of computer image processing, and in particular relates to a Transformer image defogging system with accurate color restoration effect. Background Art

[0002] Images are the primary source of human perception of the outside world. Nowadays, with the rapid development of computer vision technology, digital images have been widely used in life and production. However, images are often degraded to varying degrees due to weather and other reasons when they are acquired, among which haze has become the main influencing factor. Outdoor vision systems such as smart transportation facilities, autonomous driving, and smart security cameras are easily affected by haze weather when acquiring images. Degraded images in foggy weather not only affect people's perception, but also bring severe challenges to advanced computer vision tasks. Since image quality largely determines the final performance of technical models such as target detection and tracking, dehazing images to improve the adaptability of related technologies to adverse weather environments is an important research topic at present.

[0003] Dehazing images is a classic research topic in the field of image processing at home and abroad. From the technical perspective, dehazing algorithms can be divided into algorithms based on image enhancement, algorithms based on prior knowledge, and algorithms based on deep learning.

[0004] The dehazing method based on image enhancement utilizes the different characteristics of different information frequency domain or spatial domain signals in the image, and only considers the low contrast and low saturation characteristics of haze images for processing. Kim et al. [1] used histogram equalization with local sub-block overlap to achieve local contrast enhancement of hazy images; Land et al. [2] proposed the color constancy theory and believed that the image presented at a certain point is determined by the product of the incident light component and the reflected light component, and transformed the dehazing problem into obtaining the incident light component of the object; the Retinex algorithm [3] separates the low-frequency component and the high-frequency component of the image, then removes the low-frequency component, adjusts the dynamic range of the high-frequency component, and amplifies the high-frequency component to achieve image detail optimization. The image enhancement dehazing algorithm does not consider the reasons for image degradation. Image information will be lost or over-enhancement will occur during the processing, which has great limitations.

[0005] Prior-based methods require artificially presetting some parameters or constraints based on life experience or a large amount of data statistics, and then deriving a fog-free image based on the physical model of atmospheric scattering. Tan et al. [4] proposed a method to achieve defogging by improving the local contrast of the image based on the two characteristics that fog-free images have higher contrast and sharpness than foggy images and the trend of atmospheric light changes tends to be smoother as the depth of field increases. He et al. [5] proposed a dark channel prior; Berman et al. [6] proposed a fog line prior. Since algorithms based on priors rarely consider the impact of image depth in foggy weather, they perform poorly in areas such as the sky, areas with a deep depth of field, areas with dense fog, and other areas with high brightness.

[0006] With the continuous development of deep learning, researchers have used convolutional neural networks (CNNs) for parameter estimation in atmospheric scattering models, which has improved the accuracy of parameter estimation and optimized the dehazing effect. Cai et al. [7] proposed the DehazeNet method to directly learn and estimate the mapping relationship between blurred image patches and their transmission maps; Li et al. [8] proposed AOD-Net, which integrates atmospheric light and transmission coefficients to reduce the error in parameter estimation and introduces the adaptive optimal estimation theory into the deep learning framework; FFA-Net [9] introduces a feature attention module to selectively treat different features and improve dehazing performance. CNN-based algorithms use more convolutions. Due to their limited receptive field, the network has poor ability to capture global information. The continuous evolution of generative adversarial networks (GANs) has spawned a series of dehazing algorithms based on them. Zheng Yutong et al.

[10] proposed a mixed sample image dehazing algorithm based on latent space transformation; Huang Shuying et al.

[11] extracted features of different scales by constructing a multi-scale global feature extraction network and a local feature extraction network based on GAN, and used a feature fusion network to improve the dehazing performance of the model through multi-loss supervision. GAN-based methods have shown certain capabilities in the field of image dehazing, but due to the difficulty of network model training, the large number of hyperparameters and complex configuration, it is difficult to ensure the stability of training.

[0007] Transformer was proposed by Vaswani et al.

[12] . The introduction of ViT has enabled people to use its powerful global feature modeling capabilities for image processing and has gradually been introduced into image dehazing tasks. Zhao et al.

[13] proposed the Transformer-based image dehazing algorithm HyLoG-ViT, and Song et al.

[14] proposed DehazeFormer, which improves the dehazing effect by improving LayerNorm, GeLU and spatial aggregation mechanisms in Swin Transformer. Some researchers have also established a dual-branch image dehazing network based on Transformer. Usually, the main branch is used to model contextual information, and the secondary branch implements specific functions such as depth estimation and full-resolution feature extraction. However, there are still problems such as inaccurate color restoration and weak dehazing generalization ability. Summary of the invention

[0008] Based on the analysis of the current status of image dehazing research, the present invention designs an image dehazing system with accurate color restoration effect based on the Transformer network. The present invention studies and analyzes the influence of Transformer structure and Gaussian blur on the image feature learning of deep learning network, designs an improved multi-head attention calculation method, improves the model's ability to capture image color information by adding additional convolution to V, and uses the Charbonnier loss function to optimize the training effect of the model. The present invention's Transformer image dehazing system with accurate color restoration effect includes the following steps:

[0009] (1) Input foggy image;

[0010] (2) Embed the image blocks into feature vectors;

[0011] (3) Use the Transformer network trained with the RESIDE dataset and the Charbonnier loss function to process the image feature vector and infer the value of K(x);

[0012] (4) The haze-free image J(x) is obtained by image reconstruction calculation from J(x)=K(x)I(x)-K(x)+b. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a diagram of the steps of processing an image by the system of the present invention;

[0014] Figure 2 Schematic diagram of the system structure and image size dimension change of the present invention, wherein Patch Embedding and Patch Unembedding represent feature embedding and feature de-embedding, and Reconstructing represents image reconstruction;

[0015] Figure 3 It is a schematic diagram of the image processing process of a Transformer layer of the system of the present invention, wherein the lower right frame diagram is a detailed schematic diagram of the MSA-C calculation process, Linear represents the linear layer, MLP represents the fully connected layer, and Padding represents the pooling operation;

[0016] Figure 4 This is a comparison chart of the defogging effect of the present invention and other methods, where from left to right are the fog image, FFA-Net defogging, RefineDNet defogging, DehazeFormer defogging, the defogging image of the algorithm in this paper, and the fog-free original image. DETAILED DESCRIPTION

[0017] In order to explain the content of the present invention more clearly, the present invention is further described below in conjunction with the accompanying drawings.

[0018] The present invention provides an image defogging system with accurate color restoration effect, and the schematic diagram of the image processing is as follows: Figure 2 As shown, the specific steps include:

[0019] Step (1): Get the input image.

[0020] Step (2): embed the image blocks into the feature vector. The image block embedding is determined by the following method: the input H×W×C image is divided into blocks to form a flattened image block sequence N×(P 2 ×C), where H is the image height, W is the image width, C is the number of image channels, P is the size of the image block, N = H × W / P 2 is the number of image blocks; then the dimension of each image block is P 2 ×C, use linear transformation on it to compress the dimension to D; then it can be input into the Transformer structure for feature extraction.

[0021] Step (3): Use the Transformer network trained with the RESIDE dataset and the Charbonnier loss function to process the image feature vector and infer the value of K(x). In step (3), the RESIDE dataset is a training and testing dataset for image defogging models proposed by Li et al.

[15] , which consists of pairs of real-world fog-free images and synthetic foggy images. It contains five subsets such as ITS and OTS and is widely used in image defogging tasks; the Charbonnier loss function is a loss function commonly used in image super-resolution model training, expressed as:

[0022]

[0023] In step (3), the Transformer processing image is determined in the following way: the network contains a total of 5 levels of Transformers; the input feature map is normalized by layer, and then linearly transformed to generate three vectors of Query, Key, and Value, and then the improved multi-head attention MSA-C is calculated; the obtained feature vector passes through the linear layer and the fully connected layer to end the Transformer calculation of this layer; in the process, the feature vector after calculating the improved multi-head attention MSA-C is fused with the feature vector before layer normalization, and the feature vector after processing by the linear layer and the fully connected layer is fused with the feature vector before processing. This process is as follows Figure 3 As shown, the improved multi-head attention MSA-C is expressed as:

[0024]

[0025] In this formula, Q represents the Query vector, K represents the Key vector, V represents the Value vector, KT represents the transpose of the Key vector, d represents the dimension size of the image block, Softmax represents the application of a normalized exponential function to the latter, and Conv. represents the application of a convolution calculation to the latter.

[0026] Step (4): The fog-free image J(x) is obtained by image reconstruction calculation from J(x)=K(x)I(x)-K(x)+b. In step (4), K(x) is determined by the following method:

[0027] K(x)=((I(x)-A) / t(x)+(Ab)) / (I(x)-1),

[0028] In this formula, I(x) represents the hazy image, A represents the atmospheric light value, t(x) represents the medium transmittance, and b is a constant bias with a value of 1. In step (4), image reconstruction refers to inferring the hazy image from the hazy image and the relevant parameters estimated by the image defogging system according to the atmospheric scattering model, and improving the image quality by performing feature fusion with the feature map that has not been processed by Transformer. The atmospheric scattering model is expressed as:

[0029] I(x)=J(x)t(x)+A(1-t(x)),

[0030] In this formula, I(x) represents the foggy image, J(x) represents the clear fog-free image, t(x) represents the medium transmittance, and A represents the atmospheric light value.

[0031] The defogging effect and color reproduction accuracy of the present invention (also known as ACRT method) and other mainstream methods were tested, and the quantitative experimental results obtained are shown in Table 1. PSNR stands for peak signal-to-noise ratio, and the larger the value, the better the effect; SSIM stands for structural similarity, and the larger the value, the better the effect; ΔE * Represents color difference, the smaller the value, the better the effect.

[0032] Table 1 Test results of image defogging effect and color restoration accuracy of the present invention and other mainstream methods

[0033]

[0034] The present invention conducts ablation experiments to compare the methods designed and selected for each link. The test results are shown in Table 2, which respectively show the experimental results of replacing the improved multi-head attention calculation with ordinary attention calculation, replacing the selected loss function with other loss functions and combined ablation.

[0035] Table 2 Ablation experiment test results of the present invention

[0036]

[0037] The Transformer image defogging system with accurate color restoration effect of the present invention can perform accurate color restoration while achieving good defogging, and is superior to other mainstream methods in effect. It has achieved good results in PSNR, SSIM, and ΔE, and has high application value.

[0038] All references in this specification are as follows:

[0039] [1]Kim TK, Paik JK, Kang B S.Contrast Enhancement System Using Spatially Adaptive Histogram Equalization with Temporal Filtering[J]. IEEETransactions on Consumer Electronics, 1998, 44(1):82–87.

[0040] [2]Land EH,McCann J J.Lightness and Retinex Theory[J].Journal of theOptical Society of America,1971,61(1):1–11.

[0041] [3]Rahman Z,Jobson D J,Woodell G A.Multi-Scale Retinex for ColorImage Enhancement[A].In:Proceedings of 3rd IEEE International Conference onImage Processing vol.3[C].Lausanne,Switzerland:1996,3:1003–1006.

[0042] [4]Tan R T.Visibility In Bad Weather from a Single Image[A].In:Proceedings of 2008IEEE Conference on Computer Vision and Pattern Recognition[C].Anchorage,AK,USA:IEEE,2008.1–8.

[0043] [5]He K,Sun J,Tang X.Single Image Haze Removal Using Dark ChannelPrior[A].In:Proceedings of 2009IEEE Conference on Computer Vision and PatternRecognition[C].Miami,FL,USA:IEEE,2009.1956–1963.

[0044] [6]Berman D,Treibitz T,Avidan S.Non-local Image Dehazing[A].In:Proceedings of 2016IEEE Conference on Computer Vision and Pattern Recognition[C].Las Vegas,NV,USA:IEEE,2016.1674–1682.

[0045] [7]Cai B, Xu

[0046] [8]Li B, Peng

[0047] [9]Qin

[0048]

[10] Zheng Yutong, Song Wei, Sun Haoying. Hybrid sample image dehazing based on latent space transformation[J]. Computer Engineering and Applications, Computer Engineering and Applications, 2023, 59(9): 225–236.

[0049]

[11] Huang Shuying, Wang Bin, Li Hongxia, et al. Image dehazing algorithm based on generative adversarial network[J]. Pattern Recognition and Artificial Intelligence, 2021, 34(11): 990–1003.

[0050]

[12] Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[A]. In: Proceedings of the 31st International Conference on Neural Information Processing Systems[C]. Red Hook, NY, USA: Curran Associates Inc., 2017.6000–6010.

[0051]

[13] Zhao S,Zhang L,Shen Y,et al.RefineDNet:A Weakly SupervisedRefinement Framework for Single Image Dehazing[J].IEEE Transactions on ImageProcessing,2021,30:3391–3404.

[0052]

[14] Song Y,He Z,Qian H,et al.Vision Transformers for Single ImageDehazing[J].IEEE Transactions on Image Processing,2023,32:1927–1941.

[0053]

[15] Li,Boyi and Ren,Wenqi and Fu,Dengpan,et al.Benchmarking Single-Image Dehazing and Beyond[J].IEEE Transactions on Image Processing,2019,28(1):492–505.

Claims

1. A Transformer image dehazing system with accurate color restoration effect, characterized by The following steps are involved: (1) Input foggy image I(x); (2) Embed the image blocks into feature vectors; (3) Use the Transformer network trained with the RESIDE dataset and the Charbonnier loss function to process the image feature vector and infer the value of K(x); (4) The haze-free image J(x) is obtained by image reconstruction calculation from J(x)=K(x)I(x)-K(x)+b.

2. A Transformer image defogging system with accurate color restoration effect according to claim 1, characterized in that: In step (2), the image blocks are embedded into feature vectors in the following way: The input H×W×C image is divided into blocks to form a flattened image block sequence N×(P 2 ×C), where H is the image height, W is the image width, C is the number of image channels, P is the size of the image block, N = H × W / P 2 is the number of image blocks; then the dimension of each image block is P 2 ×C, use linear transformation on it to compress the dimension to D; then it can be input into the Transformer structure for feature extraction.

3. A Transformer image defogging system with accurate color restoration effect according to claim 1, characterized in that: The Transformer network processed image feature vector in step (3) is determined in the following way: The network contains a total of 5 levels of Transformers. The input feature map is normalized by the layer, and then linearly transformed to generate three vectors: Query, Key, and Value. Then the improved multi-head attention MSA-C is calculated. The obtained feature vector passes through the linear layer and the fully connected layer to end the Transformer calculation of this layer. In this process, the feature vector after calculating the improved multi-head attention MSA-C is fused with the feature vector before layer normalization, and the feature vector after processing by the linear layer and the fully connected layer is fused with the feature vector before processing. Among them, the improved multi-head attention MSA-C is expressed as: In the formula, Q represents the Query vector, K represents the Key vector, V represents the Value vector, and K T represents the transpose of the Key vector, d represents the dimension size of the image block, Softmax represents the application of the normalized exponential function to the latter, and Conv. represents the application of convolution calculation to the latter; The Transformer network in step (3) is trained using the Charbonnier loss function, expressed as: In the formula, I represents the true value image, represents the predicted image, H is the image height, W is the image width, C is the number of image channels, i, j, k represent the three channels of the image, and ε is a value of 1×10 -3 The constant of K(x) in step (3) is determined as follows: Among them, I(x) represents the foggy image, A represents the atmospheric light value, t(x) represents the medium transmittance, and b is a constant bias with a value of 1.

4. The Transformer image defogging system with accurate color restoration effect according to claim 1, characterized in that: The image reconstruction calculation in step (4) refers to inferring the fog-free image from the foggy image and the relevant parameters estimated by the image defogging system according to the atmospheric scattering model; and improving the image quality by performing feature fusion with the feature map that has not been processed by the Transformer; wherein the atmospheric scattering model is expressed as: I(x)=J(x)t(x)+A(1-t(x)), Among them, I(x) represents the foggy image, J(x) represents the clear fog-free image, t(x) represents the medium transmittance, and A represents the atmospheric light value.

Images