Method and system for recovering compressed images with raindrops

Abstract

A computer-implemented method for recovering a compressed image with raindrops, the method comprising applying dual branches in a complementary manner for capturing low-frequency and high-frequency features from the compressed image, extracting the high-frequency features by high-frequency deep convolution (HFDC) with a zero-mean kernel; fusing the low-frequency features and the high-frequency features through a low-high attention module (LHAM); high-frequency features are extracted through high-frequency deep convolution (HFDC) with a zero mean kernel (kernel); and the low-frequency features and the high-frequency features are fused by adaptively distributing the importance of branches among the channels through the LHAM. According to the novel converter architecture provided by the invention, the compressed raindrop image is effectively recovered at the frame level, the assembly level and the module level by utilizing the design advantages of an attention mechanism and high-frequency friendliness.

Description

Technical Field

[0001] The present invention relates to a method and system for restoring a compressed image with raindrops. Background Art

[0002] Raindrops covering glass windows or camera lenses can reduce the visibility of images, seriously hindering many downstream computer vision tasks such as video surveillance and autonomous driving. Specifically, raindrops can severely blur the focus of the camera and introduce images that are different from those without raindrops. Therefore, removing raindrops from images has become an important task and has attracted widespread research interest. This work is extremely challenging because recovering large areas occluded by raindrops requires sufficient prior information, especially global context information. Recently, many studies [1]–[9] have been devoted to raindrop removal, including convolutional neural network (CNN)-based methods [1]–[8] and transformer-based methods [9]. Some CNN-based methods are committed to utilizing more raindrop priors, such as mask-guided [2]–[5] and edge shape-guided [6] methods. Other works aim to improve the capabilities of CNNs by introducing elegant modules and structures such as dual residual blocks (DuRB) [7] and Laplacian pyramid encoder-decoders [8]. However, due to fixed kernels and local operations, the inherent limitations of CNNs in modeling global context information still exist. Recently, a transformer-based raindrop removal method [9] has shown better performance than CNN-based methods due to its strong non-local modeling ability. However, most existing raindrop removal methods are designed for uncompressed or slightly compressed images, and they cannot remove raindrops well in real-world scenarios on normal or strongly compressed images.

[0003] In real-world applications, images are inevitably compressed to reduce storage overhead and improve transmission efficiency. Today, traditional block-based image compression methods such as JPEG

[10] , JPEG 2000

[11] , and BPG

[12] are widely deployed as coding infrastructures. They apply artificially customized transforms, quantization, and deentropy coding to reduce the spatial and statistical redundancy of images. Transform techniques in image compression aim to convert images from the spatial domain to the frequency domain, resulting in compact and mutually independent coefficients. Quantization is then used to eliminate unimportant redundant information for lossy compression. Typically, high-frequency components in transform coefficients are always quantized to zero to achieve higher compression efficiency. The significant loss of high-frequency information leads to texture loss and blurred output. In addition, block-based transforms and quantization also introduce blocking artifacts, resulting in discontinuity between adjacent blocks.

[0004] When raindrop interference and visual degradation caused by compression are intertwined, the introduced mixed distortion becomes more complicated. Raindrops will block the background texture, while in the non-raindrop area, high-frequency details will be lost due to compression, such as Figure 1 shown. Figure 1 Examples of real images (first column), raindrop images (second column), and compressed raindrop images (third column) are shown. Compressed raindrop images contain more complex degradations, including background occluded by raindrops and high-frequency details lost in non-raindrop areas due to compression. More importantly, the background associated with raindrops is destroyed by the block-based coding scheme. These coding blocks segment the raindrops, resulting in irregular raindrop shapes and destroying the context, resulting in uneven raindrop distribution. Therefore, these intertwined factors make removing raindrops from images more challenging.

[0005] References

[0006] The following references mentioned in this specification are indicated by number brackets, and their entire contents are incorporated into this application by reference.

[0007] [1] D.Eigen, D.Krishnan, and R.Fergus, “Restoring an image taken through a window covered with dirt or rain,” in Proceedings of the IEEE / CVFInternational Conference on Computer Vision, 2013, pp.633–640.

[0008] [2] R. Qian, RT Tan, W. Yang, J. Su, and J. Liu, “Attentive generative adversarial network for raindrop removal from a single image,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 2482–2491.

[0009] [3]Z.Hao,S.You,Y.Li,K.Li,and F.Lu,“Learning from syntheticphotorealistic raindrop for single image raindrop removal,”in Proceedings ofthe IEEE / CVF International Conference on Computer Vision Workshop,2019,pp.4340–4349.

[0010] [4]M.-W.Shao,L.Li,D.-Y.Meng,and W.-M.Zuo,“Uncertainty guided multi-scale attention network for raindrop removal from a single image,”IEEETransactions on Image Processing,vol.30,pp.4828–4839,2021.

[0011] [5]M.Shao,L.Li,H.Wang,and D.Meng,“Selective generative adversarialnetwork for raindrop removal from a single image,”Neurocomputing,vol.426,pp.265–273,2021.

[0012] [6]Y.Quan,S.Deng,Y.Chen,and H.Ji,“Deep learning for seeing throughwindow with raindrops,”in Proceedings of the IEEE / CVF InternationalConference on Computer Vision,2019,pp.2463–2471.

[0013] [7]X.Liu,M.Suganuma,Z.Sun,and T.Okatani,“Dual residual networksleveraging the potential of paired operations for image restoration,”inProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition,2019,pp.7000–7009.

[0014] [8]S.Zini and M.Buzzelli,“Laplacian encoder-decoder network forraindrop removal,”Pattern Recognition Letters,vol.158,pp.24–33,2022.

[0015] [9]J.Xiao,X.Fu,A.Liu,F.Wu,and Z.-J.Zha,“Image de-rainingtransformer,”IEEE Transactions on Pattern Analysis and Machine Intelligence,pp.1–18,2022.

[0016]

[10] G.Wallace,“The JPEG still picture compression standard,”IEEETransactions on Consumer Electronics,vol.38,no.1,pp.xviii–xxxiv,1992.

[0017]

[11] M.Rabbani and R.Joshi,“An overview of the jpeg 2000 still imagecompression standard,”Signal processing:Image communication,vol.17,no.1,pp.3–48,2002.

[0018]

[12] F.Bellard,“BPG image fromat,”2015.[Online].Available:https: / / bellard.org / bpg /

[0019]

[13] N.Park and S.Kim,“How do vision transformers work?”inInternational Conference on Learning Representations,2022.

[0020]

[14] J.Bai,L.Yuan,S.-T.Xia,S.Yan,Z.Li,and W.Liu,“Improving visiontransformers by revisiting high-frequency components,”in Computer Vision–ECCV2022.Springer,2022,pp.1–18.

[0021]

[15] C.Si,W.Yu,P.Zhou,Y.Zhou,X.Wang,and S.YAN,“Inception transformer,”Advances in Neural Information Processing Systems,2022.

[0022]

[16] J.M.Prewitt et al.,“Object enhancement and extraction,”Pictureprocessing and Psychopictorics,vol.10,no.1,pp.15–19,1970.

[0023]

[17] I.Sobel and G.Feldman,“A 3×3 isotropic gradient operator forimage processing,”Pattern Classification and Scene Analysis,pp.271–272,011973.

[0024]

[18] D.Marr and E.Hildreth,“Theory of edge detection,”Proceedings ofthe Royal Society of London.Series B.Biological Sciences,vol.207,no.1167,pp.187–217,1980.

[0025]

[19] C.Dong,Y.Deng,C.C.Loy,and X.Tang,“Compression artifacts reductionby a deep convolutional network,”in Proceedings of the IEEE / CVF InternationalConference on Computer Vision,2015,pp.576–584.

[0026]

[20] P.Svoboda,M.Hradis,D.Barina,and P.Zemcik,“Compression artifactsremoval using convolutional neural networks,”arXiv preprint arXiv:1605.00366,2016.

[0027]

[21] J.Guo and H.Chao,“Building dual-domain representations forcompression artifacts reduction,”in Computer Vision–ECCV 2016.Springer,2016,pp.628–644.

[0028]

[22] X.Zhang,W.Yang,Y.Hu,and J.Liu,“DMCNN:Dual-domain multiscaleconvolutional neural network for compression artifacts removal,”in 2018 25thIEEE International Conference on Image Processing,2018,pp.390–394.

[0029]

[23] H.Chen,X.He,L.Qing,S.Xiong,and T.Q.Nguyen,“DPW-SDNet:Dual pixel-wavelet domain deep CNNs for soft decoding of JPEGcompressed images,”inProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition Workshops,2018,pp.711–720.

[0030]

[24] L.Galteri,L.Seidenari,M.Bertini,and A.D.Bimbo,“Deep universalgenerative adversarial compression artifact removal,”IEEE Transactions onMultimedia,vol.21,no.8,pp.2131–2145,2019.

[0031]

[25] P.Zhang,X.Wang,Y.Zhang,L.Ma,J.Jiang,and S.Kwong,“Compressionartifacts reduction for depth map by deep intensity guidance,”in Advances inMultimedia Information Processing–PCM 2017.Springer,2018,pp.863–872.

[0032]

[26] Y.Kim,J.W.Soh,J.Park,B.Ahn,H.-S.Lee,Y.-S.Moon,and N.I.Cho,“Apseudo-blind convolutional neural network for the reduction of compressionartifacts,”IEEE Transactions on Circuits and Systems for Video Technology,vol.30,no.4,pp.1121–1135,2019.

[0033]

[27] Y.Kim,J.W.Soh,and N.I.Cho,“AGARNet:adaptively gated JPEGcompression artifacts removal network for a wide range quality factor,”IEEEAccess,vol.8,pp.20160–20170,2020.

[0034]

[28] M.Wang,X.Fu,Z.Sun,and Z.-J.Zha,“JPEG artifacts removal viacompression quality ranker-guided networks,”in Proceedings of the TwentyNinthInternational Conference on International Joint Conferences on ArtificialIntelligence,2021,pp.566–572.

[0035]

[29] J.Jiang,K.Zhang,and R.Timofte,“Towards flexible blind JPEGartifacts removal,”in Proceedings of the IEEE / CVF International Conference onComputer Vision,2021,pp.4997–5006.

[0036]

[30] Q.Xing,M.Xu,T.Li,and Z.Guan,“Early exit or not:Resourceefficientblind quality enhancement for compressed images,”in Computer Vision–ECCV2020.Springer,2020,pp.275–292.

[0037]

[31] Y.Chen,Y.Liu,M.Chen,Z.Wang,W.Yang,and Q.Liao,“Blind JPEGcompression artifacts removal by integrating channel regulation with exitstrategy,”IEEE Transactions on Multimedia,pp.1–14,2022.

[0038]

[32] A.Vaswani,N.Shazeer,N.Parmar,J.Uszkoreit,L.Jones,A.N.Gomez, Kaiser,and I.Polosukhin,“Attention is all you need,”Advances in NeuralInformation Processing Systems,vol.30,2017.

[0039]

[33] J.Devlin,M.-W.Chang,K.Lee,and K.Toutanova,“BERT:Pre-training ofdeep bidirectional transformers for language understanding,”arXiv preprintarXiv:1810.04805,2018.

[0040]

[34] A.Dosovitskiy,L.Beyer,A.Kolesnikov,D.Weissenborn,X.Zhai,T.Unterthiner,M.Dehghani,M.Minderer,G.Heigold,S.Gelly et al.,“An image isworth 16x16 words:Transformers for image recognition at scale,”inInternational Conference on Learning Representations,2021.

[0041]

[35] Z.Liu,Y.Lin,Y.Cao,H.Hu,Y.Wei,Z.Zhang,S.Lin,and B.Guo,“Swintransformer:Hierarchical vision transformer using shifted windows,”inProceedings of the IEEE / CVF International Conference on Computer Vision,2021,pp.10012–10022.

[0042]

[36] X.Dong,J.Bao,D.Chen,W.Zhang,N.Yu,L.Yuan,D.Chen,and B.Guo,“Cswintransformer:A general vision transformer backbone with cross-shaped windows,”in Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition,2022,pp.12124–12134.

[0043]

[37] H.Chen,Y.Wang,T.Guo,C.Xu,Y.Deng,Z.Liu,S.Ma,C.Xu,C.Xu,and W.Gao,“Pre-trained image processing transformer,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2021,pp.12299–12310.

[0044]

[38] J.Liang,J.Cao,G.Sun,K.Zhang,L.Van Gool,and R.Timofte,“SwinIR:Image restoration using swin transformer,”in Proceedings of the IEEE / CVFInternational Conference on Computer Vision,2021,pp.1833–1844.

[0045]

[39] Z.Wang,X.Cun,J.Bao,W.Zhou,J.Liu,and H.Li,“Uformer:A general u-shaped transformer for image restoration,”in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2022,pp.17683–17693.

[0046]

[40] S.W.Zamir,A.Arora,S.Khan,M.Hayat,F.S.Khan,and M.-H.Yang,“Restormer:Efficient transformer for high-resolution image restoration,”inProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition,2022,pp.5728–5739.

[0047]

[41] Z.Chen,Y.Zhang,J.Gu,L.Kong,X.Yuan et al.,“Cross aggregationtransformer for image restoration,”Advances in Neural Information ProcessingSystems,vol.35,pp.25478–25490,2022.

[0048]

[42] J.Zhang,Y.Zhang,J.Gu,Y.Zhang,L.Kong,and X.Yuan,“Accurate imagerestoration with attention retractable transformer,”in InternationalConference on Learning Representations,2023.

[0049]

[43] D.Ye,Z.Ni,W.Yang,H.Wang,S.Wang,and S.Kwong,“Glow in the dark:Low-light image enhancement with external memory,”IEEE Transactions onMultimedia,pp.1–16,2023.

[0050]

[44] D.Ye,Z.Ni,H.Wang,J.Zhang,S.Wang,and S.Kwong,“Csformer:Bridgingconvolution and transformer for compressive sensing,”IEEE Transactions onImage Processing,vol.32,pp.2827–2842,2023.

[0051]

[45] Z.Pan,J.Cai,and B.Zhuang,“Fast vision transformers with hiloattention,”Advances in Neural Information Processing Systems,2022.

[0052]

[46] K.Yuan,S.Guo,Z.Liu,A.Zhou,F.Yu,and W.Wu,“Incorporatingconvolution designs into visual transformers,”in Proceedings of the IEEE / CVFInternational Conference on Computer Vision,2021,pp.579–588.

[0053]

[47] K.B.Krishnan,S.P.Ranga,and N.Guptha,“A survey on different edgedetection techniques for image segmentation,”Indian J.Sci.Technol,vol.10,no.4,pp.1–8,2017.

[0054]

[48] P.A.Mlsna and J.J.Rodriguez,“Gradient and laplacian edgedetection,”in The Essential Guide to Image Processing.Elsevier,2009,pp.495–524.

[0055]

[49] A.S.Ahmed,“Comparative study among sobel,prewitt and canny edgedetection operators used in image processing,”J.Theor.Appl.Inf.Technol,vol.96,no.19,pp.6517–6525,2018.

[0056]

[50] S.R.Gunn,“On the discrete representation of the laplacian ofgaussian,”Pattern Recognition,vol.32,no.8,pp.1463–1472,1999.

[0057]

[51] F.Chollet,“Xception:Deep learning with depthwise separableconvolutions,”in Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition,2017,pp.1800–1807.

[0058]

[52] D.P.Kingma and J.Ba,“Adam:A method for stochastic optimization,”arXiv preprint arXiv:1412.6980,2014.

[0059]

[53] Z. Wang, AC Bovik, HR Sheikh, and EPSimoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Transactionson Image Processing, vol. 13, no. 4, pp. 600–612, 2004. Summary of the Invention

[0060] To address the above-mentioned challenging obstacles, certain embodiments of the present invention strive to provide an integrated solution. Their main idea is to build an effective framework to capture local and global information, thereby promoting accurate context modeling. In addition, in the module design, high-frequency information is given special attention to achieve the recovery of high-frequency texture details. Specifically, a new transformer-based model is provided, which achieves effective global context modeling through a self-attention mechanism and promotes the recovery of high-frequency information by introducing high-frequency-friendly designs at the framework, component, and module levels.

[0061] According to a first aspect of the present invention, a computer-implemented method for restoring a compressed image with raindrops is provided, the method comprising applying dual branches in a complementary manner for capturing low-frequency features and high-frequency features from a compressed image, extracting the high-frequency features by a high-frequency deep convolution (HFDC) with a zero-mean kernel; fusing the low-frequency features and the high-frequency features by a low-high attention module (LHAM); extracting the high-frequency features by a high-frequency deep convolution (HFDC) with a zero-mean kernel; and fusing the low-frequency features and the high-frequency features by the LHAM by adaptively distributing the importance of branches between channels.

[0062] In some embodiments, low-frequency features may include global context information under the raindrops, and high-frequency features may include local high-frequency details that may be lost due to compression.

[0063] In some embodiments, the dual branches may include a low-frequency branch that extracts low-frequency features through a self-attention mechanism, and a high-frequency branch that extracts high-frequency features through HFDC.

[0064] In some embodiments, the self-attention mechanism may include relative position multi-head self-attention (RMSA).

[0065] In some embodiments, extracting high-frequency features through the high-frequency branch may include dividing the input of the high-frequency branch into multiple channels, applying high-frequency convolution (HFConv) on each channel, and concatenating features from each channel.

[0066] In some embodiments, applying HFConv on each channel may include removing the spatial mean from an initial kernel and applying the resulting kernel to the convolution.

[0067] In some embodiments, the high frequency branch may include a reshape / flatten operation, HFDC, and a 1×1 point-wise convolution (PConv).

[0068] In some embodiments, LHAM can be a window-level attention scheme, performed in each local window.

[0069] In some embodiments, the process in the local window may include: "mixing low-frequency features and high-frequency features from two branches to obtain integrated features, and reshaping the integrated features into spatial features, which are further fed into the ReLU layer to obtain mixed features; obtaining compact features by applying average pooling to each channel; obtaining weight matrices of low-frequency features and high-frequency features from the compact features; and generating weighted low-frequency and high-frequency features by performing channel-by-channel addition on the low-frequency and high-frequency features and their corresponding weight matrices, and generating fused features by element-by-element addition.

[0070] In some embodiments, when generating a fused feature, different channels of the fused feature may be configured as different combinations of low-frequency and high-frequency features.

[0071] In some embodiments, the above method may further include merging the fused features into full-resolution features, and adding the full-resolution features to the input features of the compressed image.

[0072] In certain embodiments, the method may further include applying a locally enhanced feed-forward network (LeFF) on the features to generate output features.

[0073] In certain embodiments, LeFF can be configured to perform dimensionality manipulation and nonlinear processing on features.

[0074] In some embodiments, applying the dual branch may be performed at a frame level, extracting high-frequency features through HFDC may be performed at a component level, and fusing low-frequency features with high-frequency features through LHAM may be performed at a module level.

[0075] According to a second aspect of the present invention, a low-high frequency transformer (LHFT) module is provided, which is configured to perform the above-mentioned computer-implemented method, including a self-attention mechanism for extracting low-frequency features from a compressed image, a high-frequency deep convolution (HFDC) with a zero-mean kernel for extracting high-frequency features, and a low-high frequency attention module (LHAM) for fusing low-frequency features and high-frequency features.

[0076] According to a third aspect of the present invention, a hierarchical U-shaped encoder-decoder network with residual learning is provided for restoring compressed images with raindrops. The network includes an input projection block for extracting features from an input image; and an encoder block for extracting multi-scale features. The encoder block includes two or more encoder sub-blocks and a bottleneck block, wherein each encoder sub-block includes a sequential stack of low-frequency transformer (LHFT) modules and a downsampling layer for reducing spatial resolution and expanding feature channel dimensions. The network also includes a decoder module for recovering features, which includes two or more decoder sub-modules corresponding to two or more spatial resolutions, wherein each decoder sub-module includes a channel for doubling the spatial resolution and reducing it by half, a concatenation unit for concatenating the upsampled features with features from the corresponding encoder sub-block via jumper connections, and a sequential stack of LHFT modules. The network also includes an output projection block for reconstructing a residual image based on the recovered features, and an addition unit for obtaining a reconstructed image by adding the residual image to the input image. The LHFT module can be the LHFT of the second aspect of the present invention.

[0077] According to a fourth aspect of the present invention, a system for restoring a compressed image with raindrops is provided, the system comprising one or more processors and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising instructions for performing or facilitating the performance of the above-mentioned computer-implemented method.

[0078] According to a fifth aspect of the present invention, there is provided a non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the above-mentioned computer-implemented method.

[0079] Other features and aspects of the present invention will become clear from the following description of specific embodiments and the accompanying drawings. Any feature described in this disclosure in relation to one aspect or embodiment may be combined with any other feature described in this disclosure in relation to any other aspect or embodiment as needed. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0081] Figure 1 Examples of ground truth images (first column), raindrop images (second column), and compressed raindrop images (third column) are shown.

[0082] Figure 2 The overall architecture of a layered U-shaped encoder-decoder network according to one embodiment of the present invention is shown.

[0083] Figures 3A to 3C Different operational diagrams of existing transformers for extracting low-frequency and high-frequency information are shown.

[0084] Figure 4 An operational diagram of a converter according to an embodiment of the present invention is shown.

[0085] Figure 5A A schematic diagram of High Frequency Deep Convolution (HFDC) according to an embodiment of the present invention is shown.

[0086] Figure 5B FIG. 4 is a schematic diagram of high frequency convolution (HFConv) according to an embodiment of the present invention.

[0087] Figure 6 Shown is the visualization of features extracted by ordinary depthwise convolution (DC) and HFDC in the first block of the first encoder stage.

[0088] Figure 7 The structure of the low-high attention module (LHAM) according to one embodiment of the present invention is shown, which includes four stages: mixing, compression, excitation and fusion.

[0089] Figure 8 The visualization values V for different channels in the first block of the third encoder stage are shown.

[0090] Figure 9 The proposed method according to one embodiment of the present invention is shown and qualitatively compared with the existing method in processing compressed images with raindrops under QF10.

[0091] Figure 10 The proposed method according to one embodiment of the present invention is shown and qualitatively compared with the existing method in processing compressed images with raindrops under QF50.

[0092] Figure 11 Shows the input value features V and low-frequency features F at the first block of the first encoder stage in Backbone+DC and Backbone+HFDC L and high-frequency features F H Fourier analysis of .

[0093] Figure 12 Shows the weights w of the channels in different local windows at the first block of the second encoder stage. L and w H .

[0094] Figure 13 An example information handling system is shown in accordance with some embodiments of the present invention.

[0095] Before explaining any embodiments of the present invention in detail, it should be understood that the invention is not limited in its application to the details and arrangement of components of the embodiments described below or illustrated in the accompanying drawings. The invention is capable of other embodiments and of being carried out in various ways. Furthermore, it should be understood that the phraseology and terminology used herein are for descriptive purposes only and are not to be construed as limiting. DETAILED DESCRIPTION

[0096] Some embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0097] An embodiment of the present invention integrates a self-attention mechanism and a convolutional layer at the framework level to form a low-high frequency transformer (LHFT) with rich low and high frequencies. At the component level, the constraint of the zero-mean signal structure is introduced to develop a high-frequency deep convolution (HFDC). This module is HF-friendly and can extract and recover richer high-frequency features than ordinary deep convolution. At the module level, the low-high frequency attention module (LHAM) aims to adjust the composition of low and high frequencies by adaptively distributing the importance of branches between channels. It is motivated by the observation that features from different channels tend to extract different frequencies. The main contributions are summarized as follows.

[0098] The present invention attempts to solve the practical problem of raindrop removal in compressed images for the first time and establishes a JPEG compressed raindrop image dataset. This dataset is extensively evaluated at various compression rates.

[0099] This paper proposes a novel high-frequency-friendly transformer architecture, named HFGlobalFormer, for raindrop removal from compressed images. This architecture integrates the high-frequency extraction capabilities of convolutional layers and the global context modeling capabilities of self-attention mechanisms at the framework, component, and module levels. Experimental results consistently demonstrate that HFGlobalFormer outperforms existing methods without incurring additional computational cost.

[0100] The embodiments of the present invention propose LHFT at the framework level to handle mixed degradation, where dual branches are introduced in a complementary manner to globally extract low frequencies and locally extract high frequencies.

[0101] The present invention proposes HFDC with a zero-mean kernel at the component level to facilitate high-frequency feature extraction. The zero-mean property enables convolution to extract high-frequency-dominated features.

[0102] The embodiments of the present invention propose LHAM at the module level to achieve effective feature fusion by adaptively allocating the importance of low and high frequencies between channels, thereby eliminating various frequency-related degradations.

[0103] Related work

[0104] A. Single Image Raindrop Removal Based on Deep Learning

[0105] Many deep learning-based methods have been proposed to remove raindrops from a single image. Eigen et al. [1] first used a deep neural network to remove raindrops from a single image. Qian et al. [2] proposed an attention-based generative adversarial network (GAN) to generate raindrop-free images guided by raindrop masks. Hao et al. [3] developed a raindrop-aware neural network for joint detection and removal of raindrops. Shao et al. [4] further explored the fuzzy information of raindrops and proposed an uncertainty-guided multi-scale attention network (UMAN). In addition, Shao et al. [5] used the background texture information and features of raindrops to remove raindrops. For raindrops of various shapes, Quan et al. [6] proposed a shape-based attention model. Liu et al. [7] proposed a dual residual network (DuRN) to fully utilize the information between different layers in the network. Zini et al. [8] designed a Laplacian pyramid encoder-decoder network that contains Laplacian decomposition and alignment. However, due to the poor ability of convolution to model global information, the performance of these CNN-based methods is limited. To capture long-range information, Xiao et al. [9] proposed an image deraining transformer (IDT) with position modeling and complementary transformer modules for rain streak and raindrop removal. Although many efforts have been made to remove raindrops from single images, deraining compressed images has been rarely studied. Due to the complexity of mixed degradation, it is a more challenging task for existing methods.

[0106] B. JPEG Artifact Removal Based on Deep Learning

[0107] Due to the success of CNN in many tasks, deep learning-based JPEG artifact removal methods have made significant progress. Dong et al.

[19] proposed the Artifact Reduction Convolutional Neural Network (ARCNN), which is a pioneering work on CNN-based JPEG artifact removal. Svoboda et al.

[20] developed a deeper network based on residual learning, which outperformed ARCNN. Some works use DCT domain priors

[21] ,

[22] or wavelet information

[23] to improve performance. Galteri et al.

[24] proposed a GAN-based conversion method for photorealistic textures. Zhang et al.

[25] proposed an intensity-guided CNN (IGNet) model for removing artifacts from JPEG compressed depth images. Recently, people have tried to use quality factor (QF) for enhancement. Kim et al.

[26] estimated the QF and used it to select a suitable removal network. Kim et al.

[27] further proposed an adaptive artifact removal system (AGARNet), which uses a single model to handle various QFs. Wang et al.

[28] proposed a differentiable compression quality ranker to guide the removal network. Jiang et al.

[29] used the QF estimation branch to guide the restoration network. Xing et al.

[30] proposed a dynamic quality enhancement network and utilized an early exit strategy. Chen et al.

[31] combined channel regulation with an early exit strategy, where the estimated QF was used to determine the exit stage. Existing deep learning-based JPEG artifact removal methods are designed for single distortion and may not be well adapted to the raindrop removal problem of compressed images without effectively utilizing global context and high-frequency information.

[0108] C. Visual Transformer

[0109] Recently, transformer-based architectures

[32] ,

[33] have been introduced to high-level vision tasks, such as ViT

[34] , Swin Transformer

[35] , and Cswin Transformer

[36] . Transformer-based image restoration has also attracted the attention of many researchers. Chen et al.

[37] proposed an image restoration method based on image processing transformer (IPT), which utilizes vanilla ViT for image restoration. Liang et al.

[38] developed an image restoration method based on Swin Transformer (SwinIR), and Wang et al.

[39] proposed Uformer with a window-based attention mechanism and a hierarchical encoder-decoder architecture. Zamir et al.

[40] designed the Restoration Transformer (Restormer) to capture global relationships with linear complexity and calculate cross-channel cross-covariance. Chen et al.

[41] explored rectangular window self-attention and axial shifting to improve performance. To obtain a wider receptive field, Zhang et al.

[42] proposed dense and sparse attention modules for image restoration. To exploit the specific properties of samples, Ye et al.

[43] introduced the External Memory Augmented Network (EMNet) into the Transformer benchmark. In addition, Ye et al.

[44] proposed the CSformer architecture for compressed sensing.

[0110] method

[0111] A. Motivation

[0112] Raindrop removal from compressed images is a realistic and challenging problem that has been rarely studied in recent advanced raindrop removal methods [1]–[9]. Given a compressed image with raindrops, embodiments of the present invention aim not only to remove raindrops but also to restore the blurred background caused by compression, which requires both global low-frequency and local high-frequency context. CNN-based methods [1]–[8] have limited receptive fields and cannot model global context information well. In contrast, transformer-based methods [9] focus on capturing global dependencies but ignore high-frequency information and relevant low-frequency / high-frequency combinations. In addition, most existing methods are designed for uncompressed or slightly compressed images and cannot perform well in such challenging mixed degradations.

[0113] The embodiments of the present invention fill the above gaps and deficiencies. The embodiments of the present invention integrate frequency domain information and context modeling into the enhanced model through the following three aspects:

[0114] Combination of MSA and convolution. The embodiments of the present invention introduce two complementary branches in the proposed HFGlobalFormer, extracting low-frequency and high-frequency information through MSA and convolution layers, and further fusing them together to improve performance.

[0115] Injecting frequency prior into module design: Embodiments of the present invention inject a zero-mean core constraint into the high-frequency DC filter to better capture high-frequency features.

[0116] Rebalancing different frequencies by recalibration. Since the features of different channels tend to focus on different frequencies

[15] , embodiments of the present invention propose to use LHAM to adaptively adjust the composition of complementary branches between channels to more effectively fuse low-frequency / high-frequency information.

[0117] To achieve the above goals, the overall framework and detailed structure of each module will be described in the following sections.

[0118] B. Overall Architecture

[0119] The overall architecture of the proposed HFGlobalFormer is a hierarchical U-shaped encoder-decoder network with residual learning, which consists of input / output projection blocks, stacked low-high frequency transformer (LHFT) blocks, down / up sampling layers, and skip connections between encoder / decoder stages, as shown in Figure 2 As shown in the figure. The number of LHFT modules in each stage is denoted as {N1, N2, N3, N4, N5, N6, N7, N8, N9}. Specifically, an input projection module consisting of a 3×3 convolutional layer with LeakyReLU is first applied to extract features from the input image. The extracted features are then input into the encoder stage, which includes, for example, four encoder stages and one bottleneck stage. Each encoder stage consists of a sequential stack of LHFT modules and a downsampling layer. The downsampling layer consists of a 4×4 convolution with a stride of 2, which is used to reduce the spatial resolution and expand the channel dimension of the features. At the end of the encoder stage, the bottleneck stage contains only one stack of LHFT modules. Multi-scale features are extracted by the encoder stage and are further input into the decoder stage.

[0120] For example, the decoder stage also includes four decoder stages, corresponding to four spatial resolutions. Each decoder stage first uses an upsampling layer with a 2×2 transposed convolution with a stride of 2 to double the spatial resolution and reduce the number of channels by half. Then, the upsampled features and the features from the corresponding encoder stage are concatenated through a skip connection to achieve more efficient information flow between the encoder and decoder. The concatenated features are further input into the LHFT module. After the decoder stage, the recovered features are input into the output projection module, which has a 3×3 convolution layer for reconstructing the residual image. Finally, the reconstructed image is obtained by adding the residual image and the input image.

[0121] C. Low-Frequency Transformer (LHFT)

[0122] Restoring compressed images with raindrops is challenging because both the blurred areas under the raindrops and the high-frequency details lost due to compression should be restored, which requires fully extracting and interacting global low-frequency and local high-frequency information.

[0123] Recent research on transformers has focused on the efficient utilization of both low-frequency and high-frequency information, such as the Initialization Transformer (IFormer)

[15] , the High-Low Attention Transformer (HiLo)

[45] , and the Image De-raining Transformer (IDT) [9]. Figures 3A to 3C The operation diagrams of different transformers for extracting low-frequency and high-frequency information are shown respectively according to the Initialization Transformer (IFormer)

[15] , High-Low Attention Transformer (HiLo)

[45] , and Image De-Raindrop Transformer (IDT) [9]. LN and LP represent layer normalization and linear projection, respectively. FFN and LeFF represent feedforward network and local enhancement feedforward network, respectively. PConv is point-wise convolution. It is worth noting that IFormer and HiLo are designed for high-level vision tasks. Figure 3A As shown in , IFFormer uses the maximum pooling operation and depthwise convolution (DC) to capture high-frequency features. For low-frequency features, it uses multi-head self-attention (MSA). Figure 3B As shown in , HiLo uses two MSAs to capture low-frequency and high-frequency information. Figure 3C As shown in

[15] , IDT utilizes DC to model high-frequency information and uses relative position multi-head self-attention (RMSA) to capture global low-frequency information. Although the performance is improved by the cooperation between MSA and convolution, the high-frequency-friendly design and the effective fusion of low-frequency and high-frequency information have not been well explored.

[0124] To solve the complex mixed degradation problem, an embodiment of the present invention proposes LHFT, which uses self-attention (such as RMSA) to capture global low-frequency information; HFDC with a high-frequency friendly design to extract high-frequency features (i.e., modeling local high-frequency details); and LHAM to achieve effective fusion of low-frequency and high-frequency information. Then, LeFF

[39] ,

[46] is used to perform dimensionality manipulation and nonlinear processing on the features, which consists of a 3×3 depthwise convolution between two linear projection layers.

[0125] Specifically, if Figure 4 As shown, the input features of block l are first layer normalized (LN), and the normalized features are divided into non-overlapping windows to obtain the features

[0126] Z P =Partition(LN(Z l-1 )), (1)

[0127] where Z l-1 It also represents the output features of block l-1. Then, three linear projection layers with reshape operations are applied to generate

[0128] Q,K,V=Reshape(Z P W Q ,Z P W K ,Z P W V ), (2)

[0129] in, is the projection matrix, which is shared between windows. Q, K, V represent the query, key, and value of self-attention. m represents the number of heads, and the head dimension is d = C / m.

[0130] In order to capture global low-frequency and local high-frequency features, the value V is fed into the complementary branch. It is worth noting that since the input features are partitioned, there are two complementary branches in each local window. In order to extract low-frequency information, RMSA

[35] with a reshape operation is applied to obtain low-frequency features.

[0131]

[0132] Where B is a learnable relative position encoding. The attention operation is performed in parallel on each head, and the result is further reshaped into feature Z P To effectively capture high-frequency features, the value V is input to the high-frequency branch, which consists of a reshape / flattening operation, HFDC, and 1×1PConv. To facilitate the convolution operation, the V value is reshaped into a size of Next, the proposed HFDC and PConv are used in sequence to obtain high-frequency features, which can be regarded as high-frequency depth-separable convolution. Finally, the high-frequency features are flattened to a size of F L Features The entire high-frequency branch can be expressed as:

[0133] F H =Flatten((PConv(HFDC(Reshape(V)))).(4)

[0134] After complementary branching, F is fused via LHAM L and F H Then, the fused features are merged into full-resolution features of size H×W×C. Then, the merged features and Z l-1 Add up and get the features

[0135]

[0136] According to [9],

[39] , LeFF is applied as a feed-forward network after LN to produce the output features of the proposed LHFT

[0137]

[0138] D. High-Frequency Deep Convolution (HFDC)

[0139] The recovery of high-frequency information is crucial in raindrop removal of compressed images. Although Park et al.

[13] revealed that convolution is more like a high-pass filter, the response of vanilla convolution may contain the DC bias of the input (mainly low-frequency information) due to the lack of constraints on the kernel, so the effectiveness of extracting high-frequency features can be improved. The kernel of traditional high-pass filters, such as Prewitt

[16] , Sobel

[17] , and LoG

[18] operators, always has a zero-mean property

[47] –

[50] . In

[48] , the authors pointed out that this property helps to suppress the response caused by the DC bias in the input, which can be explained by frequency decomposition.

[0140] HFDC. In order to capture high-frequency features more effectively, the zero-mean property is introduced into high-frequency deep convolution (HFDC). The ordinary deep convolution proposed in the Xception

[51] framework processes each input channel independently, which can improve learning efficiency and reduce computational overhead. Certain embodiments of the present invention propose to implement HFDC by replacing ordinary convolution with HFConv with a zero-mean kernel to extract features for each channel. Figure 5A As shown in Figure 1, the input is divided into multiple channels, each channel is convolved with a 5×5 HFConv, and the result is the concatenation of the features of each channel.

[0141] HFConv. HFConv is the key component of the proposed HFDC, which aims to extract high-frequency features from the input.

[0142] like Figure 5B As shown, f and Represent the input and output signals of HFConv respectively. ini Represents the initial kernel, which is a learnable parameter. It is obtained by removing the spatial mean from it and applying it to convolution with the input. Finally, the output is generated by adding the convolution result and Bias The internal processing of HFConv can be expressed as:

[0143] W final =W ini -Mean(W ini ),(7)

[0144]

[0145] To further understand the proposed HFDC, Figure 6 This figure shows feature map visualization. In the first block of the first encoder stage, the degraded input block and the features extracted by ordinary depthwise convolution (DC) and HFDC are visualized. It can be seen that HFDC can focus more on high-frequency information and maintain clear texture details, while the features extracted by DC still contain some low-frequency regions. In addition, HFDC can extract structural information and suppress the influence of raindrops.

[0146] E. Low-High Attention Module (LHAM)

[0147] Compressed images with raindrops contain various degradations of different contents caused by different raindrops and compression mechanisms. Specifically, raindrops with different shapes and sizes cover the background in different ways, and the covered contents are also different due to the random positions of raindrops. Since complex texture regions always lose more high-frequency information compared to smooth regions, the distortion caused by compression is different in regions with different textures. In addition, as shown in

[15] , features from different channels always tend to focus on different frequencies, which can be seen in the Figure 8 It is observed that the features of channel 1 and channel 3 contain more high frequencies than the features of channel 2. Although the complementary branches in the proposed LHFT can effectively extract low / high frequencies, the model adaptability of content and channels remains to be explored. To this end, certain embodiments of the present invention propose LHAM to fuse low-frequency and high-frequency features by adaptively allocating the importance of branches between channels. In addition, the proposed LHAM is a window-level attention scheme, such as Figure 4 As shown, it is executed in each local window. For simplicity, the process in the local window is as follows Figure 7 As shown, it includes four stages: mixing, compression, excitation and fusion.

[0148] Mixed. Given from F L Low-frequency features of local windows and from F H High-frequency characteristics First, they are concatenated and the fully connected layer FC0 is used to mix the information from the two branches. Then, the integrated features are reshaped into spatial features, which are further fed into the ReLU layer to obtain the mixed features. The mixing stage can be expressed as:

[0149] X C =Concatenation(X L ,X H ),(9)

[0150] X 0 =ReLU(Reshape(FC0(X C ))).(10)

[0151] Squeeze. The squeezing stage aims to aggregate spatial information by applying average pooling to each channel:

[0152] X a =Avgpooling(X 0 ).(11)

[0153] Excitation. Compression feature Enter the incentive phase to obtain X L and X H Specifically, the fully connected layer FC1 is used to process the input and double the channels. Next, a ReLU layer and another fully connected layer FC2 are applied to shrink and merge the expanded features. The shrunk features are reshaped and fed into the softmax layer. Finally, the resulting 2×C×1 matrix is divided into two weight matrices and The entire process of the incentive phase is as follows:

[0154] X 2 =FC2(ReLU(FC1(X a ))),(12)

[0155] W L ,W H =Split(Softmax(Reshape(X 2 ))).(13)

[0156] Fusion. The fusion stage aims to generate weighted low-frequency and high-frequency features and fuse them. These features and their corresponding weight matrices are multiplied channel by channel. Finally, element-wise addition is used to generate the fused features. In short, different channels of fused features are different combinations of low-frequency and high-frequency features. The fusion process can be expressed as:

[0157]

[0158] in, and Represents X L ,X H and X out The kth channel of . and Respectively represent W L and W H In addition, due to the existence of the Softmax layer,

[0159] Experiments and analysis

[0160] A. Experimental Setup

[0161] 1) JPEG compressed raindrop image dataset: In order to facilitate training and evaluate the performance of removing raindrops from compressed images, a JPEG compressed raindrop image dataset was constructed. This JPEG compressed raindrop image dataset is based on the dataset created by Qian et al. [2]. Qian et al. took raindrop / clean image pairs through two pieces of glass, one of which was clean and the other was splashed with water. According to the settings of the existing raindrop removal method, 861 pairs of images were used for training, and 58 pairs of well-aligned images in Test_a were used for testing. In addition, all raindrop images were compressed using JPEG under 5 quality factors (QF) {10, 20, 30, 40, 50} to obtain image pairs (compressed raindrop images and clean images) and collect five sub-datasets.

[0162] 2) Implementation: In this disclosure, certain embodiments of the present invention train a model for each QF from scratch and test each QF to demonstrate the effectiveness of the proposed HFGlobalFormer under various compression rates. During training, the training images are randomly cropped into 128×128 patches and randomly flipped in the horizontal and vertical directions to increase the amount of data. The learning rate and batch size are set to 1e respectively. -4 and 4. Adam optimizer

[52] is used to train the network 8.6×10 5 step. In one example, the proposed HFGlobalFormer contains 4 encoder stages, 1 bottleneck stage, and 4 decoder stages. In the experiments, the number of LHFT modules in each stage is denoted as {N1,N2,N3,N4,N5,N6,N7,N8,N9} and is set to {3,3,2,2,1,1,2,2,3}. The number of heads in each stage is set to {1,2,4,8,16,16,8,4,2}, and the head size d is set to 32.

[0163] 3) Loss function: Based on the loss function in IDT [9], a single negative SSIM

[53] is used on the RGB channels to guide the training:

[0164] L=-SSIM(I Degraded ,I GT ), (15)

[0165] Among them I Degraded and I GT represent the compressed raindrop image and the ground truth image respectively.

[0166] B. Performance Comparison

[0167] Comparison Setting: The proposed HFGlobalFormer is compared with several existing raindrop removal methods, including AttenGAN[2], RaindropAtten[6], DuRN[7], and IDT[9]. For fair comparison, all models are retrained using their released codes, since their pre-trained models for a single raindrop removal task cannot adapt well to the recovery of raindrops in compressed images. For each method, five models corresponding to five QFs are also trained and evaluated for each QF. Following the existing raindrop removal methods, PSNR and SSIM are calculated on the Y channel of the YCbCr space.

[0168] Quantitative Results: Table 1 lists the quantitative evaluation results of raindrop removal on compressed images on the JPEG compressed raindrop image dataset. It can be seen that the proposed method outperforms the existing methods on average in terms of PSNR and SSIM. In addition, the proposed method performs best on QF20, QF30, QF40, and QF50, demonstrating its generalization ability under various compression rates. For QF10, the proposed method significantly surpasses RaindropAtten[6] in SSIM and achieves comparable results in PSNR.

[0169] Table 1

[0170] Quantitative comparison with existing compressed image denoising methods. The best and second-best results are shown in brackets and underlined, respectively.

[0171]

[0172] Compared with the IDT, which ranks second in average performance, the method proposed in the exemplary embodiment of the present invention achieves significant improvements. For raindrop removal of compressed images, the proposed method benefits from the effective collaboration between the self-attention mechanism and the high-frequency-friendly design, thereby improving the ability to utilize low-frequency and high-frequency information. In addition, the proposed method adopts adaptive fusion of low-frequency / high-frequency features, thereby improving the performance under various complex degradation conditions. Compared with the CNN-based methods (AttenGAN, RaindropAtten and DuRN), significant gains can be observed in terms of PSNR and SSIM on average. CNN-based methods have limited capabilities in global dependency modeling, which hinders their effective removal of raindrops with complex shapes and appearances.

[0173] 3) Qualitative results: This paper provides qualitative comparison results of different compressed image raindrop removal methods to demonstrate the effectiveness of the proposed method. Figure 9 and Figure 10The visual results of the existing method and the proposed method on QF10 and QF50 are shown respectively. For the highly compressed image with raindrops under QF10, it can be observed that the cropped area in the degraded input image contains strong blocking artifacts and severe occlusion caused by irregular raindrops due to compression. Figure 9 As shown in Figure 3, the proposed method can significantly reduce the impact of raindrops and recover more textures from highly distorted input images. In contrast, the compared methods are still affected by raindrops and cannot reconstruct image details. For compressed images with raindrops under QF50, areas with dense raindrops and complex structures (such as buildings) are selected to verify the recovery ability of the proposed method. Figure 10 As shown in Figure 3, the proposed method can effectively remove raindrops. These visualization results demonstrate that the proposed HFGlobalFormer has a stronger ability to remove raindrops and restore texture details by effectively utilizing and fusing global context and high-frequency information.

[0174] 4) Complexity Comparison: The complexity comparison of existing methods and the proposed method is shown in Table II. MAC (Multiply Accumulate Operation) is calculated on an image of size 3×128×128. It can be seen that the proposed method has the lowest computational complexity and the best performance in raindrop removal on compressed images, demonstrating our higher efficiency.

[0175] Table 2 shows the complexity comparison with existing methods. MAC is calculated on a 3×128×128 image.

[0176]

[0177] C. Ablation studies

[0178] Ablation studies are used to evaluate the effectiveness of each important component in the proposed HFGlobalFormer. The same experimental setup as described previously is adopted. SSIM is used to evaluate the performance because it is the loss function of the proposed method and can reflect the ability to recover structural information. In all the proposed LHFT modules of HFGlobalFormer, the high-frequency branches and LHAM are removed to form the backbone network. Then, ordinary depthwise convolution (DC) is used on the backbone network to construct the high-frequency branches instead of the proposed HFDC to study the impact of high-frequency branches that do not have an HF (high-frequency) friendly design, denoted as backbone+DC. In order to evaluate the effectiveness of the proposed high-frequency branch depthwise convolution, the proposed high-frequency branch depthwise convolution is used to replace the ordinary depthwise convolution in the backbone+DC, denoted as backbone+high-frequency branch depthwise convolution. Finally, LHAM is applied on the basis of the backbone+high-frequency branch depthwise convolution to form the proposed overall method backbone+high-frequency branch depthwise convolution+LHAM.

[0179] 1) Effectiveness of high-frequency branches without high-frequency-friendly design: The purpose is to explore the effective collaboration between the self-attention mechanism and convolution to extract low-frequency and high-frequency features. To this end, the values in the local window are fed to the low-frequency and high-frequency branches. The performance of the backbone and backbone + DC are compared to show the effectiveness of the high-frequency branch without high-frequency-friendly design. As can be seen from Table III, the introduction of ordinary high-frequency branches based on depthwise convolution can bring gains on average. However, performance degradation can be observed in QF20, and similar results are also seen in QF40 and QF50. The lack of high-frequency-friendly design leads to limited improvement, so HFDC is developed to improve the ability to extract high-frequency features.

[0180] Table 3 Ablation study of different components of the proposed method (from the perspective of SSIM)

[0181]

[0182] 2) Effectiveness of HFDC: The zero-mean feature introduced in the proposed HFDC can prevent DC bias and focus on high-frequency information. As can be seen from Table III, compared with the backbone and backbone+DC, backbone+HFDC can produce significant improvements in various quality factors, which proves the effectiveness of HFDC.

[0183] To illustrate the effect in the frequency domain, Figure 11 Fourier analysis is shown. For input value feature V and low frequency feature F L and high-frequency features F H The discrete Fourier transform (DFT) is performed for the first block of the first encoder stage for Backbone+DC and Backbone+HFDC, respectively. According to

[13] , the relative logarithmic amplitude of the Fourier transform feature map (the difference between the logarithmic amplitude of 0.0π and 1.0π) is presented. For the convenience of visualization, the semi-diagonal components of the 2D Fourier transform feature map are provided. Figure 11 As shown in Figure 2, RMSA tends to discard high-frequency signals, while both DC and HFDC act like high-pass filters. However, it can be observed that HFDC can amplify high-frequency information better than DC and pays more attention to high-frequency components guided by the zero-mean property.

[0184] 3) Effectiveness of LHAM: The proposed LHAM aims to achieve effective fusion of low-frequency and high-frequency information by assigning channel and window weights to low-frequency and high-frequency branches according to their characteristic features. As shown in Table III, the proposed LHAM can further improve the performance of Backbone+HFDC, which proves its effectiveness. Figure 12 shows the weights w of the first block of the second encoder stage L and w H, where the number of channels is 64 and two windows are selected. It can be seen that the weights of the low-frequency and high-frequency branches vary across channels, with some channels focusing more on low-frequency information and others on high-frequency information. Regarding the weights in different local windows, the specific proportions of complementary branches are different, indicating that the proposed LHAM can adaptively adjust the allocation based on channel properties and window content.

[0185] in conclusion

[0186] When transmission media and compression degradation intersect, new challenges arise. Embodiments of the present invention address the problem of removing raindrops from compressed images, where these raindrops obscure large areas of the background, while compression results in the loss of high-frequency (HF) information. Restoring the former requires global contextual information, while the latter relies on guidance from high-frequency details, leading to a conflict in the design of existing methods when leveraging these two types of information. To address this issue, a novel transformer architecture is proposed that leverages the advantages of an attention mechanism and a high-frequency-friendly design to effectively restore compressed raindrop images at the framework, component, and module levels. Specifically, at the framework level, relative position multi-head self-attention and convolutional layers are integrated into the proposed LHFT. The former captures global contextual information, while the latter focuses on high-frequency information. Their combination effectively addresses the problem of mixed degradation. At the component level, HFDC with a zero-mean kernel is utilized to improve the ability to extract high-frequency features. Finally, at the module level, LHAM is introduced to adaptively distribute the importance of low- and high-frequency information for effective fusion. A dataset of JPEG-compressed raindrop images is constructed, and extensive experiments are conducted at different compression rates. Experimental results demonstrate that the proposed method outperforms existing methods without increasing computational cost.

[0187] In summary, the embodiments of the present invention provide a novel architecture, the so-called HFGlobalFormer, which pursues the effective restoration of compressed images with raindrops. In LHFT, the self-attention mechanism and the convolutional layer effectively cooperate at the framework level to capture complementary global context information and high-frequency information, which is conducive to restoring areas severely occluded by raindrops and high-frequency details lost due to compression. In order to improve the ability to extract high-frequency features, high-frequency decomposition coefficients with zero-mean properties are developed as an important component in LHFT. With the help of LHAM, the effective fusion of low-frequency and high-frequency information is achieved, thereby adapting to various degradation situations of raindrop removal in compressed images. A large number of experimental results show that the proposed method has superior performance compared with existing methods.

[0188] system

[0189] Figure 13An example information processing system 1300 is shown, which can be used to implement one or more methods for restoring a compressed image with raindrops in embodiments of the present invention. Information processing system 1300 generally includes appropriate components required to receive, store, and execute appropriate computer instructions, commands, and / or code. The main components of information processing system 1300 include a processor 1302 and memory 1304. Processor 1302 may include one or more of a CPU, an MCU, a GPU, logic circuits, a Raspberry Pi chip, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other digital or analog circuitry for interpreting and / or executing program instructions and / or processing signals, information, and / or data. Memory 1304 may include one or more volatile memories (e.g., RAM, DRAM, SRAM, etc.), one or more non-volatile memories (e.g., ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.), or any combination thereof. Appropriate computer instructions, commands, code, information, and / or data may be stored in memory 1304. Computer instructions for executing or facilitating the execution of embodiments of the present invention may be stored in memory 1304. Processor 1302 and memory (storage) 1304 may be integrated or separate (and operably connected). Optionally, information processing system 1300 further includes one or more input devices 1306. Examples of such input devices 1306 include a keyboard, a mouse, a stylus, an image scanner, a microphone, a tactile / touch input device (e.g., a touch screen), an image / video input device (e.g., a camera), etc. Optionally, information processing system 1300 further includes one or more output devices 1308. Examples of such output devices 1308 include a display (e.g., a monitor, screen, projector, etc.), speakers, headphones, a printer, an additive manufacturing machine (e.g., a 3D printer), etc. The display may include a liquid crystal display, an LED / organic light emitting diode display, or other suitable display, and may or may not be touch-sensitive. Information processing system 1300 may further include one or more disk drives 1312, which may include one or more of the following: a solid-state drive, a hard drive, an optical drive, a flash drive, a tape drive, etc. A suitable operating system may be installed in the information processing system 1300, for example, on a disk drive 1312 or in the memory 1304. The processor 1302 may operate the memory 1304 and the disk drive 1312. Optionally, the information processing system 1300 further includes a communication device 1310 for establishing one or more communication links (not shown) with one or more other computing devices (e.g., a server, a personal computer, a terminal, a tablet computer, a phone, a watch, an IoT device, or other wireless computing devices).The communication device 1310 may include one or more of the following: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, a Bluetooth transceiver, a radio frequency transceiver, a cellular (2G, 3G, 4G, 5G, 5G+, or similar) transceiver, an optical port, an infrared port, a USB connection, or other wired or wireless communication interface. The transceiver may be implemented by one or more devices (integrated transmitter and receiver, separate transmitter and receiver, etc.). The communication link may be wired or wireless and is used to transmit commands, instructions, information, and / or data. In one example, the processor 1302, memory 1304 (optional input devices 1306, output devices 1308, communication devices 1310, and disk drive 1312 (if any)) are directly or indirectly connected to each other via a bus, a peripheral component interconnect (PCI) such as PCI Express, a universal serial bus (USB), a fiber optic bus, or other similar bus structure. In one embodiment, at least some of these components may be connected wirelessly, such as via a network such as the Internet or a cloud computing network. Those skilled in the art will understand that. Figure 13 The illustrated information processing system 1300 is merely an example, and in other embodiments, the information processing system 1300 may have a different configuration (eg, include additional components, have fewer components, etc.).

[0190] Although not required, one or more embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries for use by developers, or may be included in another software application, such as a terminal or computer operating system or a portable computing device operating system. In one or more embodiments, since program modules include routines, programs, objects, components, and data files that help perform specific functions, skilled artisans will understand that the functionality of the software application may be distributed across multiple routines, objects, and / or components to achieve the same functionality required herein.

[0191] It should also be understood that the methods and systems of the present invention may be implemented entirely or in part by a computing system, and any suitable computing system architecture may be used. This would include stand-alone computers, networked computers, and dedicated or non-dedicated hardware devices. When the terms "computing system" and "computing device" are used, these terms are intended to include, but are not limited to, any suitable arrangement of computers or information processing hardware capable of performing the functions described.

[0192] Those skilled in the art will appreciate that various changes and / or modifications may be made to the described and / or illustrated embodiments of the present invention to provide other embodiments of the present invention. Therefore, the description and / or illustrated embodiments of the present invention should be considered illustrative and not restrictive. Optional features of certain embodiments of the present invention are provided in the abstract and description. Certain embodiments of the present invention may include one or more of these optional features (some of which are not specifically described in the drawings). Certain embodiments of the present invention may lack one or more of these optional features (some of which are not specifically described in the drawings).

Claims

1. A computer-implemented method for restoring a compressed image with raindrops, comprising: Two branches are applied in a complementary manner to capture low-frequency and high-frequency features from the compressed image; Extracting the high-frequency features through high-frequency depth-wise convolution with a zero-mean kernel; as well as The low-frequency features and the high-frequency features are fused by the low-high attention module by adaptively distributing the importance of the branches among channels.

2. The computer-implemented method of claim 1, wherein the low-frequency features include global context information under the raindrops; and the high-frequency features include local high-frequency details that may be lost due to compression.

3. The computer-implemented method according to claim 1, wherein the two branches include a low-frequency branch that extracts the low-frequency features through a self-attention mechanism, and a high-frequency branch that extracts the high-frequency features through the high-frequency depth convolution.

4. A computer-implemented method according to claim 3, wherein the self-attention mechanism comprises relative position multi-head self-attention.

5. The computer-implemented method of claim 1 , wherein extracting the high-frequency features by the high-frequency depthwise convolution comprises: Dividing the input of the high-frequency depth convolution into multiple channels; Apply high-frequency convolution on each channel; as well as Concatenate the features from each channel.

6. The computer-implemented method of claim 5, wherein applying the high-frequency convolution on each channel comprises removing a spatial mean from an initial kernel and applying the resulting kernel to the convolution. 7 . The computer-implemented method of claim 1 , wherein the high-frequency branch comprises a reshape / flatten operation, the high-frequency depthwise convolution, and a 1×1 point-wise convolution.

8. A computer-implemented method according to claim 1, wherein the low-high attention module is a window-level attention scheme that is performed in each local window.

9. The computer-implemented method of claim 8, wherein the process in the local window comprises: Mixing the low-frequency features and the high-frequency features of the two branches to obtain an integrated feature, and reshaping the integrated feature into a spatial feature, which is further fed into a ReLU layer to obtain a mixed feature; Aggregating the mixed features by applying average pooling to each channel to obtain compact features; Obtaining a weight matrix of the low-frequency feature and the high-frequency feature from the compact feature; as well as The low-frequency features and the high-frequency features and their corresponding weight matrices are added channel by channel to generate weighted low-frequency features and weighted high-frequency features, and fusion features are generated by element-by-element addition. 10 . The computer-implemented method according to claim 9 , wherein when generating the fused feature, different channels of the fused feature are configured as different combinations of low-frequency features and high-frequency features.

11. The computer-implemented method of claim 1 , further comprising: Merging the fused features into full-resolution features; as well as The full-resolution features are added to the input features of the compressed image.

12. The computer-implemented method of claim 11 , further comprising: A local augmented feed-forward network is applied on the features to produce output features.

13. The computer-implemented method of claim 12, wherein the locally enhanced feed-forward network is configured to perform dimensional operations and nonlinear processing on the features.

14. A computer-implemented method according to claim 1, wherein the step of applying two branches is performed at the frame level, extracting the high-frequency features through the high-frequency deep convolution is performed at the component level, and fusing the low-frequency features and the high-frequency features through the low-high attention module is performed at the module level.

15. A low-high frequency converter module for performing the computer-implemented method according to claim 1, comprising: Self-attention mechanism for extracting low-frequency features from compressed images; High-frequency depthwise convolution with zero-mean kernel to extract high-frequency features; as well as The low-frequency and high-frequency attention module is used to fuse the low-frequency features and the high-frequency features.

16. A hierarchical U-shaped encoder-decoder network with residual learning for restoring compressed images with raindrops, comprising: Input projection block, used to extract features from the input image; Encoder block, used to extract multi-scale features; The encoder block includes two or more encoder sub-blocks and a bottleneck block, wherein each encoder sub-block includes a sequential stack of low- and high-frequency transformer modules and a downsampling layer for reducing spatial resolution and expanding the channel dimension of features; A decoder module for recovering features; the decoder module comprises two or more decoder sub-modules corresponding to two or more spatial resolutions, wherein each of the decoder sub-modules comprises an upsampling layer for doubling the spatial resolution and reducing the number of channels by half, and a series connection unit for connecting the upsampled features in series with the features from the corresponding encoder sub-module through a jumper connection, and a sequential stack of the low- and high-frequency transformer modules; an output projection module for reconstructing a residual image based on the recovered features; as well as an adding unit for obtaining a reconstructed image by adding the residual image and the input image; Wherein, the low-high frequency converter module comprises the low-high frequency converter module according to claim 15.

17. A system for restoring a compressed image with raindrops, comprising: one or more processors; as well as a memory storing one or more programs configured to be executed by one or more processors; The one or more programs include instructions for performing or facilitating the performance of the computer-implemented method of claim 1 .

18. A non-transitory computer-readable medium having instructions stored thereon; when executed by one or more processors, the instructions cause the one or more processors to perform the computer-implemented method of claim 1.

Images