A degraded image restoration method based on a pollution region mask guide

Abstract

The present application relates to the technical field of computer vision image restoration, and discloses a degraded image restoration method based on a pollution area mask guide; an image containing a pollution area and a pollution area mask are input into a trained restoration model, and a restored image is output; the construction and training method of the restoration model comprises the following steps: the image containing the pollution area and the corresponding pollution area mask are input into a soft area mask generator to obtain a soft area mask; an encoding stage; global information interaction processing of deep features; a decoding stage; the restored feature map is input into an output convolution layer, and a residual connection is established with the image containing the pollution area, and a final restored image is output; the restoration model is trained based on training samples and an overall loss function; the present application not only plays the powerful local feature extraction capability of the convolution operation in the traditional convolutional neural network (CNN), but also realizes feature information interaction in the global range by the Transformer self-attention mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision image restoration, and particularly relates to a degraded image restoration method based on a pollution area mask guide. BACKGROUND

[0002] In the initial imaging and information dissemination process, digital images are easily affected by environmental media, imaging systems and human intervention, resulting in image pixel damage, significant quality degradation, image blur, texture loss, color distortion and other pollution phenomena, i.e. image degradation. Some of the degradation does not occur in the entire image, but shows regional location characteristics, i.e. the pixels in the region are severely polluted and degraded, while the pixels outside the region are generally not affected. Common regional pollution phenomena include shadows, flares and defects. In image classification, image segmentation, target detection and other high-level vision tasks, the quality of the input image has an important influence on the performance and efficiency of the algorithm. The pollution region may cause target deformation, boundary blur, detail loss and other situations, which seriously interfere with the processing of subsequent downstream tasks. For example, in medical image processing, shadows can cause the lesion area to be blurred, image detail information to be lost, and the identification and diagnosis of the target condition to be disturbed; in image semantic segmentation, the semantic recognition of shadow and flare regions may be misjudged, reducing the accuracy of the information; in video image processing, changes in scene and lighting conditions can change the formation and manifestation of shadows and flares, increasing the complexity of related processing algorithms.

[0003] Image restoration, as a basic bottom computer vision task in digital image processing, aims to use software algorithms to alleviate and eliminate the influence of the above-mentioned image degradation, improve the detail quality of the image, eliminate unnecessary interference for subsequent image understanding vision tasks, and provide a more sufficient image information basis. In the current deep learning era, existing image restoration work has shown its charm in various related vision tasks, and the structure design of the model has gradually diversified, with specific task-oriented, and has greatly improved the recovery quality and inference efficiency.

[0004] However, for the image restoration task with partial pollution regions, the existing deep learning-based solutions, whether parameter estimation strategies or end-to-end network architectures, mainly focus on exploring the pixel-level differences between the pollution region and the restored non-pollution version, without considering the consistency and correlation of the texture patterns and other features of the same image in the pollution region and the non-pollution region, resulting in a significant sense of discomfort in color and texture between different regions of the restored image, specifically, the texture information in the pollution region is not fully restored, the lighting is inconsistent (too bright or too dark) compared to the clean region, and there are artifacts in the region edge, the transition is unnatural, and other regional incoordination phenomena.

[0005] The area mask is a kind of binary image, which can effectively represent the positions of the pollution area and the clean area in the original image (the pollution area is by default 1, and the clean area is 0). How to correctly and effectively extract and construct the feature relationship between the pollution area and the clean area by means of the area mask, and use the features of the clean area to help restore the features of the pollution area, is the key to the recovery task of such a pollution area image.

[0006] The present application designs an algorithm module based on area mask guidance to solve the problems and defects in the related image recovery methods. SUMMARY

[0007] To solve the above technical problems, the present application provides a degraded image recovery method based on pollution area mask guidance. This method can be adapted to specific image recovery tasks containing partial pollution areas through model training. Due to the targeted and related feature extraction and information interaction of different areas in the image, the present application makes full use of the feature information of the clean area to help the step-by-step reconstruction and recovery of each pixel in the pollution area. Compared with the current mainstream area degradation image recovery method, the present application can obtain a model with stronger expandability and better generalization ability.

[0008] To solve the above technical problems, the present application adopts the following technical solutions:

[0009] A degraded image recovery method based on pollution area mask guidance inputs the image containing pollution area and the pollution area mask into the trained recovery model, and outputs the recovered image; the recovery model includes a soft area mask generator, an encoder, a multi-scale channel attention module and a decoder, and the encoder and the decoder adopt a U-Net type architecture; the construction and training method of the recovery model includes the following steps:

[0010] Step one, input the image I in containing pollution area and the corresponding pollution area mask M in into the soft area mask generator G softmask (·) to obtain the soft area mask

[0011] Step two, encoding stage: input the pollution image I in , the corresponding pollution area mask M in and the soft area mask The convolution operation is performed in the channel dimension to obtain an initial feature map F0; the initial feature map is sequentially subjected to a plurality of levels of encoding blocks in the encoder, and a plurality of times of feature extraction is performed under the guidance of the pollution area mask to obtain a shallow feature map; the encoding block is composed of a region-level guidance module and a down-sampling operation; the region-level guidance module guides the extraction of the feature of the pollution area by taking the feature information of the non-pollution area as an affine transformation parameter;

[0012] Step three, global information interaction processing of deep features; the shallow feature map output by the encoder is input into a plurality of cascaded pixel-level guidance modules, and a pixel-level restored feature map is obtained under the guidance of the soft area mask; the pixel-level guidance module is improved on the basis of the visual Transformer for the pollution area, and based on the guidance of the soft area mask, only the attention mechanism of the relevant pixels in the non-pollution area to the pollution area needs to be constructed, without the need to obtain the relevance of all pixels in the global range;

[0013] Step four, decoding stage: the pixel-level restored feature map is sequentially subjected to a plurality of levels of reversible network modules in the decoder, and a plurality of times of feature restoration is performed under the guidance of the pollution area mask to obtain a restored feature map; the reversible network module is realized based on the region-level guidance module; the encoding block and the reversible network module are connected through a multi-scale channel attention module;

[0014] Step five, inputting the restored feature map into an output convolution layer, and outputting a final restored image I in Constructing a residual connection to output the final restored image

[0015] Step six, training the restoration model based on training samples and an overall loss function; the training samples include an image containing a pollution area, a pollution area mask, and a clean image not containing a pollution area; the overall loss function includes a pixel-level loss function a perception loss function a boundary gradient loss function a frequency domain loss function and a soft area mask loss function λ pix , λ per , λ grad , λ freq , λ soft are balance parameters.

[0016] Further, the region-level guidance module guides the extraction of the feature of the pollution area by taking the feature information of the non-pollution area as an affine transformation parameter, specifically including:

[0017] taking the feature map as input, and the scaled pollution area mask As the prior region information, H, W and C represent the height, width and channel number of the input feature map respectively;

[0018] Firstly, the region-level guiding module splits the feature map F in into two feature maps F in1 and F in2 in the channel dimension, which satisfies For the feature map F in1 , it is split into the polluted region feature map F in and the non-polluted region feature map F in the spatial dimension according to the corresponding pollution region mask M , that is:

[0019]

[0020] Then, F is respectively standardized by the embodiment: the mean and variance extracted from the non-polluted region feature are used to perform affine transformation on the polluted region feature, that is, for each feature point p located at (h, w, ch) in the polluted region, the specific affine transformation process is as follows:

[0021]

[0022] where p h，w，ch and p represent the original value and the value after affine transformation of the point p, and represent the mean and standard deviation of the polluted region in the feature channel dimension, and are the mean and standard deviation of the non-polluted region, which are calculated as follows:

[0023]

[0024] |M in = 0| represents the number of pollution region mask values equal to 0, that is, the number of non-polluted region pixels, and |M in = 1| represents the number of polluted region pixels;

[0025] Then, the two region feature maps after transformation are recombined in the spatial dimension to obtain the feature map F

[0026] Subsequently, the feature map F in2 is re-spliced with the feature map F in the channel dimension to obtain the feature map F Finally, the feature map F out is subjected to convolution and activation function operation, and the straight-through feature map F By connecting the residual structures, the final feature map output by the region-level guidance module is obtained.

[0027] Furthermore, the pixel-level guidance module improves upon the visual Transformer for contaminated regions. Based on the guidance of the contaminated region mask, it only needs to construct an attention mechanism from the relevant pixels in the uncontaminated region to the contaminated region, without needing to obtain the correlation of all pixels globally. Specifically, it includes:

[0028] With feature map As input to the pixel-level guide module, it corresponds to the proportionally scaled soft contamination area mask. As prior information, H, W, and C represent the height, width, and number of channels of the input feature map, respectively;

[0029] First, regarding X in Position encoding is performed to obtain feature map X p X p Each element x in p =x in +pe, where pe is obtained from the 2D positional encoding of the image:

[0030]

[0031] Where x and y represent the horizontal and vertical positions of the pixel, and i, j∈[0, C / 4) specifies the corresponding dimensions; each dimension of the position encoding is composed of a sinusoidal signal with a specific frequency and phase, and the wavelength range increases geometrically from 2π to 10000·2π; the first half of the full dimension encodes the horizontal position x, and the second half encodes the vertical position y.

[0032] Next, the feature map X is processed on each channel. p Perform a tiling operation to obtain N is the number of pixels in the feature map;

[0033]

[0034] Then, After three sets of matrix multiplications, a linear transformation yields the query matrix Q, the key matrix K, and the value matrix V; note that the contaminated area mask is inverted here. And it applies to operations on the key matrix:

[0035]

[0036] in The learnable linear transformation matrix; the resulting query matrix will then be... With the transposed key matrix Perform a dot product and introduce a temperature coefficient. The attention correlation matrix is obtained after balancing the weight values

[0037]

[0038] The feature map after information interaction is obtained by normalizing the attention relationship by using the softmax function and multiplying the value matrix V

[0039]

[0040] The The feature map is obtained by using the linear transformation of the feedforward network FFN for further feature extraction and integration The FFN is composed of two linear fully connected layers and a ReLU activation function in the middle, that is:

[0041]

[0042] The residual connection of the input feature map X in is constructed to obtain the feature map Y out output by the pixel-level guidance module:

[0043]

[0044] Further, step one specifically includes:

[0045] First, calculate the image I in containing the contaminated area and the clean image I gt corresponding to the contaminated area without the contaminated area diff :

[0046]

[0047] Where c is the RGB color channel; then the normalized function N(·) is used to normalize each color channel of I diff ; finally, the soft region mask reference value is obtained by calculating the average value

[0048]

[0049] The element value range of M softmaxsk is [0, 1], and M softmaxsk is used as the learning and training target of the soft region mask generator;

[0050] The soft region mask generator G softmask (·) uses the U-Net structure to obtain the image I in containing the contaminated area and the corresponding contaminated area mask Min To learn and generate a reference value M that approximates the soft region mask. softmask soft area mask

[0051] Furthermore, the step of making skip connections between the encoded block and the reversible network module through the multi-scale channel attention module specifically includes:

[0052] The feature map input to the multi-scale channel attention module is denoted as The multi-scale channel attention module includes global and local branches; feature map X in In the global branch, the feature map is first obtained through global average pooling. Local branches and feature map X in Maintain consistency Subsequently, both the global and local branches undergo pointwise convolution operations at the bottleneck layer, first reducing the dimensionality of the features and then increasing it, i.e.:

[0053]

[0054] In this process, all convolutions use a 1×1 kernel. Convolution operations Conv1 and Conv3 first reduce the dimensionality of the feature map to C / 4 channels, and then convolution operations Conv2 and Conv4 increase the dimensionality back to the original number of channels C. This process passes through the local features after the bottleneck layer. Preservation and feature map X in Same size; global features after the bottleneck layer Then, scalar values ​​are used to reflect the full image information of each feature channel;

[0055] Subsequently, and Aggregation and weight allocation are performed to obtain the feature map after feature recalibration.

[0056]

[0057] in, This represents the weights of each feature point constructed by the multi-scale channel attention module. This indicates a broadcast addition operation. This indicates element-wise multiplication.

[0058] Compared with the prior art, the beneficial technical effects of the present invention are:

[0059] This invention addresses the common problem of insufficient interaction of regional feature information in current methods for restoring degraded images with regional degradation. It proposes a novel method based on contaminated region mask guidance. The core of this method comprises two feature extraction and interaction modes: adaptive guided restoration using regional statistical information and self-attention guided restoration using pixel-level long-range highly correlated features. Through these two modes, this invention fully utilizes the information from the region mask throughout the entire algorithm process. It not only leverages the powerful local feature extraction capabilities of convolutional operations in traditional Convolutional Neural Networks (CNNs) but also achieves global feature information interaction through the Transformer's self-attention mechanism. Experimental results show that this method can achieve high-quality image restoration in shadow removal and flare removal tasks with relatively low parameter and computational costs, and demonstrates good generalization and scalability within a reasonable range of contaminated and clean region proportions. Furthermore, based on the region mask guidance approach, this invention has the potential to support personalized user needs and can be effectively embedded and extended to other related image restoration tasks. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the overall process of the degraded image restoration method based on contaminated area masking according to the present invention;

[0061] Figure 2 This is a schematic diagram of the regional-level guidance module of the present invention;

[0062] Figure 3 This is a schematic diagram of the pixel-level guidance module of the present invention;

[0063] Figure 4 This is a schematic diagram of the reversible network module of the present invention;

[0064] Figure 5 This is a schematic diagram of the structure of the multi-scale channel attention module of the present invention. Detailed Implementation

[0065] This invention proposes a method for restoring degraded images based on a mask-guided approach for contaminated areas. The overall idea is as follows:

[0066] 1. In order to solve the problem of insufficient feature extraction and information interaction between image regions (clean region and pollution region), two guide modules based on region mask are designed, which realize feature processing at region level and pixel level. Among them, the region level guide module applies the statistical characteristics of the clean region feature to the pollution region feature in an adaptive affine manner from the perspective of the whole region, realizing regional recovery; the pixel level guide module uses self-attention mechanism to accurately match the high correlation features from the non-pollution region for global pixel-by-pixel recovery. The two modules make full use of the information of the region mask to build the relationship between the clean non-pollution region and the pollution region in the image, and establish the feature information transmission path from "clean to pollution".

[0067] 2. The overall network framework is in the classic "encoder-decoder" mode, which combines the advantages of traditional convolutional neural network (CNN) architecture and Transformer architecture in local feature and global information extraction and interaction, and further designs a soft region mask generator, a reversible network module and a multi-scale channel attention module to empower the model, realizing a more complete and comprehensive deep learning image restoration algorithm based on region mask guidance.

[0068] 3. Through sufficient experiments and analysis, the effectiveness of the overall framework and the region mask guidance module is proved, which can realize high-quality image restoration in shadow removal and flare removal tasks with low parameter amount and operation amount, and has the potential to expand to other related image restoration tasks with regional pollution.

[0069] The degraded image restoration method based on pollution region mask guidance in the application comprises the following steps:

[0070] Step 1: Construct a guide module based on region mask, including image feature extraction and information interaction process at region level and pixel level.

[0071] (1-1) The region level guide module COAN is mainly used in the encoding and decoding stages of the algorithm. Through the structure of feature segmentation and fusion in channel dimension and spatial dimension, part of the original content feature information is better preserved, the information loss caused by the data deviation between the pollution region and the clean region before standardization is reduced, and based on the guidance of the region mask, the feature information of the non-pollution region is used as the affine transformation parameter to guide the pollution region feature, which strengthens the information interaction between regions to maintain statistical consistency, and significantly improves the image restoration performance of the pollution region.

[0072] (1-2) Pixel-level guidance module CAAT, mainly used for the global information interaction processing stage of deep features. CAAT is optimized and improved on the basis of the basic visual Transformer, and based on the guidance of soft region mask, only needs to build the attention mechanism from the high correlation pixels in the non-polluted area to the polluted area, without obtaining the correlation of all pixels in the global range, not only realizes the accurate recovery of point-to-point at the pixel level, but also reduces the calculation cost of the model and improves the calculation efficiency.

[0073] Step 2: Construct an auxiliary module, including a soft region mask generator, a reversible network module and a multi-scale channel attention module.

[0074] (2-1) Soft region mask generator, used in the global information interaction processing stage of deep features. The soft region mask generator uses the traditional small U-Net structure to learn and generate a high-quality soft region mask close to the reference value by inputting the original polluted image and its corresponding polluted region hard threshold mask. The generated soft region mask can effectively measure the size of the pixel pollution degree, and provide more valuable regional representation for the pixel-level guidance module CAAT.

[0075] (2-2) Reversible network module, used in the decoding stage of the algorithm. The reversible network module is realized based on the general affine coupling layer of the reversible neural network for image processing designed by RealNVP. Among them, the region-level guidance module designed symmetrically with the encoding process is used as the transformation function, and through the characteristic mapping of the bijection function of its forward and reverse processes, it can not only ensure that the deep feature map does not lose detailed information in the progressive recovery process, but also can reduce the model memory occupation to a certain extent, while cooperating with the upsampling and cross-layer jump connection operation, gradually perfecting and restoring the image detail features.

[0076] (2-3) Multi-scale channel attention module, used in the cross-layer jump connection of the model. By introducing a lightweight attention mechanism to the feature maps of each layer in the encoding process, the multi-scale channel attention module assigns corresponding weights to the features, selects valuable detailed information, and helps the network model to better retain and fuse the feature maps of high-resolution rich spatial information and low-resolution high-semantic information, and gradually restores the image features in the decoding stage.

[0077] Step 3: Build an overall image restoration model. Embed the above region mask-based guidance module and various auxiliary modules into the U-Net-type network of the "encoder-decoder" architecture. The process mainly includes three stages: the encoding stage of shallow feature extraction, the global information interaction processing stage of deep features, and the decoding stage of image progressive recovery, as follows:

[0078] (3-1) In the encoding process, the image local feature extraction is realized based on two-dimensional convolution and max pooling downsampling operation, the feature map spatial size is gradually reduced, and the regional level guide module COAN is introduced in the standardization operation, the statistical information of the non-polluted area is used as the affine transformation parameter of the polluted area, so that the statistical consistency of the feature distribution in the polluted area and the clean area is ensured.

[0079] (3-2) In the deep feature processing, first, the soft region mask generator is used to measure the pollution degree of the damaged area of the image, and the edge of the polluted area is characterized more finely, while avoiding the problem of information loss in the subsequent feature transmission process caused by the "01" hard threshold mask. Further, a remote feature modeling is realized by cascading multiple pixel-level guide modules based on the regional mask, so as to select the pixels with high correlation from the non-polluted area to help the feature point recovery of the polluted area.

[0080] (3-3) In the decoding stage, a reversible network module and an upsampling operation are used to construct a symmetric structure consistent with the encoding stage. The reversible network module can preserve the data detail information recovered layer by layer and reduce the memory occupation, so that the training is more stable. In addition, in the cross-layer skip connection between encoding and decoding, a multi-scale channel attention module is used to better fuse the shallow and deep features with different semantic information, so that the image recovery result is more perfect.

[0081] Step 4: Obtain paired data for model training. Since the modules and the overall image restoration algorithm introduced in the present application need the support of the pollution area mask, all paired data contain image triplets, i.e. images containing pollution areas, clean images without pollution and corresponding pollution area masks. After data enhancement including random operations such as rotation, scaling, translation, skewing and flipping, the images are cropped to 256x256 image size and input into the image restoration model.

[0082] Step 5: Design the optimization target of the network model, including the pixel-level loss function perception loss function boundary gradient loss function frequency domain loss function and soft region mask loss function

[0083] wherein, λ pix , λ per , λ grad , λ freq , λ soft are balance parameters in the network optimization target.

[0084] Step 6: The training process is divided into two steps. Firstly, a preheating and periodic cosine annealing learning rate variation scheme and a sharpness-aware minimization (SAM) optimizer based on Adam are used. The preheating operation and the periodic learning rate variation can help the model to be more stable during the training process, prevent the model from overfitting and falling into a local optimal solution, and obtain a preliminary trained model. Subsequently, the preliminary model is further retrained and optimized by using the Adam optimizer and a dynamic learning rate update strategy based on the setting of the stagnation period to obtain the final model.

[0085] Embodiments

[0086] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further illustrate the present application in conjunction with relevant embodiments and with reference to the accompanying drawings.

[0087] Step 1: Construct a guidance module based on a region mask, including an image feature extraction and information interaction process at the region level and the pixel level.

[0088] (1-1) Construct a region-level guidance module COAN, which is used in the encoding and decoding stages of the algorithm, and its structure is as shown in Figure 2 The feature map is taken as the input, and the scaled pollution region mask is taken as the prior region information (m = 1 represents the pollution region, and m = 0 represents the non-pollution region). H, W and C represent the height, width and channel number of the input feature map, respectively.

[0089] Firstly, COAN uniformly divides the feature F in into two parts F in1 and F in2 in the channel dimension, which satisfies For the F in1 feature, it is divided into a pollution region in and a clean non-pollution region in the spatial dimension according to the corresponding pollution region mask M , that is:

[0090]

[0091] Then, the two regions are respectively standardized. Compared with the region standardization which independently operates on different regions and ignores any details and semantic connections between region features, the instance standardization used in the present application does not introduce a learnable affine parameter, but only performs affine transformation on the pollution region feature by using the mean and variance extracted from the non-pollution region feature, that is, for each feature point p located at (h, w, ch) in the pollution region, the specific affine transformation process is as follows:

[0092]

[0093] where p h，w，ch and denote the original value and the value after affine transformation of the point p, and denote the mean and standard deviation of the pollution region in the feature channel dimension, and are the mean and standard deviation of the non-pollution region, which are calculated as follows:

[0094]

[0095] |M in = 0 means the number of pixels in the pollution region mask whose value is equal to 0, i.e., the number of non-pollution region pixels, and vice versa |M in = 1 means the number of pixels in the pollution region.

[0096] Then, the transformed two region features are recombined in the spatial dimension to obtain

[0097] Subsequently, another part of F in2 with the original features is recombined with the normalized and affine-transformed in the channel dimension to obtain which contains both normalized features and context feature information. Finally, F out is subjected to convolution and activation function operations, and the straight-through feature that has undergone 1x1 convolution is output in a residual structure to obtain the corresponding result Y out :

[0098]

[0099] (1-2) The pixel-level guidance module CAAT is constructed for the global information interaction processing stage of deep features, and its structure is shown in Figure 3 . Similarly, the feature map is taken as input, and the scaled soft pollution region mask is taken as prior information (the soft pollution region mask is generated by the soft region mask generator, and the pollution region value tends to 1 and the clean region value tends to 0), H, W and C represent the height, width and channel number of the input feature map, respectively.

[0100] First, the position encoding is performed on X in to obtain X p , where each element x p = x in + pe, and pe is obtained by 2D position encoding of the image:

[0101]

[0102] where x and y represent the horizontal and vertical positions of the pixel, i, j ∈ [0, C / 4) specify the corresponding dimensions. Each dimension of the position encoding is composed of a sinusoidal signal with a specific frequency and phase. The wavelength ranges from 2π to 10000·2π geometrically. The first half of the full dimension encodes the horizontal position x, and the second half encodes the vertical position y.

[0103] Next, X p The feature map is tiled on each channel to obtain N is the number of feature map pixels.

[0104]

[0105] Subsequently, the After three sets of matrix multiplication, the query matrix Q, the key matrix K, and the value matrix V are obtained by linear transformation. Note that the pollution area mask is inverted here and acts on the operation of the key matrix:

[0106]

[0107] where is a learnable linear transformation matrix. Then the obtained query matrix is multiplied by the transposed key matrix and the temperature coefficient is introduced. After balancing the weight values, the attention relevance matrix

[0108]

[0109] Further, after normalizing the attention relationship using the softmax function, the output after updating the information interaction is obtained by multiplying it with the value matrix V

[0110]

[0111] In order to ensure that the number of channels before and after the module is consistent, generally set d = C. Intuitively, attention operation measures the similarity between query elements and each key element, outputs the sum of value elements weighted based on similarity, and completes the relevant information transmission. If the similarity is high, the corresponding more critical feature information is extracted from the value vector.

[0112] Finally, the output of the attention layer is further feature extraction and integration using the linear transformation of the feedforward network FFN. FFN is composed of two linear fully connected layers and ReLU activation function in the middle, that is:

[0113]

[0114] And then reconverted to feature map form on each channel, resulting in Constructing residual connections with the input allows the network to better learn differential features:

[0115]

[0116] Step 2: Construct auxiliary modules, including a soft region mask generator, a reversible network module, and a multi-scale channel attention module.

[0117] (2-1) Constructing a soft region mask generator G softmaxsk (·) is used in the global information interaction processing stage of deep features to measure the degree of pixel contamination and provide more valuable regional representations for the pixel-level guidance module.

[0118] First, calculate the input contaminated image I. in Corresponding real, unpolluted image I gt Differences in RGB color channels I diff :

[0119]

[0120] Where c represents each color channel of the RGB spectrum. Then, the normalization function N(·) is applied to I. diff Normalize each color channel N(I) = (II) min ) / (I max -I min ), I min and I max These are the minimum and maximum values ​​of image I in the corresponding color channel. Finally, the reference value for the soft region mask is obtained by calculating the average value.

[0121]

[0122] M softmask The element values ​​range from [0,1], and are used as the learning and training target for the soft region mask generator.

[0123] Soft region mask generator G softmask (·) Utilizing a traditional small U-Net structure, by inputting the original contaminated image I in and its corresponding contaminated area hard threshold mask M in To learn and generate a value close to the reference value M softmaask High-quality soft area mask

[0124] (2-2) Construct a reversible network module for the decoding stage of the algorithm. The reversible network module is implemented based on the general affine coupling layer of the reversible neural network for image processing designed by RealNVP. The architecture diagram is as shown in Figure 4 The specific operation is as follows:

[0125] For the forward process, the input of the reversible network module after 1x1 convolution is First, split into two parts in the channel dimension And

[0126]

[0127] The Split(·) operation is a uniform split in the channel, that is, Then, the addition transformation is performed on And the enhanced affine transformation is used for :

[0128]

[0129] Where φ(·), ρ(·) and η(·) all use the COAN module to enhance the regional guidance recovery. In addition, in order to avoid the exponential explosion caused by the exp(·) function, this chapter uses the Sigmoid(·) function in the ρ(·) function, and adds a scale item a to control the size of the exponential. Finally, the two parts of the output are spliced together as the output X out of the reversible network module.

[0130]

[0131] Similarly, the reverse process of the reversible network module is as follows:

[0132]

[0133] (2-3) Construct a multi-scale channel attention module for the cross-layer skip connection of the model, to better preserve and fuse the feature maps of high-resolution rich spatial information and low-resolution high semantic information.

[0134] The architecture of the multi-scale channel attention module is as shown in Figure 5 Two branches of global and local methods are used to extract channel attention, respectively, where the global feature processing link preferentially uses global average pooling to extract relevant attention, and the local feature processing link directly convolves to extract information.

[0135] The input The global branch first passes through the global average pooling to obtain The local branch maintains the same input Subsequently, two branches are respectively processed by the bottleneck layer point-wise convolution operation, and the feature is first reduced and then increased in dimension, that is:

[0136]

[0137] wherein each convolution is a 1x1 convolution kernel, Conv1 and Conv3 first reduce the feature map to C / 4 channel numbers, and then Conv2 and Conv4 increase to the original channel number C. The local feature after the bottleneck layer maintains the same size as the input, and can retain and highlight the subtle details in the shallow feature; the global feature reflects the global information of each feature channel with a scalar value, and intuitively analyzes the relevance and importance of each feature channel to learn the global channel attention.

[0138] Subsequently, the module aggregates and weights the global channel feature information and the local channel feature information to obtain the improved feature map

[0139]

[0140] wherein, represents the weight of each feature point constructed by the multi-scale channel attention module, represents the broadcast addition operation, represents the element-wise multiplication operation.

[0141] Step 3: Construct the overall image restoration model. Embed the above-mentioned region mask-based guidance module and various auxiliary modules into the U-Net type network of the "encoder-decoder" architecture, as shown in Figure 1 . Among them, Figure 1 the upper part represents the soft region mask generation process, the middle frame and the lower right frame simply describe the CAAT and COAN modules proposed in this paper, and the corresponding dashed line is the guidance process based on the region mask information. The process mainly includes three stages: the encoding stage of shallow feature extraction, the global information interaction processing stage of deep feature, and the decoding stage of image step-by-step recovery, which are described as follows:

[0142] (3-1) First, the input contaminated image and the corresponding contaminated region mask are processed by the soft region mask generator G softmask (·) to obtain the soft region mask

[0143] (3-2) Then, in the encoding stage of the network model, the contaminated image I in and the corresponding contaminated region mask M in ​and soft region mask In the channel dimension, merge, input into the initial convolutional layer, and upgrade to C=64 feature channels through 3x3 convolution, get the initial feature map

[0144] (3-3) input F0 into the first COAN module, and based on the guidance of the pollution region mask , realize the first level feature extraction of region level guidance according to step 1-1, get Then, Downsampled by max pooling, get

[0145] (3-4) Similarly, input F1 into the second COAN module, and based on the guidance of the pollution region mask after equal proportion scaling, realize the second level feature extraction of region level guidance, get Then, Downsampled by max pooling, get

[0146] (3-5) input F2 into the third COAN module, and based on the guidance of the pollution region mask after equal proportion scaling, realize the third level feature extraction of region level guidance, get

[0147] (3-6) Then, in the deep feature processing stage, first input into the first CAAT module, and based on the guidance of the pollution region mask after equal proportion scaling, realize the feature extraction and information interaction of pixel level guidance according to step 1-2, get

[0148] (3-7) Through cascading multiple CAAT modules (default 4), repeat step 3-6, get the feature map of pixel level recovery

[0149] (3-8) Then, in the decoding stage of the network model, input Y2 into the first COAN-based reversible network module, and based on the guidance of the pollution region mask , realize the first level feature recovery of region level guidance according to step 2-2, get

[0150] (3-9) input F2 obtained in step 3-4 through multi-scale channel attention module, get the feature map after feature recalibration Merge in the channel dimension , get

[0151] (3-10) Similarly, input Y1 into the second COAN-based reversible network module, and based on the guidance of the pollution region mask , realize the second-level feature recovery of the region-level guidance, and obtain

[0152] (3-11) The F1 obtained in step 3-3 is input into a multi-scale channel attention module to obtain a feature map after feature re-labeling , which is merged in the channel dimension with , and an up-sampling operation after deconvolution is performed to obtain

[0153] (3-12) Y0 is input into the third COAN-based reversible network module, and based on the guidance of the pollution region mask , realize the third-level feature recovery of the region-level guidance, and obtain

[0154] (3-13) Finally, input into the output convolutional layer, and perform 3x3 convolution to reduce the dimension to C=3 feature channels to obtain , and construct a residual connection with the input pollution image I in to obtain the final output recovery image

[0155] The above is a simple model architecture and process of the present application. In addition, the overall algorithm has strong expandability. According to actual needs, the feature channel dimension can be further increased, the network structure can be deepened, and operations such as stacking COAN and CAAT two guidance modules can be performed to obtain more superior and powerful image restoration effects.

[0156] Step 4: Obtain paired data for model training. Since the modules and the overall image restoration algorithm introduced in the present application require the support of the pollution region mask, all paired data contain image triplets, i.e. images containing pollution regions, clean images without pollution, and corresponding pollution region mask images. After data enhancement including random operations such as rotation, scaling, translation, skewing, and flipping, the images are cropped to 256x256 image size and input into the image restoration model.

[0157] Step 5: Design the optimization target of the network model, including the pixel-level loss function perception loss function boundary gradient loss function frequency domain loss function and soft region mask loss function

[0158] where λpix , λ per , λ grad , λ freq , λ soft is a balance parameter in network optimization target.

[0159] (5-1) Pixel-level loss function

[0160] the decontamination image output by the network model and the real non-contaminated image I gt The pixel-level loss is calculated using the L1 distance between them:

[0161]

[0162] (5-2) Perceptual loss function

[0163] According to the pre-trained VGG19 network on ImageNet, the perceptual feature difference between the output image and the real image I gt is calculated:

[0164]

[0165] (5-3) Boundary gradient loss function

[0166] For the recovery quality of the boundary of the contaminated area image, based on the idea of Poisson image editing, the boundary gradient loss function is defined:

[0167]

[0168] where represents the Laplacian gradient operator, MSE(·) calculates the mean square error between the predicted value and the true value in the corresponding area, is obtained by morphological processing of the contaminated mask M in by 7 pixels. maintains the gradient domain between the output image and the input image I in in the non-contaminated area, and minimizes the gradient domain difference between the output image and the target real image I gt in the contaminated area.

[0169] (5-4) Frequency domain loss function

[0170] the decontamination image output by the network model and the real non-contaminated image I gt, first a 2D discrete Fourier transform is performed, converting to its frequency representation:

[0171]

[0172] and the frequency domain loss function is computed as follows:

[0173]

[0174] (5-4) Soft region mask loss function

[0175] In the auxiliary module - soft region mask generator, to generate high-quality soft region masks close to the reference value, a soft region mask loss based on L1 distance is introduced:

[0176]

[0177] where, is the soft region mask generator G softmask (·) learned and generated soft region mask, M softmask the contaminated image I input by the model in and the real non-contaminated image L gt calculated by F(·) described in step 2-1.

[0178] Step 6: The experimental training process is divided into two steps, first using the periodic cosine annealing learning rate variation scheme with preheating and the sharpness-aware minimization SAM optimizer based on Adam. The number of iterations in the preheating period is 10 times, the initial learning rate of the periodic cosine annealing base-lr = 0.002, the initial period T0 = 50, the period extension factor T mult = 2, the specific calculation is as follows:

[0179]

[0180] where, i represents the number of cycle restarts, when the number of iterations T cur reaches T i , the learning rate takes the minimum value η t = η min , then the cycle is reset T cur = 0, and the learning rate takes the maximum value η t = η maxThe preheating operation and the periodic learning rate change can help the model to be more stable during the training process, prevent the model from overfitting and falling into a local optimal solution, and obtain a preliminary trained model. Subsequently, the Adam optimizer and the dynamic learning rate update strategy based on the stagnation period are further used for optimization to obtain the final model. The weight-decay of the Adam optimizer is 0.01, β1 is 0.9, β2 is 0.99, the initial learning rate base-lr is 0.0005, the stagnation period is patience = 25, and the learning rate is gradually reduced by a factor of 0.5. The number of model training iterations is empirically set to 300 times.

[0181] Step 7, model evaluation:

[0182] In order to objectively evaluate the model performance and algorithm effectiveness of the present application, the present application and the shadow removal task algorithms BMNet, AEFNet, DHAN and the general image restoration algorithms HINet, Uformer, Restormer are quantitatively evaluated on the shadow image dataset (ISTD, ISTD+ and SRD, etc.) and the flare image dataset (Flare7K, etc.). The mean absolute error (MAE), the peak signal-to-noise ratio (PSNR), the structural similarity (SSIM) and the learned perceptual image patch similarity (LPIPS) are selected as the four performance indicators for numerical evaluation. In the quantitative analysis, the present application achieves the optimal performance in both the shadow removal and the flare removal tasks of the regional pollution image restoration. The present application not only has a guiding restoration effect on the pollution region, but also has a global color brightness learning ability. It can still fit the relevant mapping for the color difference of the non-pollution region in the paired data set, and has certain adaptability and robustness to complex and variable environmental conditions, and has stronger feature learning and representation ability. In addition, the present application effectively utilizes the information of the region mask, and uses the non-pollution region to guide the image restoration of the pollution region. The target of each module in the network training is clear, and there is no need to introduce additional network parameters and redundant information accumulation. Therefore, the overall network framework only uses about 7% of the parameter amount and 5%-20% of the operation amount of other related comparative methods, and achieves the optimal performance of image restoration.

[0183] It will be obvious to a person skilled in the art that the application is not limited to the details of the foregoing exemplary embodiments and can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. The embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein and no

[0184] Furthermore, it should be understood that although the description is made on the basis of the embodiments, not every embodiment contains only one independent technical solution, and the description is made in this way only for the sake of clarity, and a person skilled in the art should consider the description as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by a person skilled in the art.

Claims

1. A method for degraded image restoration based on a pollution region mask guidance, characterized in that, The image containing the contaminated area and the contaminated area mask are input into the trained restoration model, and a restored image is output; the restoration model comprises a soft area mask generator, an encoder, a multi-scale channel attention module and a decoder, and the encoder and the decoder adopt a U-Net type architecture; The construction and training method of the restoration model comprises the following steps: Step one, generate an image I in corresponding to the contaminated region mask M in through the soft region mask generator G softmask (·), to obtain the soft region mask Step two, encoding stage: the contaminated image I in , the corresponding contaminated region mask M in , and the soft region mask In the channel dimension, the convolution operation is performed to obtain an initial feature map F0; the initial feature map is sequentially subjected to encoding blocks at multiple levels in the encoder to perform multiple feature extractions under the guidance of the contaminated region mask to obtain a shallow feature map; the encoding block is composed of a region-level guidance module and a down-sampling operation; the region-level guidance module guides the extraction of the feature of the contaminated region by taking the feature information of the non-contaminated region as an affine transformation parameter. Step three, global information interaction processing of deep features; the shallow feature map output by the encoder is input into a plurality of cascaded pixel-level guiding modules to obtain a pixel-level restored feature map under the guidance of the soft area mask; the pixel-level guiding module is improved based on visual Transformer for the contaminated area, and based on the guidance of the soft area mask, only the attention mechanism from the relevant pixels in the non-contaminated area to the contaminated area needs to be constructed, without obtaining the relevance of all pixels in the global range; Step four, decoding stage: the pixel-level restored feature map is sequentially input into a plurality of reversible network modules in the decoder to perform multiple feature restoration under the guidance of the contaminated area mask to obtain a restored feature map; the reversible network module is realized based on the region-level guiding module; the multi-scale channel attention module is used to connect the encoding block and the reversible network module by jumping; Step five, input the restored feature map into the output convolutional layer, and combine it with the image I containing the pollution area in Construct a residual connection, output the final restored image Step six, training the restoration model based on training samples and an overall loss function; the training samples include images containing contaminated areas, contaminated area masks, and clean images without contaminated areas; the overall loss function includes a pixel-level loss function perception loss function boundary gradient loss function frequency domain loss function and soft area mask loss function λ pix , λ per , λ grad , λ freq , λ soft is a balance parameter.

2. The degraded image restoration method based on a pollution region mask guide according to claim 1, characterized in that, The region-level guiding module uses the feature information in the non-contaminated area as an affine transformation parameter to guide the extraction of the feature in the contaminated area, and specifically comprises: with feature maps as input, the scaled pollution region mask as prior region information, H, W and C represent the height, width and channel number of the input feature map, respectively; First, the region-level guiding module divides the feature map F in into two feature maps F in1 and F in2 in the channel dimension, satisfying F in1 , For the feature map F in1 , it is divided into a pollution region feature map F and a non-pollution region feature map F in the spatial dimension according to the corresponding pollution region mask M in , that is: Then, the standardization of the pollution region is performed by using the mean and variance of the non-pollution region. The mean and variance of the non-pollution region are extracted and used to perform affine transformation on the pollution region. For each feature point p in the pollution region at (h, w, ch), the affine transformation is performed as follows: where p h，w，ch and denote the original and the affine-transformed value of point p, and denote the mean and standard deviation of the pollution region at the feature channel dimension, and are the mean and standard deviation of the non-pollution region, computed as follows: M in = 0 | represents the number of pixels in the pollution region mask with a value equal to 0, i.e. the number of non-pollution region pixels, |M in = 1 | represents the number of pixels in the pollution region. The two transformed region feature maps are then recombined in the spatial dimension to obtain a feature map Subsequently, the feature map F in2 is connected with the feature map F is spliced in the channel dimension to obtain a feature map F Finally, the feature map F out is connected with the feature map F is connected in a residual structure to obtain a feature map Y out : 3.The degraded image restoration method based on a pollution region mask guiding according to claim 1, wherein, The pixel-level guiding module is improved based on visual Transformer for the contaminated area, and based on the guidance of the contaminated area mask, only the attention mechanism from the relevant pixels in the non-contaminated area to the contaminated area needs to be constructed, without obtaining the relevance of all pixels in the global range, and specifically comprises: with feature maps as a pixel-level guiding module input, the soft pollution region mask after corresponding scaling and equal proportion as prior information, H, W and C represent the height, width and channel number of the input feature map respectively; First, the X in is positionally encoded to get the feature map X p , X p each element x p = x in + pe, where pe is obtained by image 2D position encoding: Wherein, x and y represent the horizontal and vertical positions of the pixel, i, j∈[0, C / 4) specify the corresponding dimensions; each dimension of the position encoding is composed of a sinusoidal signal of a specific frequency and phase, and the wavelength range is geometrically increased from 2π to 10000·2π; the first half of the full dimension encodes the horizontal position x, and the second half encodes the vertical position t; Then, a tiling operation is performed on the feature map X p to obtain N is the number of pixels in the feature map. N = H x W; Subsequently, the following is calculated: After three matrix multiplications, the linear transformation obtains the query matrix Q, the key matrix K, and the value matrix V; note that the pollution region mask is inverted here: And the operation on the key matrix: where W q , W k , is a learnable linear transformation matrix; the resulting query matrix is then dot-producted with the transposed key matrix and a temperature coefficient is introduced to balance the weight values of the attention relevance matrix The attention relationship is normalized by using a softmax function, and then multiplied by a value matrix V to obtain a feature map after information interaction is updated The Further feature extraction and integration are performed by linear transformation using a feed-forward network (FFN) to obtain a feature map The FFN is composed of two linear fully connected layers with a ReLU activation function in between, i.e.: Constructing the residual connection with the input feature map X in , obtaining the feature map Y of the pixel-level guiding module output out : 4.The degraded image restoration method based on a pollution region mask guiding according to claim 1, wherein, Step one specifically comprises: First, the image I containing the contaminated area is calculated in the difference I on the RGB color channels gt the difference I on the RGB color channels diff : where c is each color channel of RGB; then the normalized function N(·) is used to normalize each color channel of I diff ; finally, the soft region mask reference value is obtained by calculating the average value M softmask the element value range of M is [0, 1], M softmask is used for learning training target of soft area mask generator; Soft region mask generator G softmask (·) Using a U-Net structure, a soft region mask M in corresponding to the image I in containing the contaminated region is learned and generated softmask close to the soft region mask reference value M 5.The degraded image restoration method based on a pollution region mask guiding according to claim 1, wherein, The multi-scale channel attention module is used to connect the encoding block and the reversible network module by jumping, and specifically comprises: The feature map input to the multi-scale channel attention module is denoted as The multi-scale channel attention module includes a global branch and a local branch; the feature map X in In the global branch, the feature map X is first subjected to global average pooling to obtain a feature map The local branch is subjected to a point-wise convolution operation on the feature map X in Consistency is maintained Subsequently, the global branch and the local branch are respectively subjected to a bottleneck layer point-wise convolution operation, and the feature is first reduced in dimension and then increased in dimension, i.e.: wherein each convolution is a 1x1 convolution kernel, the convolution operations Conu1 and Conu3 first reduce the feature map to a channel number of C / 4, and then the convolution operations Conu2 and Conu4 increase the channel number to the original channel number C; the local features after the bottleneck layer maintain the same size as the feature map X in ; the global features after the bottleneck layer reflect the global information of each feature channel with a scalar value. Subsequently, the features are re-scaled by performing a aggregation and weight distribution to obtain a feature map with the features re-scaled wherein, represents the feature point weight constructed by the multi-scale channel attention module, represents a broadcast addition operation, represents an element-wise multiplication operation.

Images