A diversified image restoration method based on multi-scale features and split attention

Abstract

The present invention discloses a diversified image restoration method based on multi-scale features and split attention, comprising the following steps: first, resizing and data enhancement are performed on the co-open dataset images; then, a diversified image restoration network based on multi-scale features and split attention is constructed, and the multi-scale feature representation network is used to obtain the multi-scale spatial information of the image; a diversified content generation network is used to generate diversified coarse restoration results while improving the model inference speed; a refinement network based on split attention is used to further improve the quality of the coarse restoration results and maintain the appearance consistency of the known area and the mask area; the trained model weight parameters and the constructed restoration network are used to restore the damaged image and evaluate the effect; the present invention can better extract multi-scale spatial information, so that the restored image has a reasonable structure, clear texture and diversity, and the subjective and objective test evaluation results and comparative experiments have proved the effectiveness of the method of the present invention.

Description

Technical Field

[0001] The present invention relates to the field of diversified image restoration technology based on deep learning, and in particular to a diversified image restoration method based on multi-scale features and split attention. Background Art

[0002] Image inpainting involves filling in missing regions of an incomplete image to produce a visually realistic, complete image. This technology has significant implications for numerous real-world applications, such as medical imaging, CT scanning, 3D surface data, art conservation, and restoration of old films and videos. As an ill-posed inverse problem, image inpainting results are not unique, and there are many plausible possibilities for the inpainted texture and structure, especially when the missing regions are large and contain complex content. Therefore, achieving structurally sound, texture-rich, and diverse image inpainting results has been a research hotspot in the diverse field of image inpainting.

[0003] In order to obtain diverse restoration results, many methods have attempted to use variational autoencoders, generative adversarial networks, autoregressive models, Transformers, and diffusion models for image restoration. Reference 1 (Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai. Pluralistic Free-Form Image Completion [J]. International Journal of Computer Vision, 2021, Vol. 129 (10): 2786-2805) is based on a conditional variational autoencoder model to reconstruct the original image by capturing the prior distribution of the missing area. In order to make the results diverse, this method uses a parallel probabilistic architecture, which mainly includes two paths: reconstruction and generation. The reconstruction path mainly obtains the prior distribution of the missing part from the real image and reconstructs the image based on the distribution. The generation path couples the conditional prior of the visible area with the prior distribution to generate diverse restoration results. Although this method can obtain diverse restoration results, the diversity of the final results is not obvious due to insufficient conditional labels.

[0004] To obtain diverse inpainting results with reasonable structure and clear texture, Reference 2 (Jialun Peng, Dong Liu, Songcen Xu, Houqiang Li. Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE[C]. 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.) proposes a diverse image inpainting method based on vector quantized variational autoencoders. This method can separate structural and texture information, thereby improving the quality of diverse image inpainting. In addition, this method uses an autoregressive model to replace the original Gaussian distribution of the continuous latent variable with a Gaussian distribution of discrete latent variables, effectively avoiding the problem of model posterior collapse. Reference 3 (Ziyu Wan, Jingbo Zhang, Dongdong Chen, Jing Liao. High-Fidelity Pluralistic Image Completion with Transformers[C]. 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 2021.) proposes a diverse image inpainting method that combines Transformers and CNNs. This method consists of two stages. In the first stage, a Transformer is used to sample low-resolution inpainted results to obtain prior information about their appearance. In the second stage, a CNN is used to upsample the low-resolution results to supplement fine textures based on the appearance prior and the guidance of available pixels. Although the methods in References 2 and 3 can produce diverse inpainted results with reasonable structures and clear textures, the use of an autoregressive model to predict the prior distribution of structural information significantly limits the model's inference speed, making these methods difficult to apply in real life.

[0005] In the field of diverse image restoration techniques, single-stage methods (Reference 1) must strike a trade-off between diversity and consistency, while two-stage methods (References 2, 3) predict different structural priors in the first stage and focus on generating rich texture details in the second stage. Although two-stage methods alleviate the trade-off between diversity and consistency, current methods use autoregressive models to predict the probability distribution of structural priors, which greatly limits the inference speed. Summary of the Invention

[0006] To solve the problems existing in the above-mentioned diversified image restoration methods, the present invention proposes a diversified image restoration method based on multi-scale features and split attention. This method uses a multi-scale feature representation module to extract multi-scale spatial information from the image, constructs a Gaussian distribution of multi-scale features, and samples from the distribution to generate multiple rough results. The rough results are then further refined using a refinement network based on split attention, so that the restoration results have a reasonable structure, clear texture and diversity, while having a faster inference speed.

[0007] A diversified image restoration method based on multi-scale features and split attention includes the following steps:

[0008] S1. Acquire an experimental dataset including an image dataset and a mask dataset, and perform longitudinal average segmentation on images in the image dataset that are larger than a target size so that the preprocessed images have the same size, wherein the image dataset and the mask dataset each include a training set and a test set;

[0009] The target size can be determined as needed. For example, 256*256 is selected. The vertical average segmentation includes performing left-right average segmentation preprocessing on images that are twice as large as 256*256, and performing left-center-right average segmentation preprocessing on images that are three times larger than 256*256. As long as the images after segmentation have the same size, it is acceptable.

[0010] S2. Construct a diversified image restoration network with multi-scale features and split attention. The diversified image restoration network includes a multi-scale feature representation network, a diversified content generation network, and a refinement network. The multi-scale feature representation network is used to extract multi-scale spatial information of the image to enhance the modeling capability of the model. The diversified content generation network is used to construct a Gaussian distribution of image features. Sampling from the Gaussian distribution can obtain multiple coarse restoration results. The refinement network is used to improve the quality of the coarse restoration results and maintain the appearance consistency between the masked area and the known area. The network establishes a Gaussian distribution of the training data set through a diversified content inference module and samples from the distribution to obtain diversified features.

[0011] The multi-scale feature representation network is configured with a residual coding block and four multi-scale feature representation blocks from front to back. The residual coding block contains two two-dimensional convolutions and an average pooling layer, and adds the input to the output of the convolution layer through a jump connection. In addition, the two-dimensional convolutions all use spectral normalization technology. The multi-scale feature representation block contains a squeeze connection module, a squeeze activation (Squeeze and Excitation, SE) module and a softmax layer. The multi-scale feature representation block can effectively extract the multi-scale spatial features of the image. The specific operation process is as follows:

[0012] B1. Use the squeeze connection module to obtain the multi-scale channel feature map F on the input feature map i :

[0013] F i =Conv(k i ×k i ,G i )(X)

[0014] in, is the input feature map, k i and G i Represent the convolution kernel size and group size respectively;

[0015] B2. Calculate the attention weight Z of the multi-scale feature map through the SE module i :

[0016] Z i =SEWeight(F i )

[0017] B3. Use the Softmax layer to generate recalibrated weights att i :

[0018]

[0019] B4. Obtain a feature map F with multi-scale feature information through cascading out :

[0020] F out =Cat([Y0,…,Y S-1 ])

[0021] Among them, Y i =F i ⊙att i ⊙ represents channel-by-channel multiplication, and Cat represents the cascade operation. In steps B1-B4, S=4, k i =3, G i =2, i=0,…,3.

[0022] The diversified content generation network is configured with a diversified content inference module and four multi-scale feature representation blocks from front to back. The network establishes a Gaussian distribution of the training data set through the diversified content inference module and samples diversified feature maps from the Gaussian distribution. The diversified content inference module contains five residual blocks, each of which consists of a sequence model and a skip connection. The sequence model extracts features from the input through two spectral normalization convolutional layers and two LeakyReLU activation functions, and maintains the consistency of the input and output shapes. The skip connection contains a spectral normalization convolutional layer and is connected to the output of the sequence model to ensure that the number of input and output channels of the residual block is consistent.

[0023] The refinement network consists of an encoder, a decoder, and a self-attention block. Both the encoder and decoder contain four multi-scale feature representation blocks. The separate attention block helps the network copy long-range information from the encoded and decoded features, thereby improving the global attention ability of the refinement network. The specific steps for obtaining attention features using the separate attention block are as follows:

[0024] B1. For a decoder feature f d , first calculate its similarity matrix A:

[0025] A=φ(f d ) T θ(f d )

[0026] Among them, A ij represents the similarity between the i-th feature and the j-th feature, and φ and θ represent 1×1 convolution.

[0027] B2. Divide the similarity matrix A into two parts based on the mask area and the known area: A c Represents the similarity between different pixels in a known area, A m Represents the similarity between different pixels in the generated mask area. Next, calculate the high-resolution feature f e Long-term dependencies with different pixels in the known region and decoder features f d Long-term dependencies with different pixels in the generated mask region:

[0028] z c =Softmax(A c )α(f e )

[0029] z m =Softmax(A m )λ(f d )

[0030] Among them, z c Is the context feature matrix, according to the context feature matrix z c , the decoder can convert the high-resolution feature f output by the encoder into e The valid information contained in is copied to the generated mask area, z m is the self-attention feature matrix, α and λ represent 1×1 convolution;

[0031] B3. Learn the weight mapping based on the maximum similarity between pixels in each region. Specifically, first obtain A c and A mThe maximum similarity value is then used to modulate the weight ratio using 1×1 convolution μ and ν, while using Softmax to ensure that the w of each region c +w m =1:

[0032] [w c ,w m ]=Softmax([μ(max(A c )),v(max(A m ))])

[0033] Among them, max is performed on each channel of the similarity score, w c and w m They represent the weights of pixels in the known area and pixels in the mask area, respectively, and they hold different values ​​for different pixels.

[0034] B4. Multiply the learned weight values ​​of different regions by the features of the corresponding regions and sum them up to obtain the final feature matrix:

[0035]

[0036] in, It is the output feature matrix of the attention module, which contains global information and can improve the global attention ability of the refinement network.

[0037] S3. Input the images of the training set in the image dataset in step S1 and the images of the training set in the mask dataset into the diversified image restoration network constructed in step S2; set the optimization strategy and related hyperparameters to train the diversified image restoration network; optimize the diversified image restoration network through cyclic iterative optimization so that the loss function is continuously reduced and tends to converge, and save the training weight parameters of the diversified image restoration network.

[0038] S4. Use the diversified image restoration network constructed in step S2 and the network training weight parameters obtained in step S3 to construct a diversified image restoration model based on multi-scale features and separated attention, and use the image restoration model to repair and complete the damaged images consisting of the test set in the image dataset and the test set in the mask dataset to obtain diversified image restoration results.

[0039] Beneficial effects:

[0040] 1. This paper proposes a diversified image restoration method based on multi-scale features and split attention. To address the current problems in the field of diversified image restoration technology, this paper first uses a multi-scale feature representation network to obtain the multi-scale spatial information of the input image; uses a diversified content generation network to generate diversified coarse restoration results while improving the model's inference speed; and uses a refinement network to further improve the quality of the coarse restoration results.

[0041] 2. This paper proposes a multi-scale feature representation block for improving the feature extraction capabilities of encoders and decoders. It can learn multi-scale feature information to enhance the modeling capability of variational autoencoders.

[0042] 3. The present invention proposes a separation attention block to improve the restoration effect of the refinement network, which can improve the quality of the coarse restoration results and maintain the appearance consistency of the known area and the mask area. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 An overall diagram of the model structure used in the implementation of the present invention;

[0044] Figure 2 A schematic diagram of a flow chart of an embodiment of the present invention;

[0045] Figure 3 The repair results of the diversified repair method designed by the present invention on Places2 for images damaged by different masks;

[0046] Figure 4 The repair results of the damaged image in Places2 are shown in Figure 1, which shows the results of the diversified repair method designed by the present invention and the methods of References 1, 2 and 3. DETAILED DESCRIPTION

[0047] To provide a clearer understanding of the technical features, objectives, and beneficial effects of the present invention, an embodiment of the present invention is further described with reference to the accompanying drawings. The embodiment is intended only to further illustrate the present invention and is not to be construed as limiting the scope of protection of the present invention. Non-essential improvements and adjustments made by those skilled in the art based on the contents of the present invention also fall within the scope of protection of the present invention.

[0048] A diversified image restoration method based on multi-scale features and split attention includes the following steps:

[0049] S1. Obtain the experimental dataset: The image dataset is based on the Places2 dataset. For the image training dataset, we select 500 million images from the categories of hills, canyons, fields, weeds, mountains, mountain roads, mountain snow, sky, tundra, and valleys. For the image test dataset, we select 100 images from each category to form 1000 test images. The mask dataset consists of a center mask dataset and an irregular mask dataset. In the mask dataset, the mask M uses 0 to represent missing areas and 1 to represent non-missing areas.

[0050] S2, build a diversified image restoration network with multi-scale features and separated attention, such as Figure 1As shown, the diversified image restoration network includes a multi-scale feature representation network, a diversified content generation network, and a refinement network. The multi-scale feature representation network is used to extract multi-scale spatial information of the image, thereby enhancing the modeling capability; the diversified content generation network is used to construct a Gaussian distribution of image features, from which multiple coarse restoration results can be obtained by sampling; and the refinement network is used to improve the quality of the coarse restoration results and maintain the continuity and consistency of the masked area and the known area.

[0051] The multi-scale feature representation network is configured with a residual coding block and four multi-scale feature representation blocks from front to back. The residual coding block contains two two-dimensional convolutions and an average pooling layer, and adds the input to the output of the convolution layer through a jump connection. In addition, the two-dimensional convolutions all use spectral normalization technology. The multi-scale feature representation block contains a squeeze connection module, a squeeze activation (Squeeze and Excitation, SE) module and a softmax layer. The multi-scale feature representation block can effectively extract the multi-scale spatial features of the image. The specific operation process is as follows:

[0052] B1. Use the squeeze connection module to obtain the multi-scale channel feature map F on the input feature map i :

[0053] F i =Conv(k i ×k i , G i )(X)

[0054] in, is the input feature map, k i and G i Represent the convolution kernel size and group size respectively;

[0055] B2. Calculate the attention weight Z of the multi-scale feature map through the SE module i :

[0056] Z i =SEWeight(F i )

[0057] B3. Use the Softmax layer to generate recalibrated weights att i :

[0058]

[0059] B4. Obtain a feature map F with multi-scale feature information through cascading out :

[0060] F out =Cat([Y0,…,YS-1 ])

[0061] Among them, Y i =F i ⊙att i ⊙ represents channel-by-channel multiplication, and Cat represents the cascade operation. In steps B1-B4, S=4, k i =3, G i =2, i=0,…,3.

[0062] The diversified content generation network is configured with a diversified content inference module and four multi-scale feature representation blocks from front to back. The network establishes a Gaussian distribution of the training data set through the diversified content inference module and samples diversified feature maps from the Gaussian distribution. The diversified content inference module contains five residual blocks, each of which consists of a sequence model and a skip connection. The sequence model extracts features from the input through two spectral normalization convolutional layers and two LeakyReLU activation functions, and maintains the consistency of the input and output shapes. The skip connection contains a spectral normalization convolutional layer and is connected to the output of the sequence model to ensure that the number of input and output channels of the residual block is consistent.

[0063] The refinement network consists of an encoder, a decoder, and a self-attention block. Both the encoder and decoder contain four multi-scale feature representation blocks. The separate attention block helps the network copy long-range information from the encoded and decoded features, thereby improving the global attention ability of the refinement network. The specific steps for obtaining attention features using the separate attention block are as follows:

[0064] B1. For a decoder feature f d , first calculate its similarity matrix A:

[0065]

[0066] Among them, A ij represents the similarity between the i-th feature and the j-th feature, and φ and θ represent 1×1 convolution.

[0067] B2. Divide the similarity matrix A into two parts based on the mask area and the known area: A c Represents the similarity between different pixels in a known area, A m Represents the similarity between different pixels in the generated mask area. Next, calculate the high-resolution feature f e Long-term dependencies with different pixels in the known region and decoder features f d Long-term dependencies with different pixels in the generated mask region:

[0068] z c =Softmax(Ac )α(f e )

[0069] z m =Softmax(A m )λ(f d )

[0070] Among them, z c Is the context feature matrix, according to the context feature matrix z c , the decoder can convert the high-resolution feature f output by the encoder into e The valid information contained in is copied to the generated mask area, z m is the self-attention feature matrix, α and λ represent 1×1 convolution;

[0071] B3. Learn the weight mapping based on the maximum similarity between pixels in each region. Specifically, first obtain A c and A m The maximum similarity value is then used to modulate the weight ratio using 1×1 convolution μ and v, while using Softmax to ensure that w of each region c +w m =1:

[0072] [w c ,w m ]=Softmax([μ(max(A c )),v(max(A m ))])

[0073] Among them, max is performed on each channel of the similarity score, w c and w m They represent the weights of pixels in the known area and pixels in the mask area, respectively, and they hold different values ​​for different pixels.

[0074] B4. Multiply the learned weight values ​​of different regions by the features of the corresponding regions and sum them up to obtain the final feature matrix:

[0075]

[0076] in, It is the output feature matrix of the attention module, which contains global information and can improve the global attention ability of the refinement network.

[0077] S3. Input the images of the training set in the image dataset and the images of the training set in the mask dataset in step S1 into the diversified image restoration network constructed in step S2 and train them. Based on the Pytorch framework and using NVIDIA GeForceRTX 3090, the experiment was carried out. Average pooling and interpolation with convolutional layers were applied to achieve downsampling and upsampling respectively. During the optimization process, the Adam optimizer was used, and the learning rate was fixed at 1×10 -4 , batch_size is 16. Set the optimization strategy and related hyperparameters to train the diversified image restoration network and adopt the joint loss constraint to optimize the diversified image restoration network; perform cyclic iterative optimization on the diversified image restoration network so that the loss function is continuously reduced and converged, and save the training weight parameters of the diversified image restoration network.

[0078] S4. Use the diversified image restoration network constructed in step S2 and the network training weight parameters obtained in step S3 to construct a diversified image restoration model based on multi-scale features and separated attention, and use the image restoration model to repair and complete the damaged images consisting of the test set in the image dataset and the test set in the mask dataset to obtain diversified image restoration results.

[0079] In order to objectively evaluate the restoration effect of the present invention, this embodiment tested the qualitative and quantitative comparison results of the restoration method designed by the present invention and the methods of Document 1, Document 2, and Document 3 on the Places2 test set. The restoration effects of the irregular mask with different mask ratios and the restoration effect of the center mask were compared and evaluated using multiple objective evaluation indicators such as PSNR, SSIM, MAE, and LPIPS. PSNR and MAE reflect the difference between the real image and the restored image based on pixel statistical information, SSIM imitates the human visual system to extract high-order structural information to measure image similarity, and LPIPS uses deep visual representation to imitate human perception evaluation. The larger the PSNR and SSIM, the better the image restoration quality, and the smaller the MAE, the better the image restoration quality. In addition, diversified image restoration should focus on generating diverse real results, not just approximate real results. In order to measure diversity, we use learning perceptual image block similarity (LPIPS). LPIPS uses deep visual representation to imitate human perception evaluation and can be used to evaluate the diversified generation ability of the model. The larger the LPIPS, the higher the diversity of the restoration results. This embodiment uses LPIPS (I out ) and LPIPS(I out(m) ) represent the diversity of the output results and the diversity of the masked areas, respectively. The evaluation and comparison results are shown in Tables 1 and 2. The experimental results show that the inpainting method designed in this paper outperforms the methods in References 1, 2, and 3 in all evaluation metrics for various mask types.

[0080] The subjective repair effect of the present invention is as follows Figure 3 As shown, the first column is the mask images after different mask processing. The masks are divided into center masks and random masks. For the random masks, the mask ratios from top to bottom are (0.2, 0.3], (0.3, 0.4], (0.4, 0.5]; for each mask image, the present invention randomly generates three repaired images. The second to fourth columns are the repair results randomly generated by the present invention. Comparison Figure 3 It can be seen from the comparison between various mask-repaired pictures and original images that the repair method designed in the present invention can handle the repair of damaged images with various masks, and the repaired images have reasonable structures, clear textures and diversity.

[0081] Figure 4 The results of the repair method of the present invention and the methods of literature 1, literature 2 and literature 3 on the Places2 dataset are shown, where the sub-images Mask_1 and Mask_2 are damaged images to be repaired, where gray represents the area to be repaired, where sub-images Gt_1 and Gt_2 are original images, where sub-images (a1)-(c3) and sub-images (e1)-(e3) are the results of the repair of sub-images Mask_1 and Mask_2 by the method of literature 1, where sub-images (b1)-(b3) and sub-images (f1)-(f3) are the results of the repair of sub-images Mask_1 and Mask_2 by the method of document 2, among which sub-images (c1)-(c3) and sub-images (g1)-(g3) are the results of the repair of sub-images Mask_1 and Mask_2 by the method of document 3, among which sub-images (d1)-(d3) and sub-images (h1)-(h3) are the results of the repair of sub-images mask_1 and mask_2 by the repair method of the present invention. Figure 4 The restoration result diagram shows that the restoration method of the present invention can better focus on the structure and texture features of the mask area than the methods in References 1 to 3, and obtain restoration results with reasonable structure, clear texture and diversity.

[0082] According to subjective and objective evaluations and comparative experiments, the method of the present invention can repair mask images of various proportions, and is superior to existing methods in ensuring the continuity and rationality of the overall structure, and ensuring that the images are realistic, natural, and diverse.

[0083] Table 1 Comparison of different mask restoration evaluations

[0084]

[0085] Table 2 Comparison of center mask restoration diversity evaluation

[0086]

[0087] The above describes the relevant contents of the present invention, and those skilled in the art will be able to implement the present invention based on these descriptions. Based on the above contents of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts should fall within the scope of protection of the present invention.

Claims

1. A diversified image restoration method based on multi-scale features and split attention, characterized in that The following steps are involved: S1. Acquire an experimental dataset including an image dataset and a mask dataset, and perform longitudinal average segmentation on images in the image dataset that are larger than a target size so that the preprocessed images have the same size, wherein the image dataset and the mask dataset each include a training set and a test set; S2. Construct a diversified image restoration network with multi-scale features and separated attention. The diversified image restoration network includes a multi-scale feature representation network, a diversified content generation network and a refinement network. The multi-scale feature representation network is used to extract multi-scale spatial information of the image to enhance the modeling ability of the model. The diversified content generation network sequentially sets a diversified content inference module and four multi-scale feature representation blocks from front to back. The network establishes a Gaussian distribution of the training data set through the diversified content inference module and samples diversified feature maps from the Gaussian distribution. The diversified content inference module contains five residual blocks, each of which consists of a sequence model and a jump connection. The sequence model extracts features from the input through two spectral normalization convolution layers and two LeakyReLU activation functions and keeps the shapes of the input and output consistent. The jump connection contains a spectral normalization convolution layer and is connected to the output of the sequence model to ensure that the number of input and output channels of the residual block is consistent. The refinement network is used to improve the quality of the coarse restoration result and maintain the continuity and consistency of the mask area and the known area. S3, inputting the images of the training set in the image dataset and the images of the training set in the mask dataset in step S1 into the diversified image restoration network constructed in step S2; setting an optimization strategy and related hyperparameters to train the diversified image restoration network; performing cyclic iterative optimization on the diversified image restoration network so that the loss function is continuously reduced and converged, and saving the training weight parameters of the diversified image restoration network; S4. Use the diversified image restoration network constructed in step S2 and the network training weight parameters obtained in step S3 to construct a diversified image restoration model based on multi-scale features and separated attention, and use the diversified image restoration model to repair and complete the damaged images consisting of the test set in the image dataset and the test set in the mask dataset to obtain diversified image restoration results.

2. A diversified image restoration method based on multi-scale features and split attention according to claim 1, characterized in that: The multi-scale feature representation network in step S2 sequentially sets one residual coding block and four multi-scale feature representation blocks from front to back; The residual coding block contains two two-dimensional convolutions and an average pooling layer, and adds the input to the output of the convolution layer through a skip connection. In addition, the two-dimensional convolutions all use spectral normalization technology; The multi-scale feature representation block includes a squeeze connection module, a squeeze activation SE module and a Softmax layer. The multi-scale feature representation block can effectively extract the multi-scale spatial features of the image. The specific process is as follows: B1. Use the squeeze connection module to obtain the multi-scale channel feature map F on the input feature map i : F i =Conv(k i ×k i ,G i )(X) in, is the input feature map, k i and G i Represent the convolution kernel size and group size respectively; B2. Calculate the attention weight Z of the multi-scale feature map through the SE module i : Z i =SEWeight(F i ) B3. Use the Softmax layer to generate recalibrated weights att i : B4. Obtain a feature map F with multi-scale feature information through cascading out : F out =Cat([Y0,…,Y S-1 ]) Among them, Y i =F i ⊙att i ⊙ represents channel-by-channel multiplication, Cat represents cascade operation; in steps B1-B4, S=4, k i =3, G i =2, i=0,…,3.

3. A diversified image restoration method based on multi-scale features and split attention according to claim 1, characterized in that: The refinement network in step S2 consists of an encoder, a decoder, and a self-attention block; the encoder and decoder both contain four multi-scale feature representation blocks; the separate attention block can help the network copy long-range information from the encoded and decoded features, thereby improving the global attention ability of the refinement network. The specific steps of obtaining attention features using the separate attention block are as follows: B1. For a decoder feature f d , first calculate its similarity matrix A: A=φ(f d ) T θ(f d ) Among them, A ij represents the similarity between the i-th feature and the j-th feature, φ and θ represent 1×1 convolution; B2. Divide the similarity matrix A into two parts based on the mask area and the known area: A c Represents the similarity between different pixels in a known area, A m Represents the similarity between different pixels in the generated mask area. Next, calculate the high-resolution feature f e Long-term dependencies with different pixels in the known region and decoder features f d Long-term dependencies with different pixels in the generated mask region: z c =Softmax(A c )α(f e ) z m =Softmax(A m )λ(f d ) Among them, z c Is the context feature matrix, according to the context feature matrix z c , the decoder can convert the high-resolution feature f output by the encoder into e The valid information contained in is copied to the generated mask area, z m is the self-attention feature matrix, α and λ represent 1×1 convolution; B3. Learn the weight mapping based on the maximum similarity between pixels in each region. Specifically, first obtain A c and A m The maximum similarity value is then used to modulate the weight ratio using 1×1 convolution μ and ν, while using Softmax to ensure that the w of each region c +w m =1: [w c ,w m ]=Softmax([μ(max(A c )), v(max(A m ))]) Among them, max is performed on each channel of the similarity score, w c and w m Represent the weights of pixels in the known area and pixels in the mask area, respectively. They hold different values ​​for different pixels. B4. Multiply the learned weight values ​​of different regions by the features of the corresponding regions and sum them up to obtain the final feature matrix: in, It is the output feature matrix of the attention module, which contains global information and can improve the global attention ability of the refinement network.

Images