Image restoration method based on multi-scale attention mechanism
Patent Information
- Application Number
- CN202411364306.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2024-12-31
AI Technical Summary
Existing image recovery methods perform poorly when processing highly complex textures and high-resolution images in the real world. Due to the inherent characteristics of small convolutions, the receptive field is limited, which limits the enhancement of information interaction and the self-attention mechanism. The advantage comes from the cost of increasing the number of parameters and longer training time.
Build an image repair method based on a multi-scale attention mechanism, combine the advantages of CNN and Transformer to generate an EMNet network framework, extract and fuse image features through encoder and decoder, and use the MEC and MFEU modules in the MixBlock block to combine local details and information. Repair the global structure information.
It realizes the image finely repairing while maintaining the overall coherence of the image, producing a more appropriate and vivid and realistic repair effect, expanding the model's vision and feature information capture capabilities, and improving the accuracy of image repair.
Smart Images
Figure CN119228693A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image restoration method based on a multi-scale attention mechanism. Background Art
[0002] Image restoration aims to restore high-quality, clear images from degraded versions. Traditional methods exploit local relationships and edge information between pixels, but often perform poorly when dealing with highly complex textures and high-resolution images in the real world. In recent years, the field of image restoration has made significant progress in tasks such as image dehazing, desnowing, and deblurring with the help of data-driven convolutional neural network methods. Inspired by its outstanding performance in high-level vision tasks, visual Transformers have also been introduced to solve low-level vision tasks. Transformers capture long-range dependencies through a self-attention mechanism, enhancing the potential for detailed and comprehensive feature representation.
[0003] Despite the impressive performance of existing methods, they still have significant limitations. For example, due to the inherent characteristics of small convolutions, the receptive fields of these models are limited, which in turn limits the enhancement of information interaction. In contrast, the advantages of self-attention mechanisms come at the cost of increased parameters and longer training time. In addition, multi-scale representation learning is a key aspect of image restoration. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to construct an image restoration method based on a multi-scale attention mechanism, and the scheme uses deep learning technology. Based on the advantages of CNN and Transformer, the EMNet (Efficient Multi-Scale Convolutional Network) network framework is generated, which adopts the idea of first extracting image feature information and then performing feature fusion to repair the image: first, the shallow features of the damaged image are converted into deep features through the encoder, and multi-scale features are obtained, and then deeper features and detail information are obtained through the decoder. The present invention mainly adds MEC and MFEU modules with multi-scale attention mechanisms to the MixBlcok residual block. Combining local detail information and global structural information, the image is repaired accurately and comprehensively. The repair result not only retains the details well, but also maintains the overall coherence of the image.
[0005] The present invention solves the technical problem by adopting the following technical solutions:
[0006] The image restoration method based on the multi-scale attention mechanism includes the following steps:
[0007] Step 1: Collect damaged image samples under various severe weather conditions;
[0008] Step 2: Construct model input, preprocess the collected damaged image and then construct the model input;
[0009] Step 3: Construct a MixBlock block with a multi-scale attention mechanism;
[0010] Step 4: Build EMNet with multi-scale attention mechanism;
[0011] Step 5: Use the model trained in step 4 and combine local detail information with global structure information to accurately and comprehensively repair the image.
[0012] Furthermore, step 1 collects damaged image samples under various severe weather conditions, and the collection includes damaged image samples under various severe weather conditions, specifically:
[0013] (1) Collect foreground images. Collect high-resolution foreground images from the existing COCO dataset and the Internet.
[0014] (2) Collect background images. Collect background images under different weather conditions from natural scenery datasets and street view datasets.
[0015] Furthermore, the model input mentioned in step 2 is constructed by preprocessing the collected damaged image, specifically:
[0016] (1) Segmentation and extraction: Use a segmentation mask to separate these foreground objects from the original image, ensuring that only the target object is retained while removing unnecessary background information.
[0017] (2) Fusion of foreground and background: The foreground and background images are fused through image processing techniques such as color correction, light and shadow adjustment, and perspective matching.
[0018] (3) Data enhancement. Before synthesis, the foreground image is processed using random transformation. Each image is resized to 256 2 . At the same time, each image is randomly flipped horizontally.
[0019] (4) Use the image after the first three steps of preprocessing as the model input.
[0020] Furthermore, the details of the MixBlock block proposed in step 3 to construct a multi-scale attention mechanism are as follows:
[0021] For a corrupted image F with an input size of H×W×3 inFirst, the shallow features of the image are obtained through the shallow layer or 3x3 convolution layer, with a size of H×W×C. After extracting the shallow features of the image, it enters the core part of the MixBlock block composed of two modules, MEC and MFEU. The MixBlock block is a residual block. Each MixBlock block uses Extract features in blocks, and then reduce the feature resolution to go through After the blocks are divided, the features are upsampled to the input size for subsequent processing. In this process, the channels of the damaged image begin to expand, the resolution gradually decreases, and deep features begin to be generated. The MEC is mainly a parallel combination of convolutions with different expansion rates, and the MFEU is mainly a parallel combination of multi-scale attention mechanisms. Further, the details of the MEC block proposed in step 3 are as follows:
[0022] The core part of MEC is composed of parallel combination of convolutions with different expansion rates and placed in the last MixBlock block. The shallow features are processed by the previous n ordinary MixBlock blocks to obtain deep features and enter the last MixBlock block. After regularization and continuous 1x1-5x5 convolution operations, they enter the core three-branch convolution combination block, which is composed of deep convolution kernels with expansion mechanisms of sizes 3x3, 5x5, and 7x7. In this process, deeper feature information and more multi-scale feature information are extracted. Then, the deep features of the three branches are fused and then convolution-GELU-convolution operations are performed on them. Finally, channel weighting operations are performed with the input deep features to obtain rich deep feature information.
[0023] Furthermore, the details of the MFEU block proposed in step 3 are as follows:
[0024] The core part of MFEU is a parallel combination of different attention mechanisms. It follows the MEC block and is also placed in the last MixBlock block. The input features processed by MEC enter MFEU and then undergo regularization processing before entering the multi-shape attention branch combination. In this process, detailed features are further obtained. After the feature fusion of the attention branch, convolution-GELU-convolution operations are performed, and finally, channel weighting operations are performed with the input deep features to obtain richer deep feature information.
[0025] Furthermore, the details of constructing EMNet with multi-scale attention mechanism proposed in step 4 are as follows:
[0026] EMNet is a U-shaped network model with three pairs of encoders and decoders. The encoder has two outputs. One is to continue downsampling the deep features and fuse them with the shallow features collected by the shallow layer to extract the multi-scale features and deep features of the image. The other is to perform jump connections on the deep features and fuse them with the upsampled deep features of the corresponding decoders to obtain deeper features and more image details. The decoder of the last layer is convolved with the F in Perform feature fusion to obtain the final repaired image
[0027] Furthermore, the trained model in step 4 is used to combine local detail information with global structure information to accurately and comprehensively repair the image:
[0028] Given a preprocessed H×W×3 damaged image, a 3×3 convolutional layer is used to extract shallow features of the damaged image, which are then passed through three-scale encoder sub-networks to generate deep features and multi-scale information. In this process, the encoder gradually reduces the resolution while expanding the channels. In addition, the low-resolution degraded image is merged into the main path through a shallow layer to better handle the different blur levels in the image, and is concatenated and then 3×3 convolutions are performed to adjust the channels. The deepest features obtained are then input into a three-scale decoder to generate the final features by gradually restoring the features to their original size. In this process, the decoder features are concatenated with the encoder features to assist in reconstruction, and 1×1 convolutions are used to reduce the number of channels by half. Through 3×3 convolutions and image-level skip connections, after three MixBlock blocks, the multi-scale information obtained after upsampling is combined with local detail information and global structure information to finally obtain a nearly realistic repaired image.
[0029] The beneficial effects of the present invention are:
[0030] The image restoration method based on the multi-scale attention mechanism constructed by the present invention enables the damaged image to obtain a fine feature map; produces a more appropriate and vivid restoration effect, and can maintain the overall consistency of the image. It combines the advantages of CNN and Transformer and introduces a multi-scale attention mechanism, so that the model can have a broader field of view and richer feature information. At the same time, under the U-shaped network architecture, the model is more sensitive to multi-scale information capture. Based on this, an image closer to the real one is obtained after restoration. Based on the constructed deep learning model, the present invention performs corresponding preprocessing on the damaged images collected under bad weather, thereby improving the accuracy of the restored image. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a flow chart of the present invention;
[0032] Figure 2 It is a schematic diagram of the overall framework of the image restoration method based on the multi-scale attention mechanism of the present invention;
[0033] Figure 3 It is a multi-scale attention mechanism structure diagram of the image restoration method based on the multi-scale attention mechanism of the present invention;
[0034] Figure 4 This is a restoration effect diagram of the image restoration method based on the multi-scale attention mechanism of the present invention;
[0035] Figure 5 This is a restoration effect diagram of the image restoration method based on the multi-scale attention mechanism of the present invention. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0038] The image restoration method based on the multi-scale attention mechanism includes the following steps:
[0039] Step 1: Collect damaged image samples under various severe weather conditions.
[0040] (1) Collect foreground images. Collect high-resolution foreground images from the existing COCO dataset and the Internet.
[0041] (2) Collect background images. Collect background images under different weather conditions from natural scenery datasets and street view datasets.
[0042] Step 2: Construct model input. Preprocess the collected damaged images and then construct the model input.
[0043] (5) Segmentation and extraction: Use a segmentation mask to separate these foreground objects from the original image, ensuring that only the target objects are retained while removing unnecessary background information.
[0044] (6) Fusion of foreground and background: The foreground and background images are fused through image processing techniques such as color correction, light and shadow adjustment, and perspective matching.
[0045] (7) Data enhancement. Before synthesis, the foreground image is processed using random transformation. Each image is resized to 256 2 . At the same time, each image is randomly flipped horizontally.
[0046] (8) The image after the first three steps of preprocessing is used as the model input.
[0047] Step 3: Construct a MixBlock block with multi-scale attention mechanism.
[0048] For a corrupted image F with an input size of H×W×3 in First, the shallow features of the image are obtained through the shallow layer or 3x3 convolution layer, with a size of H×W×C. After extracting the shallow features of the image, the core part enters the MixBlock block composed of two modules, MEC and MFEU. The MixBlock block has n residual blocks, which are used Extract features in blocks, and then reduce the feature resolution to go through After the blocks are divided, the features are upsampled to the input size for subsequent processing. In this process, the channels of the damaged image begin to expand, the resolution gradually decreases, and deep features begin to be generated. The MEC is mainly a parallel combination of convolutions with different expansion rates, and the MFEU is mainly a parallel combination of multi-scale attention mechanisms;
[0049] (1) Figure 2 As shown in Figure 2, when entering the core part of the last MixBlock residual block, the MEC (Multi-scale Enhanced Convolution) block, F in After n rounds of feature extraction, it becomes It passes through Norm, 1x1Conv, 5x5Conv in sequence, and then enters three-branch parallel convolution to extract features of different scales. , i represents the branch number, j represents the expansion rate, and the j value is 3; k is the size of the convolution kernel, and the k values are 3, 5, and 7. represents a dilated convolution with a kernel size of k×k. Then the features of each branch are fused, and 1x1Conv, GELU, and 1x1Conv are performed on them. Perform channel weighting to obtain deep features
[0050] (2) Deep Features Then it enters the MFEU block, and after passing through Norm, it enters the three-branch attention channel, namely DFE (Dual-branch Feature Enehnced) attention, AFE (Adaptive Feature Enhenced) attention, and FE (Feature Enhenced) attention, such as Figure 3 As shown, the features with deeper details are obtained Then the branch features are fused, and 1x1Conv, GELU, and 1x1Conv operations are performed on them. Perform channel weighting to obtain deep features Step 4: Construct EMNet with multi-scale attention mechanism.
[0051] EMNet (Efficient Multi-Scale Convolutional Network) is a U-shaped network model with three pairs of encoders and decoders. The encoder has two outputs. One is to continue downsampling the deep features and fuse them with the shallow features collected by the shallow layer to extract the multi-scale features and deep features of the image. The other is to skip the deep features and fuse them with the deep features after upsampling of the corresponding decoder to obtain deeper features and more image details. The decoder of the last layer is convolved with F after 3x3 convolution. in Perform feature fusion to obtain the final repaired image Step 5: Train the model and combine local detail information with global structure information to accurately and comprehensively repair the image.
[0052] Given a preprocessed H×W×3 damaged image, a 3×3 convolutional layer is used to extract shallow features of the damaged image, which are then passed through three-scale encoder sub-networks to generate deep features and multi-scale information. In this process, the encoder gradually reduces the resolution while expanding the channels. In addition, the low-resolution degraded image is merged into the main path through the shallow layer to better handle the different blur levels in the image, and is concatenated and then 3×3 convolutions are performed to adjust the channels. The deepest features obtained are then input into a three-scale decoder to generate the final features by gradually restoring the features to their original size. In this process, the decoder features are concatenated with the encoder features to assist in reconstruction, and 1×1 convolutions are used to reduce the number of channels by half. Through 3×3 convolutions and image-level skip connections, the multi-scale information obtained after three MixBlock blocks and upsampling is combined with local detail information and global structure information to finally obtain a nearly realistic repaired image.
[0053] The image restoration method based on the multi-scale attention mechanism constructed by the present invention enables the damaged image to obtain a fine feature map; produces a more appropriate and vivid restoration effect, and can maintain the overall consistency of the image. Combining the advantages of CNN and Transformer and introducing a multi-scale attention mechanism, the model can have a broader field of view and richer feature information. At the same time, under the U-shaped network architecture, the model is more sensitive to multi-scale information capture. Based on this, the restored image is closer to the real one.
[0054] Based on the constructed deep learning model, the present invention performs corresponding preprocessing on damaged images collected under severe weather conditions, thereby improving the accuracy of image repair.
Claims
1. An image restoration method based on a multi-scale attention mechanism, characterized in that: The steps include: Step 1: Collect damaged image samples under various severe weather conditions; Step 2: Construct model input, preprocess the collected damaged image and then construct the model input; Step 3: Construct a MixBlock block with a multi-scale attention mechanism; Step 4: Build EMNet with multi-scale attention mechanism; Step 5: Train the model to learn the multi-scale information of the image, combine local detail information with global structure information, and accurately and comprehensively repair the image.
2. The image restoration method based on the multi-scale attention mechanism according to claim 1, characterized in that: The details of step 2, collecting damaged image samples under various severe weather conditions, are as follows: The collection includes damaged image samples under various adverse weather conditions, which refers to collecting foreground and background images under snowy and foggy conditions.
3. The image restoration method based on the multi-scale attention mechanism according to claim 1, characterized in that: The second step is to construct a model input, and the details of constructing the model input after preprocessing the collected damaged image are as follows: The preprocessing is to synthesize the collected damaged images and unify the size of the input damaged images to 256 2 , each image is randomly flipped horizontally and data augmentation is performed with a probability of 0.
5.
4. The image restoration method based on the multi-scale attention mechanism according to claim 1, characterized in that: The details of step 3, by constructing a MixBlock block with a multi-scale attention mechanism, are as follows: The MixBlock block is mainly composed of two modules: MEC (Multi-scale Enhenced Convolution) and MFEU (Multi-shape Feature Enhenced Unit). After a 3x3 convolution operation, it enters the MixBlock block. In this process, the channels of the damaged image begin to expand, the resolution gradually decreases, and deep features begin to be generated. The MEC and MFEU modules are respectively composed of convolution combination operations with different expansion rates and multi-scale attention combination mechanisms.
5. The image restoration method based on the multi-scale attention mechanism according to claim 1, characterized in that: The details of building EMNet with multi-scale attention mechanism involved in step 4 are as follows: EMNet (Efficient Multi-Scale Convolutional Network) is a U-shaped network model with three pairs of encoder-decoders. The encoder has two outputs. One is to further downsample the deep features for feature fusion, and the other is to perform jump connections on the deep features and fuse them with the upsampled deep features of the corresponding decoder.
6. The image restoration method based on the multi-scale attention mechanism according to claim 1, characterized in that: The training model involved in step 5 learns the multi-scale information of the image, combines local detail information with global structure information, and accurately and comprehensively repairs the details of the image as follows: Given a preprocessed H×W×C damaged image, a 3×3 convolutional layer is used to extract shallow features of the damaged image, which are then passed through three-scale encoder sub-networks to generate deep features and multi-scale information. In this process, the encoder gradually reduces the resolution while expanding the channels. In addition, the low-resolution degraded image is merged into the main path through a shallow layer to better handle the different blur levels in the image, and is concatenated and then 3×3 convolutions are performed to adjust the channels. The deepest features obtained are then input into a three-scale decoder to generate the final features by gradually restoring the features to their original size. In this process, the decoder features are concatenated with the encoder features to assist in reconstruction, and 1×1 convolutions are used to reduce the number of channels by half. Through 3×3 convolutions and image-level skip connections, after three MixBlock blocks, the multi-scale information obtained after upsampling is combined with local detail information and global structure information to finally obtain a nearly realistic repaired image.