A spatially variable blur estimation method based on cross-modal feature fusion

Abstract

This invention relates to the field of image super-resolution technology, specifically to a spatially variable blur estimation method based on cross-modal feature fusion. This method addresses the characteristics of spatially variable blur by proposing cross-modal feature interaction and supervision incorporating semantic information, and designs a hierarchical feature interaction network based on the PyTorch deep learning framework. Through this method, spatially variable blur estimation in real images can be achieved, contributing to image super-resolution technology's ability to recover clearer, higher-resolution results from blurred, low-resolution images. Furthermore, although semantic information is involved, this information is only used during the network training phase and does not require additional input during the testing phase, ensuring the versatility and convenience of our method.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution technology, and more specifically to a spatially variable fuzzy estimation method based on cross-modal feature fusion. Background Technology

[0002] Image super-resolution technology aims to recover the corresponding high-resolution image from a low-resolution image, which is often blurry. Blur estimation techniques can estimate the blur degradation in a low-resolution image to obtain a corresponding blur kernel. This helps to recover the corresponding sharp high-resolution image during the image super-resolution process and has great application potential in many scenarios, such as natural image super-resolution, optical microscopy super-resolution, face image super-resolution, hyperspectral image super-resolution, and video super-resolution.

[0003] With the development of deep learning and the improvement of computing power, many deep learning-based fuzzy estimation methods have achieved good results. However, existing methods can generally only estimate a unique fuzzy kernel for each image and cannot be applied to situations with spatially variable fuzziness such as defocusing, which greatly limits the application potential of such methods.

[0004] To address this issue, a spatially variable blur estimation method more suitable for real-world images is urgently needed. This method can perform fine-grained and distinct blur estimations for each pixel in the image, thus handling both spatially invariant and spatially variable blur scenarios simultaneously. Furthermore, the more realistic blur estimation obtained by this method helps the image super-resolution process recover clearer, higher-resolution results from low-resolution, blurred images, further advancing the practical application of image super-resolution technology. Summary of the Invention

[0005] To address the limitation of current fuzzy estimation techniques in effectively estimating spatially variable fuzzy images, this paper proposes a spatially variable fuzzy estimation method based on cross-modal feature fusion. Considering that the boundaries of fuzzy variations in real images are often aligned with semantic edges, sharp semantic edge information can be used to supervise fuzzy estimation. Based on this, the method utilizes deep learning techniques to construct a network model based on cross-modal feature fusion. This model takes a low-resolution fuzzy image as input and outputs a fuzzy estimation map and a semantic segmentation probability map, thus simultaneously performing fuzzy estimation and semantic segmentation on the input image. In this way, semantic information is introduced to assist fuzzy estimation. Furthermore, the estimated semantic segmentation probability map is identical to the fuzzy quantity, and both can serve as prior information in image super-resolution. Notably, since the model only requires semantic information for supervision during training, no additional conditions are added during application, ensuring the effectiveness and versatility of the proposed method.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A spatially variable fuzzy estimation method based on cross-modal feature fusion, the method comprising:

[0008] A hierarchical cross-modal feature interaction network model is used to input low-resolution blurred images.

[0009] In the hierarchical feature interaction model, multi-scale features of the input image are extracted through a fully convolutional backbone network.

[0010] Except for the first scale, features at all other scales are fused with features from the previous scale using the GIA module;

[0011] For each scale of features, the corresponding task head module is used to predict fuzzy and semantic features respectively;

[0012] Except for the last scale, the fuzzy and semantic features of each scale are interacted and fused through the GIA module;

[0013] The final scale of fuzzy and semantic features are obtained directly through a convolutional layer to obtain the fuzzy quantity map and the semantic segmentation probability map, respectively, which are used for auxiliary supervision of the overall network.

[0014] The fuzzy and semantic features extracted by the corresponding task head modules at each scale are also interacted and aggregated by an additional GIA module; the aggregated features of the two modalities are each passed through a convolutional layer and an upsampling layer to obtain the final fuzzy estimation and semantic segmentation results, respectively.

[0015] More preferably, the method includes:

[0016] First, the low-resolution blurred image I LR Multi-scale features {F} are extracted using a fully convolutional backbone network. 0 ,F 1 ,...,F n}; For the feature F of the first scale 0 Using two specific task header modules ( and The prediction yields the initial fuzzy features at the first scale. and semantic features Then via the GIA module right and By interacting and fusing, the adjusted fuzzy features corresponding to the first scale are obtained. and semantic features The adjusted fuzzy features of the first scale and semantic features Features F at the adjacent second scale 1 Through the GIA module ( and The components are fused together to obtain the output result. and The obtained output results and As a specific task head module of the second scale ( and The input is used to predict the initial fuzzy features at the second scale. and semantic features Repeat the above process until the feature interaction and fusion at the nth scale are completed;

[0017] The last scale of fuzzy and semantic features and The fuzzy map and semantic segmentation probability map are obtained directly through a single convolutional layer, respectively, and used for auxiliary supervision of the overall network.

[0018] Each scale also applies an additional GIA module. To obtain the final multi-scale features of fuzziness and semantics and The final i-scale features of both fuzziness and semantics and The data are aggregated separately and then passed through a convolutional layer and an upsampling layer to obtain the output of the overall network, namely the fuzzy estimation result B and the semantic segmentation result S.

[0019] Further preferably, the fully convolutional backbone network employs an arbitrary multi-scale feature extractor, and the specific task head module... and Each consists of two basic residual blocks.

[0020] Further preferably, for the extracted multi-scale features, unlike the other scales, in the last scale, the task head module... and The output is directly passed through a convolutional layer to predict the intermediate fuzzy map and semantic segmentation probability map, which are used for auxiliary supervision, and an L1 loss function is employed. and Cross-Entropy loss function Apply constraints.

[0021] Further preferred, for the final estimation results, namely the fuzzy quantity map B and the semantic segmentation probability map S, the L1 loss function is applied respectively. and Cross-Entropy loss function Apply constraints. The sum of these values ​​constitutes the overall network loss function.

[0022] More preferably, the GIA module takes two different features as input, outputs two corresponding adjusted features, and includes an upsampling layer that can support input features of different resolutions.

[0023] When two input features have different resolutions, the upsampling layer will upsample the input with the lower resolution.

[0024] When the resolution of the two input features is the same, the upsampling layer does not perform any operation.

[0025] More preferably, in the GIA module, the interaction of features is achieved through two parallel streams in the spatial and channel dimensions. For each input feature, the corresponding outputs of these two streams are aggregated to form the final output feature of that feature.

[0026] More preferably, in the GIA module, Spatial Stream divides the two input features into fixed-size windows through a partition layer, and there is a convolution layer before and after to adjust the features;

[0027] The two features after being divided into windows are grouped and interacted through the Feature Interaction module, and the number of channels is adjusted to 1 through a convolutional layer to obtain weights containing interaction information.

[0028] After being divided into windows, the two features are each directly passed through a convolutional layer to obtain weights that contain only their own features;

[0029] The weights containing interactive information and the weights containing only their own features are both H×W, where H represents the height of the window and W represents the width of the window. They are added together and used to adjust the corresponding features.

[0030] The adjusted two window features are converted back to the original input feature size through a Reverse layer, and then smoothed through a convolution layer to obtain the final output of the Spatial Stream.

[0031] More preferably, in the GIA module, the two input features of the Channel Stream are respectively passed through a convolutional layer and a global average pooling layer to obtain weights that contain only their own features;

[0032] These two weights interact through the Feature Interaction module, with a fully connected layer before and after each layer adjusting them to obtain weights that include interaction information;

[0033] The weights containing interactive information and the weights containing only their own features are both C, where C represents the number of channels of the two input features. They are added together and used to adjust the corresponding features.

[0034] The two adjusted features are smoothed by a convolutional layer to obtain the final output of the Channel Stream.

[0035] More preferably, in the GIA module, the Feature Interaction module groups and interacts the features, taking two different features as input and outputting two corresponding interacted features;

[0036] When the size of two input features in the Spatial Stream is H×W×C, the input features are divided into H×W groups, with each group having C elements.

[0037] When the size of two input features in a Channel Stream is C, the input features are divided into k groups, each with r elements, where C = k × r;

[0038] The two input features after grouping interact through an inner product.

[0039] This method proposes a hierarchical cross-modal feature interaction network framework and a GIA module for feature interaction. By introducing additional semantic information—namely, simultaneously predicting the fuzzy quantity map and the semantic segmentation probability map—it improves the accuracy of spatially variable fuzzy estimation. Specifically, the hierarchical cross-modal feature interaction network framework uses low-resolution fuzzy images I... LR As input, the fuzzy quantity map B and the semantic segmentation probability map S are used as outputs.

[0040] This method, based on the Py-Torch deep learning framework, first extracts deep features at multiple scales from the input image through a shared backbone network. Then, a hierarchical structure is used to interact and fuse fuzzy and semantic information, allowing them to supervise each other to obtain more accurate estimation results. This design not only enables spatially variable fuzzy estimation in images but also predicts additional semantic information, which can help recover more natural image details in subsequent image super-resolution processes. Specifically, the method includes a hierarchical feature interaction framework and a Grouped Feature Fusion (GIA) module, which can effectively perform interaction, supervision, and fusion of the two modalities. The GIA module takes two original features as input and outputs two corresponding adjusted features, while also including an upsampling layer to support inputs at different resolutions. During training, L1 loss and Cross-Entropy loss functions are used for fuzzy estimation and semantic segmentation, respectively, and their sum is used as the overall network loss function.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] This invention addresses spatially variable blur degradation in the real world, specifically estimating spatially variable blur in images. The estimation results can be used as prior information during image super-resolution to help recover sharper, higher-resolution images. Since the boundaries of blur variations are typically aligned with semantic edges, this invention proposes using sharp semantic edge information to supervise blur estimation; specifically, it simultaneously predicts both the blur magnitude map and the semantic segmentation probability map. This design not only enables spatially variable blur estimation in images but also predicts additional semantic information, both of which can help recover more natural image details during subsequent image super-resolution.

[0043] Meanwhile, although the method of the present invention involves the introduction of semantic information to assist in estimating spatially variable blur in an image, this information is only used during the network training phase and does not require any additional input during the testing phase, which ensures the versatility and convenience of our method. Attached Figure Description

[0044] Figure 1 The diagram provided shows the overall structure of the method of this invention.

[0045] Figure 2 The diagram shown is a structural diagram of the GIA module of this invention.

[0046] Figure 3 The figures shown are the processing results of Embodiments 1-6 of the present invention on the NYUv2 dataset.

[0047] Figure 4The following are the processing results of embodiments 7-10 of the present invention on the Cityscapes dataset. Detailed Implementation

[0048] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Based on existing technical problems, this invention specifically develops and studies a spatially variable fuzzy estimation method based on cross-modal feature fusion. This method utilizes the Py-Torch deep learning framework, proposing and constructing a hierarchical cross-modal feature interaction network model and a GIA module for feature interaction. By introducing additional semantic information—namely, simultaneously predicting the fuzzy quantity map and the semantic segmentation probability map—the accuracy of spatially variable fuzzy estimation is improved. This model uses low-resolution fuzzy images I... LR The fuzzy estimation result B and the semantic segmentation result S are used as inputs. The structure of the hierarchical cross-modal feature interaction network model is as follows: Figure 1 As shown, the GIA module is used to interact with two features, and the network's image processing includes the following steps.

[0050] This method is a spatially variable fuzzy estimation method based on cross-modal feature fusion. Specifically, the method is as follows:

[0051] A hierarchical cross-modal feature interaction network model is used as input to low-resolution blurred images.

[0052] In the hierarchical feature interaction model, multi-scale features of the input image are extracted through a fully convolutional backbone network; the fully convolutional backbone network employs an arbitrary multi-scale feature extractor, and the specific task head module... and Each consists of two basic residual blocks.

[0053] Regarding the extracted multi-scale features, unlike the other scales, in the last scale, the task header module... and The output is directly passed through a convolutional layer to predict the intermediate fuzzy map and semantic segmentation probability map, which are used for auxiliary supervision, and an L1 loss function is employed. and Cross-Entropy loss function Apply constraints.

[0054] Except for the first scale, features at all other scales are fused with features from the previous scale using the GIA module;

[0055] For each scale of features, the corresponding task head module is used to predict fuzzy and semantic features respectively;

[0056] Except for the last scale, the fuzzy and semantic features of each scale are interacted and fused through the GIA module;

[0057] The final scale of fuzzy and semantic features are obtained directly through a convolutional layer to obtain the fuzzy quantity map and the semantic segmentation probability map, respectively, which are used for auxiliary supervision of the overall network.

[0058] The fuzzy and semantic features extracted by the corresponding task head modules at each scale are further interacted and aggregated through an additional GIA module. The aggregated features from the two modalities are then passed through a convolutional layer and an upsampling layer, respectively, to obtain the final fuzzy estimation and semantic segmentation results. For the final estimation results, namely the fuzzy quantity map B and the semantic segmentation probability map S, the L1 loss function is applied respectively. and Cross-Entropy loss function Apply constraints. The sum of these values ​​constitutes the overall network loss function.

[0059] The method is described in detail in the following steps:

[0060] First, the low-resolution blurred image I LR Multi-scale features {F} are extracted using a fully convolutional backbone network. 0 ,F 1 ,...,F n}; For the feature F of the first scale 0 Using two specific task header modules ( and The prediction yields the initial fuzzy features at the first scale. and semantic features Then via the GIA module right and By interacting and fusing, the adjusted fuzzy features corresponding to the first scale are obtained. and semantic features The adjusted fuzzy features of the first scale and semantic features Features F at the adjacent second scale 1 Through the GIA module ( and The components are fused together to obtain the output result. and The obtained output results and As a specific task head module of the second scale ( and The input is used to predict the initial fuzzy features at the second scale. and semantic features Repeat the above process until the feature interaction and fusion at the nth scale are completed;

[0061] The last scale of fuzzy and semantic features and The fuzzy map and semantic segmentation probability map are obtained directly through a single convolutional layer, respectively, and used for auxiliary supervision of the overall network.

[0062] Each scale also applies an additional GIA module. To obtain the final multi-scale features of fuzziness and semantics and The final i-scale features of both fuzziness and semantics and The data are aggregated separately and then passed through a convolutional layer and an upsampling layer to obtain the output of the overall network, namely the fuzzy estimation result B and the semantic segmentation result S.

[0063] Based on the above methods, the GIA module is explained in detail below.

[0064] The GIA module takes two different features as input, outputs two corresponding adjusted features, and includes an upsampling layer that can support input features of different resolutions.

[0065] When two input features have different resolutions, the upsampling layer will upsample the input with the lower resolution.

[0066] When the resolution of the two input features is the same, the upsampling layer does not perform any operation.

[0067] In the GIA module, feature interaction is achieved through two parallel streams in the spatial and channel dimensions. For each input feature, the corresponding outputs of these two streams are aggregated to form the final output feature of that feature.

[0068] In the GIA module, Spatial Stream divides the two input features into fixed-size windows through a partition layer, and then performs convolutional layer before and after each feature adjustment.

[0069] The two features after being divided into windows are grouped and interacted through the Feature Interaction module, and the number of channels is adjusted to 1 through a convolutional layer to obtain weights containing interaction information.

[0070] After being divided into windows, the two features are each directly passed through a convolutional layer to obtain weights that contain only their own features;

[0071] The weights containing interactive information and the weights containing only their own features are both H×W, where H represents the height of the window and W represents the width of the window. They are added together and used to adjust the corresponding features.

[0072] The adjusted two window features are converted back to the original input feature size through a Reverse layer, and then smoothed through a convolution layer to obtain the final output of the Spatial Stream.

[0073] In the GIA module, the two input features of the Channel Stream are respectively passed through a convolutional layer and a global average pooling layer to obtain weights that contain only their own features;

[0074] These two weights interact through the Feature Interaction module, with a fully connected layer before and after each layer adjusting them to obtain weights that include interaction information;

[0075] The weights containing interactive information and the weights containing only their own features are both C, where C represents the number of channels of the two input features. They are added together and used to adjust the corresponding features.

[0076] The two adjusted features are smoothed by a convolutional layer to obtain the final output of the Channel Stream.

[0077] In the GIA module, the Feature Interaction module groups and interacts features, taking two different features as input and outputting two corresponding interacted features.

[0078] When the size of two input features in the Spatial Stream is H×W×C, the input features are divided into H×W groups, with each group having C elements.

[0079] When the size of two input features in a Channel Stream is C, the input features are divided into k groups, each with r elements, where C = k × r;

[0080] The two input features after grouping interact through an inner product.

[0081] Take a practical application as an example.

[0082] First, the low-resolution blurred image I LR Input hierarchical cross-modal feature interaction network model.

[0083] Second, in the hierarchical cross-modal feature interaction network model, multi-scale features {F} of the input image are extracted through a fully convolutional backbone network. 0 ,F 1 ,...,F n The fully convolutional backbone network can employ any multi-scale feature extractor.

[0084] Third, regarding the feature F of the first scale 0 Using two specific task header modules ( and The prediction yields the initial fuzzy features at the first scale. and semantic features Specific task header module and Each consists of two basic residual blocks;

[0085] Then via the GIA module right and By interacting and fusing, the adjusted fuzzy features corresponding to the first scale are obtained. and semantic features

[0086] The adjusted fuzzy features of the first scale and semantic features Features F at the adjacent second scale 1 Through the GIA module ( and The components are fused together to obtain the output result. and

[0087] The obtained output results and Each serves as a specific task header module for the second scale. and The input is used to predict the initial fuzzy features at the second scale. and semantic features

[0088] Repeat the above process until the feature interaction and fusion at the nth scale are completed.

[0089] Fourth, the final scale of fuzzy features extracted by the specific task head module. and semantic features Unlike the previous n scales, the GIA module was not used for interaction and fusion. Instead, additional fuzzy estimation maps and semantic segmentation probability maps were obtained directly through a single convolutional layer for auxiliary supervision of the overall network. L1 loss functions were used for fuzzy estimation and semantic segmentation, respectively. and Cross-Entropy loss function Apply constraints.

[0090] Fifth, an additional GIA module is applied to each scale. To obtain the final multi-scale features of fuzziness and semantics and The obtained multi-scale features and The data is aggregated separately, and then passed through a convolutional layer (conv) and an upsampling layer (upsample) to obtain the overall network output, namely the fuzzy estimation result B and the semantic segmentation result S. The L1 loss function is then applied to the final fuzzy estimation result B and semantic segmentation result S, respectively. and Cross-Entropy loss function Apply constraints. The sum of these values ​​constitutes the overall network loss function.

[0091] In the above method, specifically, the GIA module is used for the interaction and adjustment of two different features. This module takes two different features as input and outputs two corresponding adjusted features. The structure of the GIA module is as follows: Figure 2 As shown, its feature processing includes the following steps.

[0092] First, the two input features are passed through an upsampling layer;

[0093] When two input features have different resolutions, the upsampling layer will upsample the input with the lower resolution.

[0094] When the resolution of the two input features is the same, the upsampling layer does not perform any operation.

[0095] Then, the interaction of features is achieved through two parallel streams in the spatial and channel dimensions. For a given input feature, the corresponding outputs of these two streams are aggregated to form the final output feature of that feature.

[0096] In the GIA module, Spatial Stream divides the two input features into fixed-size windows through a Partition layer, and there is a convolutional Conv layer before and after to adjust the features;

[0097] The two features after being divided into windows are grouped and interacted through the Feature Interaction module, and the number of channels is adjusted to 1 through a convolutional Conv layer to obtain weights containing interaction information.

[0098] After being divided into windows, the two features are each directly passed through a convolutional layer (Conv) to obtain weights that contain only their own features;

[0099] The weights containing interactive information and the weights containing only their own features are both H×W, where H represents the height of the window and W represents the width of the window. They are added together and used to adjust the corresponding features. The adjustment method is pixel-level multiplication.

[0100] The adjusted two window features are converted back to the original input feature size through a Reverse layer, and then smoothed through a Conv layer to obtain the final output of the Spatial Stream.

[0101] In the GIA module, the two input features of the Channel Stream are first passed through a convolutional layer (Conv) and a global average pooling layer (GAP) to obtain weights that contain only their own features.

[0102] These two weights interact through the Feature Interaction module, with a fully connected MLP layer before and after each layer adjusting them to obtain weights that include interaction information.

[0103] The weights containing interactive information and the weights containing only their own features are both C, where C represents the number of channels of the two input features. They are added together and used to adjust the corresponding features. The adjustment method is also pixel-level multiplication.

[0104] The two adjusted features are smoothed by a single convolutional layer (Conv) to obtain the final output of the Channel Stream.

[0105] In the GIA module, the Feature Interaction module is used to group two different features for interaction. This module takes two different features as input and outputs two corresponding interactive features.

[0106] When the size of two input features in the Spatial Stream is H×W×C, the input features are divided into H×W groups, with each group having C elements.

[0107] When the size of two input features in a Channel Stream is C, the input features are divided into k groups, each with r elements, where C = k × r;

[0108] The two input features after grouping interact through an inner product.

[0109] Examples 1-6: Following the method described above, this is implemented on the NYUv2 dataset to perform spatially variable blur estimation for low-resolution images. First, the dataset needs to be processed to simulate real-world defocus blur; the processed dataset is named NYUv2-BSR. Using the method proposed on pages 2-3 of the paper "Deep Defocus Map Estimation using Domain Adaptation," all RGB images are processed into a defocus blur effect based on the depth map provided by the NYUv2 dataset, and the blur amount map corresponding to each processed image is recorded. Then, the blurred images are downsampled, with a downsampling factor of 4, resulting in low-resolution images containing spatially variable blur. Similar to the NYUv2 dataset, the NYUv2-BSR dataset also includes training and testing sets. Since each image in NYUv2 is correspondingly processed into a low-resolution blurred image, the number of processed images remains unchanged.

[0110] During network training, low-resolution blurred images from the training set of the NYUv2-BSR dataset were used as input, with corresponding blur level maps and semantic segmentation probability maps used as supervision. The semantic segmentation probability map was provided in the NYUv2 dataset. Specifically, two data augmentation methods were applied to the input images during training: horizontal flipping with a probability of 0.5; and random scaling with equal probability, where the scaling factors were selected as 1, 1.2, and 1.5. The network model was trained using the Adam optimizer (β1 = 0.9, β2 = 0.99) traversing all images 700 times, with a batch size of 8. The learning rate was initialized to 0.0001, using a cosine learning rate decay method, and the number of heating cycles was selected as 10.

[0111] After training, the trained network model is used to perform blur estimation and semantic segmentation on low-resolution blurred images in the test set of the NYUv2-BSR dataset, obtaining the corresponding blur quantity map and semantic segmentation probability map. Since both the blur quantity map and the semantic segmentation probability map can serve as prior information to help non-blind image super-resolution methods recover more natural image details—which is one application of our method—the estimated blur quantity map and semantic segmentation probability map are used as prior information to input a non-blind super-resolution network to assist in image super-resolution. The ground truth of the image super-resolution result is the original RGB image in the NYUv2 dataset. The closer the super-resolution result is to the original RGB image, the more effective the blur estimation method proposed in this invention. The selected non-blind image super-resolution method is the network model proposed on page 1 of the supplementary materials in the paper "Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution". Figure 3 Examples of the effects of fuzzy estimation, semantic segmentation, and image super-resolution are given.

[0112] exist Figure 3 middle:

[0113] The first row (Low-Resolution) is a low-resolution image with out-of-focus blur generated from the depth map and RGB image.

[0114] The second row (Blur Map) shows the estimated blur level, with colors ranging from dark to light corresponding to the transition from sharp to blurry in the RGB image.

[0115] The third row (Semantic Segmentation) shows the estimated semantic segmentation results.

[0116] The fourth row (Restored Image) shows the super-resolution results obtained based on the estimated blur map and semantic segmentation results.

[0117] The last line (Ground-truth) is the corresponding high-resolution, clear RGB image.

[0118] Examples 7-10: Following the method described above, this is implemented on the Cityscapes dataset to perform spatially variable blur estimation for low-resolution images. First, the dataset needs to be processed to simulate real-world defocus blur; the processed dataset is named Cityscapes-BSR. Using the method proposed on pages 2-3 of the paper "Deep Defocus MapEstimation using Domain Adaptation," all RGB images are processed into a defocus blur effect based on the depth map provided by the Cityscapes dataset, and the blur amount map corresponding to each processed image is recorded. Then, the blurred images are downsampled, with a downsampling factor of 4, resulting in low-resolution images containing spatially variable blur. Similar to the Cityscapes dataset, the Cityscapes-BSR dataset also includes training, validation, and test sets. Since each image in Cityscapes is correspondingly processed into a low-resolution blurred image, the number of processed images remains unchanged.

[0119] During network training, low-resolution blurred images from the Cityscapes-BSR dataset training set were used as input, with corresponding blur level maps and semantic segmentation probability maps used as supervision. The semantic segmentation probability map was provided in the Cityscapes dataset. The input images were cropped to 160x120 pixels. Specifically, two data augmentation methods were applied to the input images during training: horizontal flipping with a probability of 0.5; and random scaling with equal probability, where scaling factors were selected as 1, 1.2, and 1.5. The network model was trained using the Adam optimizer (β1 = 0.9, β2 = 0.99) traversing all images 700 times, with a batch size of 8. The learning rate was initialized to 0.0001, using a cosine learning rate decay method, and the number of heating cycles was selected as 10.

[0120] After training, the trained network model is used to perform blur estimation and semantic segmentation on low-resolution blurred images in the Cityscapes-BSR dataset test set, obtaining the corresponding blur quantity map and semantic segmentation probability map. Similarly, to better verify the effectiveness of the spatially variable blur estimation method proposed in this invention, the estimated blur quantity map and semantic segmentation probability map are used as prior information to input a non-blind super-resolution network to assist in image super-resolution. The ground truth of the image super-resolution result is the original RGB image in the Cityscapes dataset. The closer the super-resolution result is to the original RGB image, the more effective the blur estimation method proposed in this invention is. The non-blind image super-resolution method selected is the network model proposed on page 1 of the supplementary materials in the paper "Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution". Figure 4 Examples of the effects of fuzzy estimation, semantic segmentation, and image super-resolution are given.

[0121] exist Figure 4 middle:

[0122] The first row (Low-Resolution) is a low-resolution image with out-of-focus blur generated from the depth map and RGB image.

[0123] The second row (Blur Map) shows the estimated blur level, with colors ranging from dark to light corresponding to the transition from sharp to blurry in the RGB image.

[0124] The third row (Semantic Segmentation) shows the estimated semantic segmentation results.

[0125] The fourth row (Restored Image) shows the super-resolution results obtained based on the estimated blur map and semantic segmentation results.

[0126] The last line (Ground-truth) is the corresponding high-resolution, clear RGB image.

[0127] In the field of image super-resolution, most previous methods only estimated a unique blur kernel for a single image, assuming that the blur is spatially invariant. The MANet method, proposed on pages 2-4 of the paper "Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution," is the first method designed to estimate spatially variable blur and is currently the most effective method for estimating spatially variable blur in this field. Therefore, to demonstrate the necessity of estimating spatially variable blur and the effectiveness of the method proposed in this invention, a comparison is made with five classic image super-resolution methods: KernelGAN, DAN, DCLS, KOALANet, and MANet. KernelGAN was proposed on pages 4-6 of the paper "Blind Super-Resolution Kernel Estimation using an Internal-GAN," DAN on pages 3-5 of the paper "Unfolding the Alternating Optimization for Blind SuperResolution," DCLS on pages 3-5 of the paper "Deep Constrained Least Squares for Blind Image Super-Resolution," and KOALANet on pages 3-5 of the paper "KOALAnet: Blind Super-Resolution using Kernel-Oriented Adaptive Local Adjustment." KernelGAN, DAN, DCLS, and KOALANet all aim to estimate spatially invariant blur.

[0128] The above method was implemented and compared on the NYUv2-BSR dataset. The PSNR and SSIM values ​​between the final super-resolution result and the corresponding high-resolution sharp image were used as evaluation metrics. Higher PSNR and SSIM values ​​indicate better performance of the corresponding method. The comparison results are shown in Table 1. Here, Ours represents the method proposed in this invention. Clearly, the method proposed in this invention is far superior to spatially invariant blur estimation methods such as KernelGAN, KOALANet, DCLS, and DAN, and its performance also surpasses the current best spatially variable blur estimation method, MANet.

[0129] Table 1 compares the effectiveness of the methods on the NYUv2-BSR dataset.

[0130] method KernelGAN KOALANet DCLS DAN MANet Ours PSNR / SSIM 23.15 / 0.7446 27.70 / 0.8758 27.89 / 0.8781 27.90 / 0.8791 30.12 / 0.9106 32.03 / 0.9154

[0131] The above method was implemented and compared on the Cityscapes-BSR dataset. The PSNR and SSIM values ​​between the final super-resolution result and the corresponding high-resolution sharp image were used as evaluation metrics. Higher PSNR and SSIM values ​​indicate better performance of the corresponding method. The comparison results are shown in Table 2. Here, Ours represents the method proposed in this invention. Clearly, the method proposed in this invention is far superior to spatially invariant blur estimation methods such as KernelGAN, KOALANet, DCLS, and DAN, and its performance also surpasses the current best spatially variable blur estimation method, MANet.

[0132] Table 2 compares the effectiveness of the methods on the Cityscapes-BSR dataset.

[0133] method KernelGAN KOALANet DCLS DAN MANet Ours PSNR / SSIM 28.96 / 0.8469 32.38 / 0.9166 32.39 / 0.9167 32.32 / 0.9156 34.25 / 0.9287 35.58 / 0.9383

[0134] This invention addresses spatially variable blur degradation in the real world, specifically estimating spatially variable blur in images. The estimation results can be used as prior information during image super-resolution to help recover sharper, higher-resolution images. Since the boundaries of blur variations are typically aligned with semantic edges, this invention proposes using sharp semantic edge information to supervise blur estimation; specifically, it simultaneously predicts both the blur magnitude map and the semantic segmentation probability map. This design not only enables spatially variable blur estimation in images but also predicts additional semantic information, both of which can help recover more natural image details during subsequent image super-resolution.

[0135] Meanwhile, although the method of the present invention involves the introduction of semantic information to assist in estimating spatially variable blur in an image, this information is only used during the network training phase and does not require any additional input during the testing phase, which ensures the versatility and convenience of our method.

[0136] It will be apparent to those skilled in the art that the above embodiments are not intended to be exhaustive or to limit the embodiments to the specific, precise forms disclosed. It will also be apparent to those skilled in the art that modifications, combinations, and variations can be made based on the above teachings.

Claims

1. A spatially variable fuzzy estimation method based on cross-modal feature fusion, characterized in that: The method includes: A hierarchical cross-modal feature interaction network model is used to input low-resolution blurred images. In the hierarchical feature interaction model, multi-scale features of the input image are extracted through a fully convolutional backbone network. Except for the first scale, features at all other scales are fused with features from the previous scale using the GIA module; For each scale of features, the corresponding task head module is used to predict fuzzy and semantic features respectively; Except for the last scale, the fuzzy and semantic features of each scale are interacted and fused through the GIA module; The final scale of fuzzy and semantic features are obtained directly through a convolutional layer to obtain the fuzzy quantity map and the semantic segmentation probability map, respectively, which are used for auxiliary supervision of the overall network. The fuzzy and semantic features extracted by the corresponding task head modules at each scale are also interacted and aggregated by an additional GIA module; the aggregated features of the two modalities are each passed through a convolutional layer and an upsampling layer to obtain the final fuzzy estimation and semantic segmentation results, respectively.

2. The spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: The method includes: First, the low-resolution blurred image I LR Multi-scale features {F} are extracted using a fully convolutional backbone network. 0 ,F 1 ,...,F n }; For the feature F of the first scale 0 Using two specific task header modules and Predicting the initial fuzzy features at the first scale and semantic features Then, through the GIA module right and By interacting and fusing, the adjusted fuzzy features corresponding to the first scale are obtained. and semantic features The adjusted fuzzy features of the first scale and semantic features Features F at the adjacent second scale 1 via GIA module and The results are obtained by fusion. and The obtained output results and As a specific task header module of the second scale and The input is used to predict the initial fuzzy features at the second scale. and semantic features Repeat the above process until the feature interaction and fusion at the nth scale are completed; The last scale of fuzzy and semantic features and The fuzzy map and semantic segmentation probability map are obtained directly through a single convolutional layer, respectively, and used for auxiliary supervision of the overall network. Each scale also applies an additional GIA module. To obtain the final multi-scale features of fuzziness and semantics and The final i-scale features of both fuzziness and semantics and The data are aggregated separately and then passed through a convolutional layer and an upsampling layer to obtain the output of the overall network, namely the fuzzy estimation result B and the semantic segmentation result S.

3. The spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: The fully convolutional backbone network employs an arbitrary multi-scale feature extractor, and the corresponding task head module is a specific task head module. and The specific task header module and Each consists of two basic residual blocks.

4. The spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: Regarding the extracted multi-scale features, unlike the other scales, in the last scale, the task header module... and The output is directly passed through a convolutional layer to predict the intermediate fuzzy map and semantic segmentation probability map, which are used for auxiliary supervision, and an L1 loss function is employed. and Cross-Entropy loss function Apply constraints.

5. The spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: For the final estimation results, namely the fuzzy quantity map B and the semantic segmentation probability map S, the L1 loss function is applied respectively. and Cross-Entropy loss function To impose constraints, The sum of these values ​​constitutes the overall network loss function.

6. The spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: The GIA module takes two different features as input, outputs two corresponding adjusted features, and includes an upsampling layer that can support input features of different resolutions. When two input features have different resolutions, the upsampling layer will upsample the input with the lower resolution. When the resolution of the two input features is the same, the upsampling layer does not perform any operation.

7. The spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: In the GIA module, feature interaction is achieved through two parallel streams in the dimensions of Spatial Stream and Channel Stream. For each input feature, the corresponding outputs of these two streams are aggregated to form the final output feature of that feature.

8. The spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: In the GIA module, Spatial Stream divides the two input features into fixed-size windows through a partition layer, and then performs convolutional layer before and after each feature adjustment. The two features after being divided into windows are grouped and interacted through the Feature Interaction module, and the number of channels is adjusted to 1 through a convolutional layer to obtain weights containing interaction information. After being divided into windows, the two features are each directly passed through a convolutional layer to obtain weights that contain only their own features; The weights containing interactive information and the weights containing only their own features are both H×W, where H represents the height of the window and W represents the width of the window. They are added together and used to adjust the corresponding features. The adjusted two window features are converted back to the original input feature size through a Reverse layer, and then smoothed through a convolution layer to obtain the final output of the Spatial Stream.

9. A spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: In the GIA module, the two input features of the Channel Stream are respectively passed through a convolutional layer and a global average pooling layer to obtain weights that contain only their own features; These two weights interact through the Feature Interaction module, with a fully connected layer before and after each layer adjusting them to obtain weights that include interaction information; The weights containing interactive information and the weights containing only their own features are both C, where C represents the number of channels of the two input features. They are added together and used to adjust the corresponding features. The two adjusted features are smoothed by a convolutional layer to obtain the final output of the Channel Stream.

10. A spatially variable fuzzy estimation method based on cross-modal feature fusion according to claim 1, characterized in that: In the GIA module, the Feature Interaction module groups and interacts features, taking two different features as input and outputting two corresponding interacted features. When the size of two input features in the Spatial Stream is H×W×C, the input features are divided into H×W groups, with each group having C elements. When the size of two input features in a Channel Stream is C, the input features are divided into k groups, each with r elements, where C = k × r; The two input features after grouping interact through an inner product.

Images