Unified approach for multiple restoration tasks based on granularity-aware image degradation estimation

Abstract

A unified multi-task restoration method based on granularity-aware image degradation estimation belongs to the field of underlying visual image restoration. The present invention aims to solve the problem of inaccurate conditional guidance modules based on degradation estimation used in existing multi-task methods. The present invention includes the following methods: S1, constructing a fine-grained simulated degradation dataset; S2, training a fine-grained degradation feature extractor; S3, constructing a multi-granularity hierarchical degradation space; S4, constructing a multi-granularity hybrid expert restoration model; S5, based on the trained degradation feature extractor and the multi-granularity hybrid expert restoration model, training a hybrid expert routing network based on degradation estimation and granularity estimation; S6, after training, inputting any low-quality image, the network assigns the optimal expert network to the current low-quality image for processing, and finally obtains a clear image corresponding to the current low-quality image.

Description

Technical Field

[0001] The present invention relates to a unified method for multiple restoration tasks based on granularity-aware image degradation estimation, and belongs to the field of underlying visual image restoration. Background Art

[0002] Low-level vision aims to improve user satisfaction and specific needs by enhancing the visual quality of images and videos, and can provide an important data foundation for other visual understanding tasks. There are many factors that cause visual quality degradation in the imaging process, and users and tasks have a wide variety of visual quality requirements. Therefore, low-level vision often involves solving basic tasks such as deblurring, denoising, removing JPEG compression artifacts, removing rain streaks, removing fog, removing snow, dark light enhancement, and hue conversion, as well as their combinations. It also presents different characteristics with changes in imaging devices and imaging methods, and has always been an important and challenging fundamental problem in computer vision. Early methods usually trained dedicated models for each task separately. The diversity of tasks resulted in relatively large time and storage overhead. Recent research has focused on how to unify multiple restoration tasks under a single framework.

[0003] Existing approaches to unifying multiple restoration tasks can be categorized as task-agnostic, prompt-learning or adapter-based, and hybrid expert network-based approaches. Due to differences in degradation processes and methods, learning how to restore different types of degradation can conflict. This conflict complicates multi-task unification, making it difficult to generalize to multiple tasks while simultaneously achieving good performance across them. Consequently, directly using a general model to handle multiple tasks simultaneously results in significant performance losses. Subsequent approaches have introduced prompt learning and adapters, enabling the base model network to modulate the features of the current base model through guidance based on degradation features or deep features, thereby achieving conditional processing across different tasks. This alleviates the conflict between different tasks to a certain extent and improves the performance of multi-task unification approaches. Furthermore, approaches based on hybrid expert networks further consider the shared nature (cooperation) and specificity (specificity) of different tasks. Based on a shared base model, these approaches train expert filters, expert low-rank adaptation weights, and expert networks separately for each task. During training and inference, these methods perform routing based on the degenerate or deep features of the current input, selecting specialized modules in combination with the base model for processing. While explicitly processing different tasks can further resolve conflicts between multiple tasks, these methods rely heavily on accurate feature representation. Incorrect feature representation can lead to incorrect routing, negatively impacting the restoration process. Summary of the Invention

[0004] Aiming at the problem of inaccuracy in the conditional guidance module based on degradation estimation used in the existing multi-task method, the present invention provides a unified multi-task restoration method based on granularity-aware image degradation estimation.

[0005] The unified multi-restoration task method based on granularity-aware image degradation estimation of the present invention comprises the following steps:

[0006] S1. Build a fine-grained simulated degradation dataset and assign corresponding content text and degradation text to each low-quality image-high-definition image pair.

[0007] S2, training a fine-grained degradation feature extractor using the constructed fine-grained simulated degradation dataset;

[0008] S3. Collect all low-quality images in the training data used for restoration tasks, use the obtained fine-grained degradation feature extractor to obtain a degradation feature set of the low-quality images in the training data, and construct a multi-granularity hierarchical degradation space based on the obtained degradation feature set;

[0009] S4. Based on the multi-granularity hierarchical degradation space, an expert restoration network is trained for each subspace in each granularity, and all expert networks are combined to form a multi-granularity hybrid expert restoration model;

[0010] S5. Based on the trained degradation feature extractor and multi-granularity hybrid expert restoration model, a hybrid expert routing network based on degradation estimation and granularity estimation is trained;

[0011] S6. After training is completed, any low-quality image is input, and the network assigns the optimal expert network to the current low-quality image for processing, and finally obtains a clear image corresponding to the current low-quality image.

[0012] Preferably, the fine-grained simulated degradation dataset in S1 synthesizes low-quality images of different degradation types by simulating degradation, and the degradation types include noise, motion blur, JPEG compression artifacts, rain, snow, fog and dim light. The fine-grained simulated degradation dataset constructs a quadruple of low-quality image-clear image-fine-grained degraded text-content text for each low-quality image.

[0013] Preferably, the fine-grained simulated degradation dataset takes the clear image y as input and synthesizes the corresponding low-quality image x. The construction process is:

[0014] The noise is synthesized as follows:

[0015] x=y+z

[0016] Where z is the 2The Gaussian noise with variance σ is classified into three levels: weak noise, moderate noise, and strong noise. σ<15 is considered weak noise, 15≤σ<35 is considered moderate noise, and 35≤σ<50 is considered strong noise.

[0017] Motion blur is synthesized as follows:

[0018]

[0019] Where k is the blur kernel, and the degradation degree is divided into two gradients, namely small blur and large blur. k<15 is considered small blur, and k>15 is considered large blur.

[0020] JPEG compression artifacts are implemented using the open source tool DIFFJPEG. The degree of degradation is divided into three gradients: heavy compression, medium compression, and light compression. An image quality level greater than 0 and less than 15 is considered heavy compression, an image quality level between 15 and 35 is considered medium compression, and an image quality level greater than 35 is considered light compression.

[0021] The synthesis of degenerate rain is expressed in rain streaks. The synthesis of rain streaks is as follows:

[0022] x=y+r

[0023] Where r is the synthetic rain streak, which is achieved by random noise and Gaussian blur. The noise is related to the intensity of the rain, and the blur kernel is related to the direction, length, and width of the rain streak.

[0024] Snow is crafted as follows:

[0025] x=y×(1-m)+m

[0026] Where m is a randomly sampled mask from the public dataset Snow100K. The degradation degree is divided into three gradients, which are defined based on the classification of heavy snow, moderate snow, and light snow in Snow100K.

[0027] The fog is synthesized as follows:

[0028] x=T(y)+a(1-T(y)),T(y)=e -βD(y)

[0029] Where a is the global atmospheric light intensity, T(y) is the transfer matrix, D(y) is the depth map, and β is the atmospheric scattering coefficient;

[0030] The depth map is estimated using the open-source model MegaDepth. The degradation level is divided into two gradients: light fog and dense fog. β < 0.1 is considered light fog, and β ≥ 0.1 is considered dense fog.

[0031] The synthesis method of dark light is:

[0032] x=yI(y) γ / I(y)

[0033] Where I(y) is the light intensity map, and γ is the dark light coefficient. The light intensity map is obtained by the deep learning model RetinexFormer. The dark light coefficient determines the intensity of the dark light. The degree of degradation is divided into two gradients: weak dark light and strong dark light. 1.5>γ>1 is considered weak dark light, and 3>γ≥1.5 is considered strong dark light.

[0034] According to the above method, a triple of low-quality image, clear image and degraded text is constructed. Then, the open source method BLIP is used to create text information related to the content of each clear image, thereby constructing the training data of the low-quality image, clear image, degraded text and content text quadruple.

[0035] Preferably, the method for training the degradation feature extractor in S2 is as follows: the visual-language model DACLIP is selected as the base model of the degradation feature extractor; based on the fine-grained simulated degradation dataset constructed in step S1, the degradation feature extractor is trained to produce a DACLIP that can simultaneously identify different degradation types and different degradation degrees; during the training process, a network with the same structure as the image encoder in DACLIP is used as an image controller to extract image degradation features; contrast loss is used to align its features with degraded text; and the trained feature extractor is used to obtain degradation features of any low-quality image.

[0036] Preferably, the process of constructing a multi-granularity hierarchical degradation space in S3 is:

[0037] S31. Integrate the low-quality images used in the training data of different restoration tasks, and use the trained degradation feature extractor to obtain the degradation features of all data to form a degradation feature set {e};

[0038] S32. A multi-granularity hierarchical degradation space is constructed through a bottom-up hierarchical clustering method. The multi-granularity hierarchical degradation space is divided into three granularity levels, where the 0th granularity level represents the coarsest granularity and the 2nd granularity level represents the finest granularity.

[0039] The bottom-up hierarchical clustering method is expressed as:

[0040] {u} n ←Kmeans({e})

[0041] where {u} n It is the set of cluster centers obtained after clustering at three granularity levels of n = 0, 1, and 2. First, clustering is performed at the finest granularity level. Then, based on the clustering at the finest granularity level, clustering is continued upward to obtain the set of cluster centers at two coarse granularity levels, and then the set of cluster centers at each granularity level {u}0, {u}1, {u}2 is obtained.

[0042] S33, for the input sample belonging to the nth level, through the sample degradation feature and all cluster centers u of the granularity level ni The minimum distance is used to divide the current sample into the cluster closest to it, i = 1, 2, ..., p, p is the number of cluster centers at the nth granularity level, so that the training data is divided into different subsets based on different granularities, and all subsets are constructed into a multi-granularity hierarchical degradation space.

[0043] Preferably, the restoration task includes a denoising task, a deblurring task, a compression artifact removal task, a rain streak removal task, a fog removal task and a dark light enhancement task.

[0044] Preferably, the construction process of the multi-granularity hybrid expert restoration model in S4 is:

[0045] For the dark light enhancement task, the low-light image enhancement network RetinexFormer is selected as the expert model;

[0046] The Transformer model Restormer is selected as the expert model for denoising, deblurring, compression artifact removal, rain streak removal, and dehazing tasks;

[0047] Dark light enhancement, deblurring, and defogging tasks are each processed using two expert models; snow removal, denoising, and compression artifact removal tasks are each processed using three expert models, and rain streak removal tasks are processed using four expert models;

[0048] The expert model is trained using L1 loss l1:

[0049]

[0050] in is the network prediction result, and Y is the label; after training, a multi-granularity multi-expert restoration model is constructed.

[0051] Preferably, the training process of S5 includes:

[0052] S51. Degradation estimation based on degradation features using two sets of multi-layer perceptrons deg and particle size estimation e gtan ;

[0053] S52, reusing the routing network based on degradation estimation and granularity estimation and Do routing based on degradation estimation and granularity estimation:

[0054]

[0055] in is a collection of multiple expert networks at the nth granularity level, Based on The kth expert network selected by the routing at the nth granularity level, For multi-granularity hybrid expert model The top-down expert network set of the branch, Based on Routing in Select the network with the best performance;

[0056] S53. The following loss function is used for the constraints of degradation estimation and granularity estimation:

[0057]

[0058] in is the i-th cluster center at the finest granularity level;

[0059] S54. Optimize the degradation estimation network, the routing network based on degradation estimation, the granularity estimation network, and the routing network based on granularity estimation by jointly using the following loss function:

[0060]

[0061] in It is a common load balancing loss in the hybrid expert model, which is used to balance the load of each expert. α and β are two hyperparameters used to determine and The influence degrees of α and β are set to 0.1 and 0.01 respectively.

[0062] Preferably, the hybrid expert routing network based on degradation estimation and granularity estimation in S6 includes a routing network based on degradation estimation and a routing network based on granularity estimation;

[0063] The degradation estimation network includes fully connected layers MD1 to MD5;

[0064] MD1 is used to perform the first full connection operation and the first activation operation on the degradation features of the input image;

[0065] MD2 is used to perform a second full connection operation and a second activation operation on the output features of MD1;

[0066] MD3 is used to perform the third full connection operation and the third activation operation on the output features of MD2;

[0067] MD4 is used to perform the fourth full connection operation and the fourth activation operation on the output features of MD3;

[0068] MD5 is used to perform the fifth full connection operation and the fifth activation operation on the output features of MD4;

[0069] The first to fifth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The first to fourth activation operations are all LeakyReLU activation functions, and the fifth activation operation is a one-dimensional BatchNorm operation.

[0070] The granularity estimation network includes fully connected layers MG1 to MG5;

[0071] MG1 is used to perform the sixth full connection operation and the sixth activation operation on the degradation features of the input image;

[0072] MG2 performs the seventh full connection operation and the seventh activation operation on the output features of MG1;

[0073] MG3 performs an eighth full connection operation and an eighth activation operation on the output features of MG2;

[0074] MG4 performs the ninth full connection operation and the ninth activation operation on the output features of MG3;

[0075] MG5 performs the tenth full connection operation and the tenth activation operation on the output features of MG4;

[0076] The sixth to tenth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The sixth to ninth activation operations are all LeakyReLU activation functions, and the tenth activation operation is a one-dimensional BatchNorm operation.

[0077] The routing network based on degradation estimation includes a fully connected layer GD1;

[0078] GD1 is used to perform the eleventh fully connected operation on the degradation estimation of the MD5 output image;

[0079] The routing network based on granularity estimation includes a fully connected layer GG1;

[0080] GG1 is used to perform the twelfth fully connected operation on the granularity estimation of the MG5 output image;

[0081] The input dimension of the eleventh fully connected operation is 512, and the output dimension is 19;

[0082] The input dimension of the twelfth fully connected operation is 512 and the output dimension is 3.

[0083] Beneficial effects of the present invention: The unified multi-restoration task method based on granularity-aware degradation estimation described in the present invention, in order to mitigate the misguidance caused by image degradation estimation errors, introduces the granularity factor in image degradation estimation for the first time to express the magnitude of the current image degradation estimation error. The present invention is the first to conduct research at a granularity finer than the degradation type level, and proposes a multi-granularity hierarchical degradation space, and constructs a multi-granularity hybrid expert restoration model. Further, in the process of multi-expert routing, the present invention proposes routing based on image degradation estimation and granularity estimation, which can find a more robust and accurate degradation expression for any input low-quality image, thereby finding a suitable expert restoration model for processing. Compared with previous methods, the present invention has achieved a huge improvement in multi-task unified performance, greatly narrowing the gap with single-task proprietary models.

[0084] Although the present invention has been described herein with reference to specific implementations, it should be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It should be understood that numerous modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that features described herein may be combined in different ways than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 is a flow chart of a unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to the present invention;

[0086] Figure 2 This is a schematic diagram of the comparison results between the method proposed in this invention and other methods on multiple restoration tasks.

[0087] Figure 3 It is the network structure diagram of degradation estimation and granularity estimation. DETAILED DESCRIPTION

[0088] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0089] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0090] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0091] Specific implementation method 1: Figure 1 and Figure 2 This embodiment describes a unified method for multiple restoration tasks based on granularity-aware image degradation estimation, see Figure 1 As shown, the method includes the following steps:

[0092] S1. Build a fine-grained simulated degradation dataset and assign corresponding content text and degradation text to each low-quality image-high-definition image pair.

[0093] S2, training a fine-grained degradation feature extractor using the constructed fine-grained simulated degradation dataset;

[0094] S3. Collect all low-quality images in the training data used for restoration tasks, use the obtained fine-grained degradation feature extractor to obtain a degradation feature set of the low-quality images in the training data, and construct a multi-granularity hierarchical degradation space based on the obtained degradation feature set;

[0095] S4. Based on the multi-granularity hierarchical degradation space, an expert restoration network is trained for each subspace in each granularity, and all expert networks are combined to form a multi-granularity hybrid expert restoration model;

[0096] S5. Based on the trained degradation feature extractor and multi-granularity hybrid expert restoration model, a hybrid expert routing network based on degradation estimation and granularity estimation is trained;

[0097] S6. After training is completed, any low-quality image is input, and the network assigns the optimal expert network to the current low-quality image for processing, and finally obtains a clear image corresponding to the current low-quality image.

[0098] The fine-grained simulated degradation dataset in S1 synthesizes low-quality images of different degradation types by simulating degradation. The degradation types include noise, motion blur, JPEG compression artifacts, rain, snow, fog, and dim light. The fine-grained simulated degradation dataset constructs a quadruple of low-quality image-clear image-fine-grained degraded text-content text for each low-quality image.

[0099] Compared with existing methods, this method provides a more fine-grained degradation representation. The fine-grained simulated degradation dataset takes a clear image y as input and synthesizes the corresponding low-quality image x. The construction process is as follows:

[0100] The noise is synthesized as follows:

[0101] x=y+z

[0102] Where z is the 2The Gaussian noise with variance σ is classified into three levels of degradation: weak noise, moderate noise, and strong noise. The value of σ determines the strength of the noise level. σ<15 is considered weak noise, 15≤σ<35 is considered moderate noise, and 35≤σ<50 is considered strong noise.

[0103] Motion blur is synthesized as follows:

[0104]

[0105] Where k is the blur kernel, and the degradation degree is divided into two gradients: small blur and large blur. The size of the blur kernel determines the degree of motion blur. k < 15 is considered small blur, and k > 15 is considered large blur.

[0106] JPEG compression artifacts are implemented using the open source tool DIFFJPEG. The degradation degree is divided into three gradients: heavy compression, medium compression, and light compression. The image quality level determines the degree of compression. An image quality level greater than 0 and less than 15 is considered heavy compression, an image quality level between 15 and 35 is considered medium compression, and an image quality level greater than 35 is considered light compression.

[0107] Rain mainly considers rain streaks, and the synthesis method of rain streaks is:

[0108] x=y+r

[0109] Where r is the synthetic rain streak, which is achieved by random noise and Gaussian blur. The noise is related to the intensity of the rain, and the blur kernel is related to the direction, length, and width of the rain streak.

[0110] Snow is crafted as follows:

[0111] x=y×(1-m)+m

[0112] Where m is a randomly sampled mask from the public dataset Snow100K. The degradation degree is divided into three gradients, which are defined based on the classification of heavy snow, moderate snow, and light snow in Snow100K.

[0113] The fog is synthesized as follows:

[0114] x=T(y)+a(1-T(y)),T(y)=e -βD(y)

[0115] Where a is the global atmospheric light intensity, T(y) is the transfer matrix, D(y) is the depth map, and β is the atmospheric scattering coefficient;

[0116] The depth map is estimated using the open-source MegaDepth model. The degradation level is divided into two gradients: light fog and dense fog. The atmospheric scattering coefficient largely determines the thickness of the fog. A value of β < 0.1 is considered light fog, and β ≥ 0.1 is considered dense fog.

[0117] The synthesis method of dark light is:

[0118] x=yI(y) γ / I(y)

[0119] Where I(y) is the light intensity map, and γ is the dark light coefficient. The light intensity map is obtained by the deep learning model RetinexFormer. The dark light coefficient determines the intensity of the dark light. The degree of degradation is divided into two gradients: weak dark light and strong dark light. The dark light coefficient determines the intensity of the dark light. 1.5>γ>1 is considered weak dark light, and 3>γ≥1.5 is considered strong dark light.

[0120] According to the above method, a triple of low-quality image, clear image and degraded text is constructed. Then, the open source method BLIP is used to create text information related to the content of each clear image, thereby constructing the training data of the low-quality image, clear image, degraded text and content text quadruple.

[0121] The method for training the degradation feature extractor in S2 is as follows: the visual-language model DACLIP is used as the base model of the degradation feature extractor. Based on the fine-grained simulated degradation dataset constructed in step S1, the degradation feature extractor is trained to produce DACLIP that can simultaneously identify different degradation types and different degradation degrees. During the training process, a network with the same structure as the image encoder in DACLIP is used as the image controller to extract image degradation features. The contrast loss is used to align its features with the degraded text. The trained feature extractor is used to obtain the degradation features of any low-quality image.

[0122] The process of constructing a multi-granularity hierarchical degradation space in S3 is as follows:

[0123] S31. Integrate the low-quality images used in the training data of different restoration tasks, and use the trained degradation feature extractor to obtain the degradation features of all data to form a degradation feature set {e};

[0124] Restoration tasks include denoising, deblurring, compression artifact removal, rain streak removal, fog removal, and dark light enhancement.

[0125] S32. A multi-granularity hierarchical degradation space is constructed through a bottom-up hierarchical clustering method. The multi-granularity hierarchical degradation space is divided into three granularity levels, where the 0th granularity level represents the coarsest granularity and the 2nd granularity level represents the finest granularity.

[0126] The bottom-up hierarchical clustering method is expressed as:

[0127] {u} n ←Kmeans({e})

[0128] where {u} n It is the set of cluster centers obtained after clustering at three granularity levels of n = 0, 1, and 2. First, clustering is performed at the finest granularity level. Then, based on the clustering at the finest granularity level, clustering is continued upward to obtain the set of cluster centers at two coarse granularity levels, and then the set of cluster centers at each granularity level {u}0, {u}1, {u}2 is obtained.

[0129] S33, for the input sample belonging to the nth level, through the sample degradation feature and all cluster centers u of the granularity level ni The minimum distance is used to divide the current sample into the cluster closest to it, i = 1, 2, ..., p, p is the number of cluster centers at the nth granularity level, so that the training data is divided into different subsets based on different granularities, and all subsets are constructed into a multi-granularity hierarchical degradation space.

[0130] The construction process of the multi-granularity hybrid expert restoration model in S4:

[0131] For the dark light enhancement task, the low-light image enhancement network RetinexFormer is selected as the expert model;

[0132] The Transformer model Restormer is selected as the expert model for denoising, deblurring, compression artifact removal, rain streak removal, and dehazing tasks;

[0133] Two expert models are used for each of the dark light enhancement, deblurring, and defogging tasks. Three expert models are used for each of the snow removal, denoising, and compression artifact removal tasks. Considering the diversity of rain streaks, four expert models are used for the rain streak removal task.

[0134] The expert model is trained using L1 loss l1:

[0135]

[0136] in is the network prediction result, and Y is the label; after training, a multi-granularity multi-expert restoration model is constructed.

[0137] To mitigate the misleading effects of image degradation estimation errors on routing, we propose a method that combines degradation estimation with granularity estimation for routing. The S5 training process includes:

[0138] S51. Degradation estimation based on degradation features using two sets of multi-layer perceptrons deg and particle size estimation egran ;

[0139] S52, reusing the routing network based on degradation estimation and granularity estimation and Do routing based on degradation estimation and granularity estimation:

[0140]

[0141] in is a collection of multiple expert networks at the nth granularity level, Based on The kth expert network selected by the routing at the nth granularity level, For multi-granularity hybrid expert model The top-down expert network set of the branch, Based on Routing in Select the network with the best performance;

[0142] First, routing based on degradation estimation The routing is performed at the finest granularity level. The routing selects the expert network with the best performance at the finest granularity level for the current input degraded image, and all the expert networks on the branch where the expert network is located in the multi-granularity hybrid expert model are used as the result of the routing based on degradation estimation. Secondly, the routing based on granularity estimation Further, select an expert network with matching granularity from the expert network set obtained in the previous step.

[0143] S53. The following loss function is used for the constraints of degradation estimation and granularity estimation:

[0144]

[0145] in is the i-th cluster center at the finest granularity level;

[0146] S54. Optimize the degradation estimation network, the routing network based on degradation estimation, the granularity estimation network, and the routing network based on granularity estimation by jointly using the following loss function:

[0147]

[0148] in It is a common load balancing loss in the hybrid expert model, which is used to balance the load of each expert. α and β are two hyperparameters used to determine and The influence degrees of α and β are set to 0.1 and 0.01 respectively.

[0149] The hybrid expert routing network based on degradation estimation and granularity estimation in S6 includes a routing network based on degradation estimation and a routing network based on granularity estimation;

[0150] See also Figure 3 ,The degradation estimation network includes fully connected layers MD1 to MD5;

[0151] MD1 is used to perform the first full connection operation and the first activation operation on the degradation features of the input image;

[0152] MD2 is used to perform a second full connection operation and a second activation operation on the output features of MD1;

[0153] MD3 is used to perform the third full connection operation and the third activation operation on the output features of MD2;

[0154] MD4 is used to perform the fourth full connection operation and the fourth activation operation on the output features of MD3;

[0155] MD5 is used to perform the fifth full connection operation and the fifth activation operation on the output features of MD4;

[0156] The first to fifth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The first to fourth activation operations are all LeakyReLU activation functions, and the fifth activation operation is a one-dimensional BatchNorm operation.

[0157] See also Figure 3 ,The granularity estimation network includes the fully connected layer MG1 to the fully connected layer MG5;

[0158] MG1 is used to perform the sixth full connection operation and the sixth activation operation on the degradation features of the input image;

[0159] MG2 performs the seventh full connection operation and the seventh activation operation on the output features of MG1;

[0160] MG3 performs an eighth full connection operation and an eighth activation operation on the output features of MG2;

[0161] MG4 performs the ninth full connection operation and the ninth activation operation on the output features of MG3;

[0162] MG5 performs the tenth full connection operation and the tenth activation operation on the output features of MG4;

[0163] The sixth to tenth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The sixth to ninth activation operations are all LeakyReLU activation functions, and the tenth activation operation is a one-dimensional BatchNorm operation.

[0164] The routing network based on degradation estimation includes a fully connected layer GD1;

[0165] GD1 is used to perform the eleventh fully connected operation on the degradation estimation of the MD5 output image;

[0166] The routing network based on granularity estimation includes a fully connected layer GG1;

[0167] GG1 is used to perform the twelfth fully connected operation on the granularity estimation of the MG5 output image;

[0168] The input dimension of the eleventh fully connected operation is 512, and the output dimension is 19;

[0169] The input dimension of the twelfth fully connected operation is 512 and the output dimension is 3.

[0170] In summary, the present invention includes a multi-granularity hierarchical degradation space, a multi-granularity expert restoration network, a fine-grained degradation degree-aware degradation feature extractor, a degradation estimation network and a granularity estimation network, and a routing network based on degradation estimation and granularity estimation.

[0171] The method of the present invention can be used to perform multi-task restoration on low-quality images with good results. The results compared with the existing technology can be seen in Figure 2 shown.

[0172] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. A unified multi-task restoration method based on granularity-aware image degradation estimation, characterized by: The method comprises the following steps: S1. Build a fine-grained simulated degradation dataset and assign corresponding content text and degradation text to each low-quality image-high-definition image pair. S2, training a fine-grained degradation feature extractor using the constructed fine-grained simulated degradation dataset; S3. Collect all low-quality images in the training data used for restoration tasks, use the obtained fine-grained degradation feature extractor to obtain a degradation feature set of the low-quality images in the training data, and construct a multi-granularity hierarchical degradation space based on the obtained degradation feature set; S4. Based on the multi-granularity hierarchical degradation space, an expert restoration network is trained for each subspace in each granularity, and all expert networks are combined to form a multi-granularity hybrid expert restoration model; S5. Based on the trained degradation feature extractor and multi-granularity hybrid expert restoration model, a hybrid expert routing network based on degradation estimation and granularity estimation is trained; S6. After training is completed, any low-quality image is input, and the network assigns the optimal expert network to the current low-quality image for processing, and finally obtains a clear image corresponding to the current low-quality image.

2. The unified multi-task restoration method based on granularity-aware image degradation estimation according to claim 1, characterized in that: The fine-grained simulated degradation dataset in S1 synthesizes low-quality images of different degradation types by simulating degradation. The degradation types include noise, motion blur, JPEG compression artifacts, rain, snow, fog, and dim light. The fine-grained simulated degradation dataset constructs a quadruple of low-quality image-clear image-fine-grained degraded text-content text for each low-quality image.

3. The unified multi-task restoration method based on granularity-aware image degradation estimation according to claim 2, characterized in that: The fine-grained simulated degradation dataset takes a clear image y as input and synthesizes the corresponding low-quality image x. The construction process is as follows: The noise is synthesized as follows: x=y+z Where z is the 2 The Gaussian noise with variance σ is classified into three levels: weak noise, moderate noise, and strong noise. σ<15 is considered weak noise, 15≤σ<35 is considered moderate noise, and 35≤σ<50 is considered strong noise. Motion blur is synthesized as follows: Where k is the blur kernel, and the degradation degree is divided into two gradients, namely small blur and large blur. k<15 is considered small blur, and k>15 is considered large blur. JPEG compression artifacts are implemented using the open source tool DIFFJPEG. The degree of degradation is divided into three gradients: heavy compression, medium compression, and light compression. An image quality level greater than 0 and less than 15 is considered heavy compression, an image quality level between 15 and 35 is considered medium compression, and an image quality level greater than 35 is considered light compression. The synthesis of degenerate rain is expressed in rain streaks. The synthesis of rain streaks is as follows: x=y+r Where r is the synthetic rain streak, which is achieved by random noise and Gaussian blur. The noise is related to the intensity of the rain, and the blur kernel is related to the direction, length, and width of the rain streak. Snow is crafted as follows: x=y×(1-m)+m Where m is a randomly sampled mask from the public dataset Snow100K. The degradation degree is divided into three gradients, which are defined based on the classification of heavy snow, moderate snow, and light snow in Snow100K. The fog is synthesized as follows: x=T(y)+a(1-T(y)),T(y)=e -βD(y) Where a is the global atmospheric light intensity, T(y) is the transfer matrix, D(y) is the depth map, and β is the atmospheric scattering coefficient; The depth map is estimated using the open-source model MegaDepth. The degradation level is divided into two gradients: light fog and dense fog. β < 0.1 is considered light fog, and β ≥ 0.1 is considered dense fog. The synthesis method of dark light is: x=yI(y) γ / I(y) Where I(y) is the light intensity map, and γ is the dark light coefficient. The light intensity map is obtained by the deep learning model RetinexFormer. The dark light coefficient determines the intensity of the dark light. The degree of degradation is divided into two gradients: weak dark light and strong dark light. 1.5>γ>1 is considered weak dark light, and 3>γ≥1.5 is considered strong dark light. According to the above method, a triple of low-quality image, clear image and degraded text is constructed. Then, the open source method BLIP is used to create text information related to the content of each clear image, thereby constructing the training data of the low-quality image, clear image, degraded text and content text quadruple.

4. The unified multi-task restoration method based on granularity-aware image degradation estimation according to claim 1, characterized in that: The method for training the degradation feature extractor in S2 is as follows: the visual-language model DACLIP is used as the base model of the degradation feature extractor. Based on the fine-grained simulated degradation dataset constructed in step S1, the degradation feature extractor is trained to produce DACLIP that can simultaneously identify different degradation types and different degradation degrees. During the training process, a network with the same structure as the image encoder in DACLIP is used as the image controller to extract image degradation features. The contrast loss is used to align its features with the degraded text. The trained feature extractor is used to obtain the degradation features of any low-quality image.

5. The unified multi-task restoration method based on granularity-aware image degradation estimation according to claim 1, characterized in that: The process of constructing a multi-granularity hierarchical degradation space in S3 is as follows: S31. Integrate the low-quality images used in the training data of different restoration tasks, and use the trained degradation feature extractor to obtain the degradation features of all data to form a degradation feature set {e}; S32. A multi-granularity hierarchical degradation space is constructed through a bottom-up hierarchical clustering method. The multi-granularity hierarchical degradation space is divided into three granularity levels, where the 0th granularity level represents the coarsest granularity and the 2nd granularity level represents the finest granularity. The bottom-up hierarchical clustering method is expressed as: {the} n ←Kmeans({e}) where {u} n It is the set of cluster centers obtained after clustering at three granularity levels of n = 0, 1, and 2. First, clustering is performed at the finest granularity level. Then, based on the clustering at the finest granularity level, clustering is continued upward to obtain the set of cluster centers at two coarse granularity levels, and then the set of cluster centers at each granularity level {u}0, {u}1, {u}2 is obtained. S33, for the input sample belonging to the nth level, through the sample degradation feature and all cluster centers u of the granularity level ni The minimum distance is used to divide the current sample into the cluster closest to it, i = 1, 2, ..., p, p is the number of cluster centers at the nth granularity level, so that the training data is divided into different subsets based on different granularities, and all subsets are constructed into a multi-granularity hierarchical degradation space.

6. The unified multi-task restoration method based on granularity-aware image degradation estimation according to claim 2 or 5, characterized in that: Restoration tasks include denoising, deblurring, compression artifact removal, rain streak removal, fog removal, and dark light enhancement.

7. The unified multi-task restoration method based on granularity-aware image degradation estimation according to claim 6, characterized in that: The construction process of the multi-granularity hybrid expert restoration model in S4: For the dark light enhancement task, the low-light image enhancement network RetinexFormer is selected as the expert model; The Transformer model Restormer is selected as the expert model for denoising, deblurring, compression artifact removal, rain streak removal, and dehazing tasks; Dark light enhancement, deblurring, and defogging tasks are each processed using two expert models; snow removal, denoising, and compression artifact removal tasks are each processed using three expert models, and rain streak removal tasks are processed using four expert models; The expert model is trained using L1 loss l1: in is the network prediction result, Y is the label; After training, a multi-granularity and multi-expert restoration model is formed.

8. The unified multi-task restoration method based on granularity-aware image degradation estimation according to claim 7, characterized in that: The S5 training process includes: S51. Degradation estimation based on degradation features using two sets of multi-layer perceptrons deg and particle size estimation e gran ; S52, reusing the routing network based on degradation estimation and granularity estimation and Do routing based on degradation estimation and granularity estimation: in is a collection of multiple expert networks at the nth granularity level, Based on The kth expert network selected by the routing at the nth granularity level, For multi-granularity hybrid expert model The top-down expert network set of the branch, Based on Routing in Select the network with the best performance; S53. The following loss function is used for the constraints of degradation estimation and granularity estimation: in is the i-th cluster center at the finest granularity level; S54. Optimize the degradation estimation network, the routing network based on degradation estimation, the granularity estimation network, and the routing network based on granularity estimation by jointly using the following loss function: in It is a common load balancing loss in the hybrid expert model, which is used to balance the load of each expert. α and β are two hyperparameters used to determine and The influence degrees of α and β are set to 0.1 and 0.01 respectively.

9. The unified multi-task restoration method based on granularity-aware image degradation estimation according to claim 1, characterized in that: The hybrid expert routing network based on degradation estimation and granularity estimation in S6 includes a routing network based on degradation estimation and a routing network based on granularity estimation; The degradation estimation network includes fully connected layers MD1 to MD5; MD1 is used to perform the first full connection operation and the first activation operation on the degradation features of the input image; MD2 is used to perform a second full connection operation and a second activation operation on the output features of MD1; MD3 is used to perform the third full connection operation and the third activation operation on the output features of MD2; MD4 is used to perform the fourth full connection operation and the fourth activation operation on the output features of MD3; MD5 is used to perform the fifth full connection operation and the fifth activation operation on the output features of MD4; The first to fifth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The first to fourth activation operations are all LeakyReLU activation functions, and the fifth activation operation is a one-dimensional BatchNorm operation. The granularity estimation network includes fully connected layers MG1 to MG5; MG1 is used to perform the sixth full connection operation and the sixth activation operation on the degradation features of the input image; MG2 performs the seventh full connection operation and the seventh activation operation on the output features of MG1; MG3 performs an eighth full connection operation and an eighth activation operation on the output features of MG2; MG4 performs the ninth full connection operation and the ninth activation operation on the output features of MG3; MG5 performs the tenth full connection operation and the tenth activation operation on the output features of MG4; The sixth to tenth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The sixth to ninth activation operations are all LeakyReLU activation functions, and the tenth activation operation is a one-dimensional BatchNorm operation. The routing network based on degradation estimation includes a fully connected layer GD1; GD1 is used to perform the eleventh fully connected operation on the degradation estimation of the MD5 output image; The routing network based on granularity estimation includes a fully connected layer GG1; GG1 is used to perform the twelfth fully connected operation on the granularity estimation of the MG5 output image; The input dimension of the eleventh fully connected operation is 512, and the output dimension is 19; The input dimension of the twelfth fully connected operation is 512 and the output dimension is 3.

Images