Multi-restoration task unification method based on granularity perception image degradation estimation

Abstract

The invention discloses a multi-restoration task unification method based on granularity perception image degradation estimation, belongs to the field of underlying visual image restoration, and aims at solving the problem that a condition guidance module based on degradation estimation used in an existing multi-task method is inaccurate. The method comprises the following steps: S1, constructing a fine-grained simulation degradation data set; s2, training a fine-grained degradation feature extractor; s3, constructing a multi-granularity hierarchical degradation space; s4, constructing a multi-granularity hybrid expert restoration model; s5, based on the trained degradation feature extractor and the multi-granularity hybrid expert restoration model, training a hybrid expert routing network based on degradation estimation and granularity estimation; and S6, after training is completed, inputting any low-quality image, distributing an optimal expert network for the current low-quality image by the network for processing, and finally obtaining a clear image corresponding to the current low-quality image.

Description

Technical Field

[0001] The present invention relates to a unified method for multiple restoration tasks based on granularity-aware image degradation estimation, and belongs to the field of underlying visual image restoration. Background Art

[0002] The underlying vision aims to improve user satisfaction and specific needs by improving the visual quality of images and videos, and can provide an important data foundation for other visual understanding tasks. There are many factors that cause visual quality degradation in the imaging process, and the visual quality requirements of users and tasks are quite diverse. Therefore, the underlying vision often involves solving basic tasks such as deblurring, denoising, removing JPEG compression artifacts, removing rain streaks, removing fog, removing snow, dark light enhancement, and hue transformation, as well as their combinations. It also presents different characteristics with the changes in imaging devices and imaging methods, and has always been an important and challenging basic problem in computer vision. Early methods usually trained a dedicated model for each task separately. The diversity of tasks caused a relatively large overhead in time and storage. In recent studies, how to unify multiple restoration tasks under a single framework has become a research hotspot.

[0003] Existing unified solutions for multiple restoration tasks can be divided into methods based on general models (task-agnostic), methods based on prompt learning (prompt-learning) or efficient adapters (adapter), and methods based on hybrid expert networks. Due to the different degradation processes and methods, there is a certain conflict in learning how to restore different types of degradation. This conflict increases the difficulty of multi-task unification, making it difficult for the method to ensure generalization to multiple tasks while ensuring good performance in each task, resulting in serious performance loss when directly using a general model to handle multiple tasks at the same time. Subsequent methods introduce prompt learning and adapters, so that the base model network can modulate the current base model through a guided method based on degradation features or deep features, thereby achieving conditional processing of different tasks, alleviating the conflict problem between different tasks to a certain extent, and improving the performance of multi-task unified methods. In addition, the method based on hybrid expert networks further considers the sharing (cooperation) and specificity (specification) of different tasks. Based on the shared base model, these methods train expert filters, expert low-rank adaptation weights, expert networks, etc. for each task separately. During training and inference, the method routes according to the degenerate features or deep features of the current input, and selects a dedicated module to be combined with the base model for processing. Although explicitly processing different tasks can further solve the conflict between multiple tasks, these methods rely more on accurate feature expression. Incorrect feature expression will cause incorrect routing, which will have a negative impact on the restoration process. Summary of the invention

[0004] Aiming at the problem that the conditional guidance module based on degradation estimation used in the existing multi-task method is inaccurate, the present invention provides a unified method for multiple restoration tasks based on granularity-aware image degradation estimation.

[0005] The unified method for multiple restoration tasks based on granularity-aware image degradation estimation of the present invention comprises the following steps:

[0006] S1. Build a fine-grained simulated degradation dataset and assign corresponding content text and degradation text to each low-quality image-high-definition image pair;

[0007] S2, training a fine-grained degradation feature extractor through the constructed fine-grained simulation degradation dataset;

[0008] S3, collect all low-quality images in the training data used in the restoration tasks, use the obtained fine-grained degradation feature extractor to obtain the degradation feature set of the low-quality images in the training data, and construct a multi-granularity hierarchical degradation space based on the obtained degradation feature set;

[0009] S4. Based on the multi-granularity hierarchical degradation space, an expert restoration network is trained for each subspace in each granularity, and all expert networks are combined to form a multi-granularity hybrid expert restoration model;

[0010] S5. Based on the trained degradation feature extractor and the multi-granularity hybrid expert restoration model, a hybrid expert routing network based on degradation estimation and granularity estimation is trained;

[0011] S6. After training is completed, any low-quality image is input, and the network assigns the optimal expert network to the current low-quality image for processing, and finally obtains a clear image corresponding to the current low-quality image.

[0012] Preferably, the fine-grained simulated degradation dataset in S1 synthesizes low-quality images of different degradation types by simulating degradation, and the degradation types include noise, motion blur, JPEG compression artifacts, rain, snow, fog and dim light. The fine-grained simulated degradation dataset constructs a quadruple of low-quality image-clear image-fine-grained degraded text-content text for each low-quality image.

[0013] Preferably, the fine-grained simulated degradation dataset takes the clear image y as input and synthesizes the corresponding low-quality image x. The construction process is:

[0014] The noise is synthesized as follows:

[0015] x=y+z

[0016] Where z is the 2is the Gaussian noise with variance σ, and the degradation degree is divided into three gradients, namely weak noise, moderate noise and strong noise; σ<15 is considered as weak noise, 15≤σ<35 is considered as moderate noise, and 35≤σ<50 is considered as strong noise;

[0017] Motion blur is synthesized as follows:

[0018]

[0019] Where k is the blur kernel, and the degradation degree is divided into two gradients, namely small blur and large blur. k<15 is considered small blur, and k>15 is considered large blur.

[0020] JPEG compression artifacts are implemented using the open source tool DIFFJPEG. The degradation levels are divided into three gradients: heavy compression, medium compression, and light compression. Image quality levels greater than 0 and less than 15 are considered heavy compression, image quality levels between 15 and 35 are considered medium compression, and image quality levels greater than 35 are considered light compression.

[0021] The synthesis of degenerate rain is expressed in rain streaks. The synthesis of rain streaks is as follows:

[0022] x=y+r

[0023] Where r is the synthetic rain streak, which is achieved by random noise and Gaussian blur. The noise is related to the intensity of the rain, and the blur kernel is related to the direction, length, and width of the rain streak.

[0024] Snow is synthesized as follows:

[0025] x=y×(1-m)+m

[0026] Where m is a randomly sampled mask image from the public dataset Snow100K. The degradation degree is divided into three gradients, which are defined according to the division of heavy snow, moderate snow, and light snow in Snow100K.

[0027] The synthesis method of fog is:

[0028] x=T(y)+a(1-T(y)),T(y)=e -βD(y)

[0029] Where a is the global atmospheric intensity, T(y) is the transfer matrix, D(y) is the depth map, and β is the atmospheric scattering coefficient;

[0030] The depth map estimation method uses the open source model MegaDepth. The degradation degree is divided into two gradients, namely light fog and dense fog. β<0.1 is considered light fog, and β≥0.1 is considered dense fog.

[0031] The synthesis method of dark light is:

[0032] x=yI(y) γ / I(y)

[0033] Where I(y) is the light intensity map, and γ is the dark light coefficient. The light intensity map is obtained by the deep learning model RetinexFormer. The dark light coefficient determines the intensity of the dark light. The degree of degradation is divided into two gradients, namely weak dark light and strong dark light. 1.5>γ>1 is considered as weak dark light, and 3>γ≥1.5 is considered as strong dark light.

[0034] According to the above method, a triple of low-quality image-clear image-degraded text is constructed, and then the open source method BLIP is used to create text information related to the content of each clear image, thereby constructing the low-quality image-clear image-degraded text-content text quadruple training data.

[0035] Preferably, the method of training the degradation feature extractor in S2 is as follows: the base model of the degradation feature extractor is the visual-language model DACLIP, based on the fine-grained simulated degradation data set constructed in step S1, the degradation feature extractor is trained to obtain a DACLIP that can simultaneously identify different degradation types and different degradation degrees, during the training process, a network with the same structure as the image encoder in DACLIP is used as an image controller to extract image degradation features, and contrast loss is used to align its features with the degraded text, and the trained feature extractor is used to obtain degradation features of any low-quality image.

[0036] Preferably, the process of constructing a multi-granularity hierarchical degradation space in S3 is:

[0037] S31, integrating the low-quality images used in the training data of different restoration tasks, and using the trained degradation feature extractor to obtain the degradation features of all data to form a degradation feature set {e};

[0038] S32. A multi-granularity hierarchical degradation space is constructed by a bottom-up hierarchical clustering method. The multi-granularity hierarchical degradation space is divided into three granularity levels, where the 0th granularity level represents the coarsest granularity and the 2nd granularity level represents the finest granularity.

[0039] The bottom-up hierarchical clustering method is expressed as:

[0040] {u} n ←Kmeans({e})

[0041] where {u} n It is the set of cluster centers obtained after clustering at three granularity levels of n = 0, 1, and 2. First, clustering is performed at the finest granularity level. Then, based on the clustering at the finest granularity level, clustering is continued upward to obtain the set of cluster centers at two coarse granularity levels, and then the set of cluster centers at each granularity level {u} is obtained. 0 ,{u} 1,{u} 2 ;

[0042] S33, for the input sample belonging to the nth level, the sample degradation feature is compared with all the cluster centers u of the granularity level ni The minimum distance is used to divide the current sample into the cluster closest to it, i = 1, 2, ..., p, p is the number of cluster centers at the nth granularity level, so that the training data is divided into different subsets based on different granularities, and all subsets are constructed as a multi-granularity hierarchical degradation space.

[0043] Preferably, the restoration task includes a denoising task, a deblurring task, a compression artifact removal task, a rain streak removal task, a fog removal task and a dark light enhancement task.

[0044] Preferably, the construction process of the multi-granularity hybrid expert restoration model in S4 is:

[0045] For the dark light enhancement task, we select the low-light image enhancement network RetinexFormer as the expert model;

[0046] The Transformer model Restormer is selected as the expert model for denoising, deblurring, compression artifact removal, rain streak removal, and dehazing tasks;

[0047] Dark light enhancement, deblurring, and defogging tasks are each processed using two expert models; snow removal, denoising, and compression artifact removal tasks are each processed using three expert models, and rain streak removal tasks are processed using four expert models;

[0048] The expert model is trained using L1 loss 1 conduct:

[0049]

[0050] in is the network prediction result, and Y is the label; after training, a multi-granularity and multi-expert restoration model is constructed.

[0051] Preferably, the training process of S5 includes:

[0052] S51. Using two sets of multi-layer perceptrons to perform degradation estimation based on degradation features deg and particle size estimation gtan ;

[0053] S52: Reusing the routing network based on degradation estimation and granularity estimation and Do routing based on degradation estimation and granularity estimation:

[0054]

[0055] in is a collection of multiple expert networks at the nth granularity level, Based on The kth expert network selected by the router at the nth granularity level, For multi-granularity hybrid expert models The top-down expert network set of the branch, Based on Routing in Select the network with the best performance;

[0056] S53. The following loss function is used for the constraints of degradation estimation and granularity estimation:

[0057]

[0058] in is the i-th cluster center at the finest granularity level;

[0059] S54, jointly using the following loss function to optimize the degradation estimation network, the routing network based on the degradation estimation, the granularity estimation network, and the routing network based on the granularity estimation:

[0060]

[0061] in It is a common load balancing loss in the hybrid expert model, which is used to balance the load of each expert. α and β are two hyperparameters used to determine and The influence degrees of ,are set to 0.1 and 0.01 respectively.

[0062] Preferably, the hybrid expert routing network based on degradation estimation and granularity estimation in S6 includes a routing network based on degradation estimation and a routing network based on granularity estimation;

[0063] The degradation estimation network includes fully connected layers MD1 to MD5;

[0064] MD1 is used to perform a first full connection operation and a first activation operation on the degradation features of the input image;

[0065] MD2 is used to perform a second full connection operation and a second activation operation on the output features of MD1;

[0066] MD3 is used to perform a third full connection operation and a third activation operation on the output features of MD2;

[0067] MD4 is used to perform a fourth full connection operation and a fourth activation operation on the output features of MD3;

[0068] MD5 is used to perform the fifth full connection operation and the fifth activation operation on the output features of MD4;

[0069] The first to fifth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The first to fourth activation operations are all LeakyReLU activation functions, and the fifth activation operation is a one-dimensional BatchNorm operation.

[0070] The granularity estimation network includes fully connected layers MG1 to MG5;

[0071] MG1 is used to perform a sixth full connection operation and a sixth activation operation on the degradation features of the input image;

[0072] MG2 performs the seventh full connection operation and the seventh activation operation on the output features of MG1;

[0073] MG3 performs an eighth full connection operation and an eighth activation operation on the output features of MG2;

[0074] MG4 performs the ninth full connection operation and the ninth activation operation on the output features of MG3;

[0075] MG5 performs the tenth full connection operation and the tenth activation operation on the output features of MG4;

[0076] The sixth to tenth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The sixth to ninth activation operations are all LeakyReLU activation functions, and the tenth activation operation is a one-dimensional BatchNorm operation.

[0077] The routing network based on degradation estimation includes a fully connected layer GD1;

[0078] GD1 is used to perform the eleventh fully connected operation on the MD5 output image degradation estimation;

[0079] The routing network based on granular estimation includes a fully connected layer GG1;

[0080] GG1 is used to perform the twelfth fully connected operation on the granularity estimation of the MG5 output image;

[0081] The input dimension of the eleventh fully connected operation is 512, and the output dimension is 19;

[0082] The input dimension of the twelfth fully connected operation is 512 and the output dimension is 3.

[0083] Beneficial effects of the present invention: The unified method of multiple restoration tasks based on granularity-aware degradation estimation described in the present invention, in order to mitigate the misleading caused by image degradation estimation errors, introduces the granularity factor in image degradation estimation for the first time to express the magnitude of the current image degradation estimation error. The present invention is the first to conduct research at a granularity finer than the level of degradation type, and proposes a multi-granularity hierarchical degradation space, and constructs a multi-granularity hybrid expert restoration model. Further, in the process of multi-expert routing, the present invention proposes routing based on image degradation estimation and granularity estimation, which can find a more robust and accurate degradation expression for any input low-quality image, thereby finding a suitable expert restoration model for processing. Compared with previous methods, the present invention has achieved a huge improvement in multi-task unified performance, greatly narrowing the gap with single-task proprietary models.

[0084] Although the present invention is described herein with reference to specific implementation methods, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the features described herein may be combined in different dependent claims and in a manner other than that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 is a flow chart of a unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to the present invention;

[0086] Figure 2 It is a schematic diagram of the comparison results between the method proposed in the present invention and other methods on multiple restoration tasks.

[0087] Figure 3 It is the network structure diagram of degradation estimation and granularity estimation. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0090] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0091] Specific implementation method 1: The following is combined Figure 1 and Figure 2 The present embodiment describes a unified method for multiple restoration tasks based on granularity-aware image degradation estimation, see Figure 1 As shown, the method comprises the following steps:

[0092] S1. Build a fine-grained simulated degradation dataset and assign corresponding content text and degradation text to each low-quality image-high-definition image pair;

[0093] S2, training a fine-grained degradation feature extractor through the constructed fine-grained simulation degradation dataset;

[0094] S3, collect all low-quality images in the training data used in the restoration tasks, use the obtained fine-grained degradation feature extractor to obtain the degradation feature set of the low-quality images in the training data, and construct a multi-granularity hierarchical degradation space based on the obtained degradation feature set;

[0095] S4. Based on the multi-granularity hierarchical degradation space, an expert restoration network is trained for each subspace in each granularity, and all expert networks are combined to form a multi-granularity hybrid expert restoration model;

[0096] S5. Based on the trained degradation feature extractor and the multi-granularity hybrid expert restoration model, a hybrid expert routing network based on degradation estimation and granularity estimation is trained;

[0097] S6. After training is completed, any low-quality image is input, and the network assigns the optimal expert network to the current low-quality image for processing, and finally obtains a clear image corresponding to the current low-quality image.

[0098] The fine-grained simulated degradation dataset in S1 synthesizes low-quality images of different degradation types by simulating degradation. The degradation types include noise, motion blur, JPEG compression artifacts, rain, snow, fog, and dim light. The fine-grained simulated degradation dataset constructs a quadruple of low-quality image-clear image-fine-grained degraded text-content text for each low-quality image.

[0099] Compared with the existing methods, a more fine-grained degradation expression is performed. The fine-grained simulated degradation dataset takes the clear image y as input and synthesizes the corresponding low-quality image x. The construction process is:

[0100] The noise is synthesized as follows:

[0101] x=y+z

[0102] Where z is the 2is the Gaussian noise with variance σ, and the degradation degree is divided into three gradients, namely weak noise, moderate noise and strong noise; the size of σ determines the strength of the noise level, σ<15 is considered as weak noise, 15≤σ<35 is considered as moderate noise, and 35≤σ<50 is considered as strong noise;

[0103] Motion blur is synthesized as follows:

[0104]

[0105] Where k is the blur kernel, and the degradation degree is divided into two gradients, small blur and large blur. The size of the blur kernel determines the degree of motion blur. k<15 is considered small blur, and k>15 is considered large blur.

[0106] JPEG compression artifacts are implemented using the open source tool DIFFJPEG. The degradation degree is divided into three gradients: heavy compression, medium compression, and light compression. The image quality level determines the degree of compression. An image quality level greater than 0 and less than 15 is considered heavy compression, an image quality level between 15 and 35 is considered medium compression, and an image quality level greater than 35 is considered light compression.

[0107] Rain mainly considers rain streaks, and the synthesis method of rain streaks is:

[0108] x=y+r

[0109] Where r is the synthetic rain streak, which is achieved by random noise and Gaussian blur. The noise is related to the intensity of the rain, and the blur kernel is related to the direction, length, and width of the rain streak.

[0110] Snow is synthesized as follows:

[0111] x=y×(1-m)+m

[0112] Where m is a randomly sampled mask image from the public dataset Snow100K. The degradation degree is divided into three gradients, which are defined according to the division of heavy snow, moderate snow, and light snow in Snow100K.

[0113] The synthesis method of fog is:

[0114] x=T(y)+a(1-T(y)),T(y)=e -βD(y)

[0115] Where a is the global atmospheric intensity, T(y) is the transfer matrix, D(y) is the depth map, and β is the atmospheric scattering coefficient;

[0116] The depth map estimation method uses the open source model MegaDepth. The degradation degree is divided into two gradients, namely light fog and dense fog. The atmospheric scattering coefficient largely determines the thickness of the fog. β<0.1 is considered light fog, and β≥0.1 is considered dense fog.

[0117] The synthesis method of dark light is:

[0118] x=yI(y) γ / I(y)

[0119] Where I(y) is the light intensity map, and γ is the dark light coefficient. The light intensity map is obtained by the deep learning model RetinexFormer. The dark light coefficient determines the intensity of the dark light. The degradation degree is divided into two gradients, namely weak dark light and strong dark light. The dark light coefficient determines the intensity of the dark light. 1.5>γ>1 is considered as weak dark light, and 3>γ≥1.5 is considered as strong dark light.

[0120] According to the above method, a triple of low-quality image-clear image-degraded text is constructed, and then the open source method BLIP is used to create text information related to the content of each clear image, thereby constructing the low-quality image-clear image-degraded text-content text quadruple training data.

[0121] The method of training the degradation feature extractor in S2 is as follows: the base model of the degradation feature extractor is the visual-language model DACLIP. Based on the fine-grained simulated degradation dataset constructed in step S1, the degradation feature extractor is trained to produce DACLIP that can simultaneously identify different degradation types and different degrees of degradation. During the training process, a network with the same structure as the image encoder in DACLIP is used as an image controller to extract image degradation features, and contrast loss is used to align its features with the degraded text. The trained feature extractor is used to obtain the degradation features of any low-quality image.

[0122] The process of constructing a multi-granularity hierarchical degradation space in S3 is as follows:

[0123] S31, integrating the low-quality images used in the training data of different restoration tasks, and using the trained degradation feature extractor to obtain the degradation features of all data to form a degradation feature set {e};

[0124] The restoration tasks include denoising, deblurring, compression artifact removal, rain streak removal, haze removal and dark light enhancement.

[0125] S32. A multi-granularity hierarchical degradation space is constructed by a bottom-up hierarchical clustering method. The multi-granularity hierarchical degradation space is divided into three granularity levels, where the 0th granularity level represents the coarsest granularity and the 2nd granularity level represents the finest granularity.

[0126] The bottom-up hierarchical clustering method is expressed as:

[0127] {u} n ←Kmeans({e})

[0128] where {u} n It is the set of cluster centers obtained after clustering at three granularity levels of n = 0, 1, and 2. First, clustering is performed at the finest granularity level. Then, based on the clustering at the finest granularity level, clustering is continued upward to obtain the set of cluster centers at two coarse granularity levels, and then the set of cluster centers at each granularity level {u} is obtained. 0 ,{u} 1 ,{u} 2 ;

[0129] S33, for the input sample belonging to the nth level, the sample degradation feature is compared with all the cluster centers u of the granularity level ni The minimum distance is used to divide the current sample into the cluster closest to it, i = 1, 2, ..., p, p is the number of cluster centers at the nth granularity level, so that the training data is divided into different subsets based on different granularities, and all subsets are constructed as a multi-granularity hierarchical degradation space.

[0130] The construction process of the multi-granularity hybrid expert restoration model in S4:

[0131] For the dark light enhancement task, we select the low-light image enhancement network RetinexFormer as the expert model;

[0132] The Transformer model Restormer is selected as the expert model for denoising, deblurring, compression artifact removal, rain streak removal, and dehazing tasks;

[0133] Dark light enhancement, deblurring, and defogging tasks are each processed using two expert models; snow removal, denoising, and compression artifact removal tasks are each processed using three expert models. Considering the diversity of rain streaks, the rain streak removal task is processed using four expert models.

[0134] The expert model is trained using L1 loss 1 conduct:

[0135]

[0136] in is the network prediction result, and Y is the label; after training, a multi-granularity and multi-expert restoration model is constructed.

[0137] In order to mitigate the misleading caused by image degradation estimation errors in the routing process, a method combining degradation estimation and granularity estimation is proposed for routing. The training process of S5 includes:

[0138] S51. Using two sets of multi-layer perceptrons to perform degradation estimation based on degradation features deg and particle size estimation gran ;

[0139] S52: Reusing the routing network based on degradation estimation and granularity estimation and Do routing based on degradation estimation and granularity estimation:

[0140]

[0141] in is a collection of multiple expert networks at the nth granularity level, Based on The kth expert network selected by the router at the nth granularity level, For multi-granularity hybrid expert models The top-down expert network set of the branch, Based on Routing in Select the network with the best performance;

[0142] First, routing based on degradation estimation The routing is performed at the finest granularity level. The routing selects the best expert network at the finest granularity level for the current input degraded image, and all the expert networks on the branch where the expert network is located in the multi-granularity hybrid expert model are used as the result of the degradation estimation-based routing. Further, select an expert network with matching granularity from the expert network set obtained in the previous step.

[0143] S53. The following loss function is used for the constraints of degradation estimation and granularity estimation:

[0144]

[0145] in is the i-th cluster center at the finest granularity level;

[0146] S54, jointly using the following loss function to optimize the degradation estimation network, the routing network based on the degradation estimation, the granularity estimation network, and the routing network based on the granularity estimation:

[0147]

[0148] in It is a common load balancing loss in the hybrid expert model, which is used to balance the load of each expert. α and β are two hyperparameters used to determine and The influence degrees of ,are set to 0.1 and 0.01 respectively.

[0149] The hybrid expert routing network based on degradation estimation and granularity estimation in S6 includes a routing network based on degradation estimation and a routing network based on granularity estimation;

[0150] See also Figure 3 ,The degradation estimation network includes fully connected layers MD1 to MD5;

[0151] MD1 is used to perform a first full connection operation and a first activation operation on the degradation features of the input image;

[0152] MD2 is used to perform a second full connection operation and a second activation operation on the output features of MD1;

[0153] MD3 is used to perform a third full connection operation and a third activation operation on the output features of MD2;

[0154] MD4 is used to perform a fourth full connection operation and a fourth activation operation on the output features of MD3;

[0155] MD5 is used to perform the fifth full connection operation and the fifth activation operation on the output features of MD4;

[0156] The first to fifth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The first to fourth activation operations are all LeakyReLU activation functions, and the fifth activation operation is a one-dimensional BatchNorm operation.

[0157] See also Figure 3 ,The granularity estimation network includes fully connected layers MG1 to MG5;

[0158] MG1 is used to perform a sixth full connection operation and a sixth activation operation on the degradation features of the input image;

[0159] MG2 performs the seventh full connection operation and the seventh activation operation on the output features of MG1;

[0160] MG3 performs an eighth full connection operation and an eighth activation operation on the output features of MG2;

[0161] MG4 performs the ninth full connection operation and the ninth activation operation on the output features of MG3;

[0162] MG5 performs the tenth full connection operation and the tenth activation operation on the output features of MG4;

[0163] The sixth to tenth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The sixth to ninth activation operations are all LeakyReLU activation functions, and the tenth activation operation is a one-dimensional BatchNorm operation.

[0164] The routing network based on degradation estimation includes a fully connected layer GD1;

[0165] GD1 is used to perform the eleventh fully connected operation on the MD5 output image degradation estimation;

[0166] The routing network based on granular estimation includes a fully connected layer GG1;

[0167] GG1 is used to perform the twelfth fully connected operation on the granularity estimation of the MG5 output image;

[0168] The input dimension of the eleventh fully connected operation is 512, and the output dimension is 19;

[0169] The input dimension of the twelfth fully connected operation is 512 and the output dimension is 3.

[0170] In summary, the present invention includes a multi-granularity hierarchical degradation space, a multi-granularity expert restoration network, a fine-grained degradation degree-aware degradation feature extractor, a degradation estimation network and a granularity estimation network, and a routing network based on degradation estimation and granularity estimation.

[0171] The method of the present invention can be used to perform multi-task restoration on low-quality images with good results. The results compared with the prior art are shown in Figure 2 shown.

[0172] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A unified approach for multiple restoration tasks based on granularity-aware image degradation estimation, characterized in that: The method comprises the following steps: S1. Build a fine-grained simulated degradation dataset and assign corresponding content text and degradation text to each low-quality image-high-definition image pair; S2, training a fine-grained degradation feature extractor through the constructed fine-grained simulation degradation dataset; S3, collect all low-quality images in the training data used in the restoration tasks, use the obtained fine-grained degradation feature extractor to obtain the degradation feature set of the low-quality images in the training data, and construct a multi-granularity hierarchical degradation space based on the obtained degradation feature set; S4. Based on the multi-granularity hierarchical degradation space, an expert restoration network is trained for each subspace in each granularity, and all expert networks are combined to form a multi-granularity hybrid expert restoration model; S5. Based on the trained degradation feature extractor and the multi-granularity hybrid expert restoration model, a hybrid expert routing network based on degradation estimation and granularity estimation is trained; S6. After training is completed, any low-quality image is input, and the network assigns the optimal expert network to the current low-quality image for processing, and finally obtains a clear image corresponding to the current low-quality image.

2. The unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to claim 1, characterized in that: The fine-grained simulated degradation dataset in S1 synthesizes low-quality images of different degradation types by simulating degradation. The degradation types include noise, motion blur, JPEG compression artifacts, rain, snow, fog, and dim light. The fine-grained simulated degradation dataset constructs a quadruple of low-quality image-clear image-fine-grained degraded text-content text for each low-quality image.

3. The unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to claim 2, characterized in that: The fine-grained simulated degradation dataset takes the clear image y as input and synthesizes the corresponding low-quality image x. The construction process is: The noise is synthesized as follows: x=y+z Where z is the 2 is the Gaussian noise with variance σ, and the degradation degree is divided into three gradients, namely weak noise, moderate noise and strong noise; σ<15 is considered as weak noise, 15≤σ<35 is considered as moderate noise, and 35≤σ<50 is considered as strong noise; Motion blur is synthesized as follows: Where k is the blur kernel, and the degradation degree is divided into two gradients, namely small blur and large blur. k<15 is considered small blur, and k>15 is considered large blur. JPEG compression artifacts are implemented using the open source tool DIFFJPEG. The degradation levels are divided into three gradients: heavy compression, medium compression, and light compression. Image quality levels greater than 0 and less than 15 are considered heavy compression, image quality levels between 15 and 35 are considered medium compression, and image quality levels greater than 35 are considered light compression. The synthesis of degenerate rain is expressed in rain streaks. The synthesis of rain streaks is as follows: x=y+r Where r is the synthetic rain streak, which is achieved by random noise and Gaussian blur. The noise is related to the intensity of the rain, and the blur kernel is related to the direction, length, and width of the rain streak. Snow is synthesized as follows: x=y×(1-m)+m Where m is a randomly sampled mask image from the public dataset Snow100K. The degradation degree is divided into three gradients, which are defined according to the division of heavy snow, moderate snow, and light snow in Snow100K. The synthesis method of fog is: x=T(y)+a(1-T(y)),T(y)=e -βD(y) Where a is the global atmospheric intensity, T(y) is the transfer matrix, D(y) is the depth map, and β is the atmospheric scattering coefficient; The depth map estimation method uses the open source model MegaDepth. The degradation degree is divided into two gradients, namely light fog and dense fog. β<0.1 is considered light fog, and β≥0.1 is considered dense fog. The synthesis method of dark light is: x=yI(y) γ / I(y) Where I(y) is the light intensity map, and γ is the dark light coefficient. The light intensity map is obtained by the deep learning model RetinexFormer. The dark light coefficient determines the intensity of the dark light. The degree of degradation is divided into two gradients, namely weak dark light and strong dark light. 1.5>γ>1 is considered as weak dark light, and 3>γ≥1.5 is considered as strong dark light. According to the above method, a triple of low-quality image-clear image-degraded text is constructed, and then the open source method BLIP is used to create text information related to the content of each clear image, thereby constructing the low-quality image-clear image-degraded text-content text quadruple training data.

4. The unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to claim 1, characterized in that: The method of training the degradation feature extractor in S2 is as follows: the base model of the degradation feature extractor is the visual-language model DACLIP. Based on the fine-grained simulated degradation dataset constructed in step S1, the degradation feature extractor is trained to produce DACLIP that can simultaneously identify different degradation types and different degrees of degradation. During the training process, a network with the same structure as the image encoder in DACLIP is used as an image controller to extract image degradation features, and contrast loss is used to align its features with the degraded text. The trained feature extractor is used to obtain the degradation features of any low-quality image.

5. The unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to claim 1, characterized in that: The process of constructing a multi-granularity hierarchical degradation space in S3 is as follows: S31, integrating the low-quality images used in the training data of different restoration tasks, and using the trained degradation feature extractor to obtain the degradation features of all data to form a degradation feature set {e}; S32. A multi-granularity hierarchical degradation space is constructed by a bottom-up hierarchical clustering method. The multi-granularity hierarchical degradation space is divided into three granularity levels, where the 0th granularity level represents the coarsest granularity and the 2nd granularity level represents the finest granularity. The bottom-up hierarchical clustering method is expressed as: {the} n ←Kmeans({e}) where {u} n It is the set of cluster centers obtained after clustering at three granularity levels of n = 0, 1, and 2. First, clustering is performed at the finest granularity level. Then, based on the clustering at the finest granularity level, clustering is continued upward to obtain the set of cluster centers at two coarse granularity levels, and then the set of cluster centers at each granularity level {u}0, {u}1, {u}2 is obtained. S33, for the input sample belonging to the nth level, the sample degradation feature is compared with all the cluster centers u of the granularity level ni The minimum distance is used to divide the current sample into the cluster closest to it, i = 1, 2, ..., p, p is the number of cluster centers at the nth granularity level, so that the training data is divided into different subsets based on different granularities, and all subsets are constructed as a multi-granularity hierarchical degradation space.

6. The unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to claim 2 or 5, characterized in that: The restoration tasks include denoising, deblurring, compression artifact removal, rain streak removal, haze removal and dark light enhancement.

7. The unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to claim 6, characterized in that: The construction process of the multi-granularity hybrid expert restoration model in S4: For the dark light enhancement task, we select the low-light image enhancement network RetinexFormer as the expert model; The Transformer model Restormer is selected as the expert model for denoising, deblurring, compression artifact removal, rain streak removal, and dehazing tasks; Dark light enhancement, deblurring, and defogging tasks are each processed using two expert models; snow removal, denoising, and compression artifact removal tasks are each processed using three expert models, and rain streak removal tasks are processed using four expert models; The expert model is trained using L1 loss l1: in is the network prediction result, and Y is the label; After training, a multi-granularity and multi-expert restoration model is formed.

8. The unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to claim 7, characterized in that: The training process of S5 includes: S51. Using two sets of multi-layer perceptrons to perform degradation estimation based on degradation features deg and particle size estimation gran ; S52: Reusing the routing network based on degradation estimation and granularity estimation and Do routing based on degradation estimation and granularity estimation: in is a collection of multiple expert networks at the nth granularity level, Based on The kth expert network selected by the router at the nth granularity level, For multi-granularity hybrid expert models The top-down expert network set of the branch, Based on Routing in Select the network with the best performance; S53. The following loss function is used for the constraints of degradation estimation and granularity estimation: in is the i-th cluster center at the finest granularity level; S54, jointly using the following loss function to optimize the degradation estimation network, the routing network based on the degradation estimation, the granularity estimation network, and the routing network based on the granularity estimation: in It is a common load balancing loss in the hybrid expert model, which is used to balance the load of each expert. α and β are two hyperparameters used to determine and The influence degree of is set to 0.1 and 0.01 respectively.

9. The unified method for multiple restoration tasks based on granularity-aware image degradation estimation according to claim 1, characterized in that: The hybrid expert routing network based on degradation estimation and granularity estimation in S6 includes a routing network based on degradation estimation and a routing network based on granularity estimation; The degradation estimation network includes fully connected layers MD1 to MD5; MD1 is used to perform a first full connection operation and a first activation operation on the degradation features of the input image; MD2 is used to perform a second full connection operation and a second activation operation on the output features of MD1; MD3 is used to perform a third full connection operation and a third activation operation on the output features of MD2; MD4 is used to perform a fourth full connection operation and a fourth activation operation on the output features of MD3; MD5 is used to perform the fifth full connection operation and the fifth activation operation on the output features of MD4; The first to fifth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The first to fourth activation operations are all LeakyReLU activation functions, and the fifth activation operation is a one-dimensional BatchNorm operation. The granularity estimation network includes fully connected layers MG1 to MG5; MG1 is used to perform a sixth full connection operation and a sixth activation operation on the degradation features of the input image; MG2 performs the seventh full connection operation and the seventh activation operation on the output features of MG1; MG3 performs an eighth full connection operation and an eighth activation operation on the output features of MG2; MG4 performs the ninth full connection operation and the ninth activation operation on the output features of MG3; MG5 performs the tenth full connection operation and the tenth activation operation on the output features of MG4; The sixth to tenth fully connected operations are all fully connected layers with 512-dimensional input and 512-dimensional output. The sixth to ninth activation operations are all LeakyReLU activation functions, and the tenth activation operation is a one-dimensional BatchNorm operation. The routing network based on degradation estimation includes a fully connected layer GD1; GD1 is used to perform the eleventh fully connected operation on the MD5 output image degradation estimation; The routing network based on granular estimation includes a fully connected layer GG1; GG1 is used to perform the twelfth fully connected operation on the granularity estimation of the MG5 output image; The input dimension of the eleventh fully connected operation is 512, and the output dimension is 19; The input dimension of the twelfth fully connected operation is 512 and the output dimension is 3.

Images