An integrated image restoration method based on content prompts and task prompts

Through the integrated image recovery method of content prompts and task prompts, complex image degradation problems are solved, especially image recovery under different weather conditions, achieving more efficient image recovery effects and improving image quality and visual effects.

CN118918046BActive Publication Date: 2024-12-06KUNMING UNIV OF SCI & TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411378707.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2024-12-06
Estimated Expiration
2044-09-30

AI Technical Summary

Technical Problem

The prior art is not effective in dealing with complex image degradation conditions, especially in different weather conditions, and image recovery methods are difficult to fully restore image quality and visual effects.

Method used

The integrated image recovery method based on content prompts and task prompts is adopted. Through content prompt generation network and task prompt generation network collaborative learning, high-frequency and low-frequency information are extracted, and combined with image recovery network, the comprehensive recovery of degraded images is achieved.

Benefits of technology

It improves the accuracy and robustness of image recovery, can restore image degradation under different weather conditions more comprehensively, improves image quality and visual effects, and enhances the reliability and stability of practical applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918046B_ABST
    Figure CN118918046B_ABST
Patent Text Reader

Abstract

The present invention relates to an integrated image restoration method based on content prompts and task prompts, and belongs to the field of computer image processing. The present invention comprises the steps of: inputting a pre-processed degraded image into a content prompt generation network to extract high-frequency prompts and low-frequency prompts and then merging them into content prompts; inputting the pre-processed degraded image into a task prompt generation network, and using these learned vectors to obtain task prompts during the forward propagation process; inputting the pre-processed degraded image into an image restoration network to perform feature extraction through the forward propagation process. At the same time, through a content prompt embedding module and a task prompt embedding module, content prompts and task prompts are embedded into features, and information related to content and tasks is used to guide the image restoration process, thereby restoring the image; training a content prompt generation network, a task prompt generation network and an image restoration network. The present invention makes the image restoration process more efficient.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an integrated image restoration method based on content prompts and task prompts, and belongs to the technical field of computer image processing. Background Art

[0002] With the rapid development of digital image processing, the restoration of image quality has attracted much attention. However, in practical applications, due to the influence of various factors, such as weather changes, uneven illumination, sensor noise, etc., images are often damaged and degraded to varying degrees, thus affecting their quality and visual effects. In response to this challenge, deep learning-based image restoration technology has emerged, among which the integrated method leads the research trend with its unique characteristics. Traditional image restoration methods often rely on mathematical models or heuristic rules. Although they have achieved certain results in some cases, they are often helpless in the face of complex degradation situations. In contrast, deep learning-based methods can restore degraded images more accurately by learning the features and rules in a large amount of image data, thereby improving image quality and visual effects. Specifically, the integrated image restoration method based on content prompts and task prompts innovatively decomposes the image restoration task into two subtasks: content prompt generation and task prompt generation, and achieves accurate restoration of degraded images through the collaborative learning of these two subtasks. The content prompt generation network uses the degraded image to extract high-frequency and low-frequency information to generate prompt information related to the image content; while the task prompt generation network uses the learned task-related information, such as classification and segmentation, to generate prompt information related to specific tasks. Subsequently, the image restoration network combines content prompts and task prompts to achieve comprehensive restoration of the degraded image, thereby producing a clearer and more realistic restored image. In summary, the integrated image restoration method based on content prompts and task prompts not only fully utilizes the advantages of deep learning, but also explores new directions and ideas in the field of image restoration. This method provides a new solution to the complex image degradation problem in actual scenes, which is of great significance for improving image quality and visual effects. Summary of the invention

[0003] In order to overcome the limitations of current methods, the present invention proposes an integrated image restoration method based on content prompts and task prompts to deal with problems such as noise, artifacts, and multiple degradations in the same scene that may occur in image restoration.

[0004] The technical solution of the present invention is: an integrated image restoration method based on content prompts and task prompts, and the specific steps of the method are as follows:

[0005] Step 1: Obtain the degraded image of RGB three channels and preprocess the degraded image;

[0006] Step 2: Construct a content hint generation network and input the preprocessed degraded image into the content hint generation network; the content hint generation network includes a high-frequency information extraction branch and a low-frequency information extraction branch, which are used to extract high-frequency hints and low-frequency hints, and then merge the high-frequency hints and low-frequency hints into content hints;

[0007] Step 3: Build a task prompt generation network and input the preprocessed degraded image into the task prompt generation network; the task prompt generation network uses a set of randomly initialized learnable vectors, which are used to capture information related to a specific task; in the forward propagation process, the task prompt generation network uses these learned vectors to obtain task prompts;

[0008] Step 4: Construct an image restoration network and input the preprocessed degraded image into the image restoration network; the image restoration network extracts shallow features through the forward propagation process to understand the features and structure of the input image; at the same time, through the content prompt embedding module and the task prompt embedding module, the content prompts generated by the content prompt generation network and the task prompts generated by the task prompt generation network are embedded into the shallow features, so that the network can better use the content and task-related information to guide the image restoration process, thereby producing a clearer and more realistic restored image;

[0009] Step 5: Train the content prompt generation network, task prompt generation network and image restoration network;

[0010] Furthermore, in Step 1, the degraded image of the RGB three channels is randomly cropped to The size is then normalized before being input into the content prompt generation network, the task prompt generation network, and the image restoration network. In addition, data enhancement methods such as random cropping, horizontal flipping, and contrast enhancement are used to increase the diversity of data samples to improve the robustness and generalization ability of the network.

[0011] Furthermore, in Step 2, extracting high-frequency prompts includes the following:

[0012] For the high-frequency information extraction branch, the preprocessed degraded image is first converted to the frequency domain through Fourier transform. This transform can better capture the high-frequency information in the image. Then, the features in the frequency domain are processed using a gating operation consisting of convolution and activation functions to obtain high-frequency weights. This gating operation can dynamically adjust the network's attention to high-frequency information, allowing the network to more flexibly screen out high-frequency features that are beneficial to image restoration. Then, the obtained high-frequency weights are multiplied by the high-frequency features in the frequency domain to highlight useful high-frequency information. Finally, the generated high-frequency cues are obtained through the inverse Fourier transform, which provides key guidance and auxiliary information for the image restoration network. Such a design enables the network to more effectively extract and utilize high-frequency information in different image scenes. The entire process is defined as:

[0013] ;

[0014] ;

[0015] ;

[0016] in, represents the preprocessed degraded image, represents the Fourier transform, Represents high frequency features, represents the convolution operation, represents the activation function, represents the high frequency weight, represents the inverse Fourier transform, represents the dot product operation, Indicates a high frequency prompt.

[0017] In Step 2, extracting low-frequency cues includes the following:

[0018] For the low-frequency information extraction branch, first, the preprocessed degraded image is subjected to a convolution operation to reduce the resolution to obtain low-frequency features related to the image contour. Next, the low-frequency features in the spatial domain are processed using a gating operation consisting of a depthwise separable convolution and an activation function to obtain low-frequency weights. This gating operation can dynamically adjust the network's attention to low-frequency information, allowing the network to more flexibly screen out low-frequency features that are beneficial to image restoration. Then, the obtained low-frequency weights are multiplied by the low-frequency features in the frequency domain to highlight useful low-frequency information. Finally, the generated low-frequency cues are obtained through a depthwise separable convolution, which provides key guidance and auxiliary information for the image restoration network. Such a design enables the network to more effectively extract and utilize low-frequency information in different image scenes. The entire process is defined as:

[0019] ;

[0020] ;

[0021]

[0022] in, represents the preprocessed degraded image, Represents low-frequency characteristics, represents the convolution operation, represents a depth-wise separable convolution operation, represents the activation function, represents the low frequency weight, represents the dot product operation, Indicates low frequency prompt.

[0023] In the Step 2, merging the high-frequency prompts and the low-frequency prompts into content prompts includes the following:

[0024] The high-frequency cues and low-frequency cues are concatenated on the channel by the concatenation operation, and the high-frequency cues and low-frequency cues are merged into content cues by the convolution operation. The whole process is defined as:

[0025] ;

[0026] in, Indicates low frequency prompt, Indicates high frequency prompts, Represents a splicing operation, represents the convolution operation, Indicates content hint;

[0027] Furthermore, in Step 3, the specific operation steps of the task prompt generation network are as follows:

[0028] First, a set of randomly initialized learnable vectors At the same time, the preprocessed degraded image is convolved, global average pooled, and convolved to obtain shallow features, and then the task prompt weights are obtained through the Softmax operation. Finally, these weights are adjusted in the learnable vector, and finally the task prompt is obtained through convolution. The whole process is defined as:

[0029] ;

[0030]

[0031] in, represents the preprocessed degraded image, represents the convolution operation, represents global average pooling, Represents shallow features, represents the activation function, represents the dot product operation, represents a set of randomly initialized learnable vectors, Indicates task prompts.

[0032] Furthermore, in Step 4, the specific operation steps of the image restoration network are as follows:

[0033] For the content hint embedding module, first, the shallow feature A and the content hint are extracted through Unet respectively, and the extracted features are recorded as and It is worth noting that the weights of the two Unets are shared. Then the extracted features are concatenated in the channel dimension, and then the dimension is reduced through maximum pooling, MLP and ReLU, and then the content embedding weight is obtained through the Softmax operation. , and then the content embedding weights are respectively and After multiplication, we get the weighted features and ,Finally, the weighted features are added to obtain the content hint embedding features. The whole process is defined as:

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] in, represents the shallow feature A, It represents the features extracted by shallow feature A after Unet. Indicates content hint. Indicates the features extracted after the content prompts through Unet. represents the maximum pooling, represents the fully connected layer, represents the activation function, represents the dot product operation, represents the addition operation, represents the content embedding weight, Represents the weighted features , Represents the weighted features , Represents content hint embedding feature.

[0041] For the task prompt embedding module, first, the shallow feature A and the task prompt are extracted through Unet respectively, and the extracted features are recorded as and It is worth noting that the weights of the two Unets are shared. Then the extracted features are concatenated in the channel dimension, and then the dimension is reduced through maximum pooling, MLP and ReLU, and then the task embedding weight is obtained through the Softmax operation. , and then the task embedding weights are respectively and After multiplication, we get the weighted features and , and finally add the weighted features to get the task prompt embedding feature. The whole process is defined as:

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, represents the shallow feature A, It represents the features extracted by shallow feature A after Unet. Indicates task prompts. It represents the features extracted after the task prompt passes through Unet. represents the maximum pooling, represents the fully connected layer, represents the activation function, represents the dot product operation, represents the addition operation, represents the task embedding weight, Represents the weighted features , Represents the weighted features , Represents the task hint embedding feature.

[0049] Finally, through the fusion operation, the content hint embedding features and the task hint embedding features are concatenated on the channel, and then the restored image is obtained through convolution. The whole process is defined as:

[0050] ;

[0051] in, represents the task prompt embedding feature, represents the content hint embedding feature, Represents a splicing operation, represents the convolution operation, Restores the image.

[0052] Furthermore, the Step 5 specifically includes the following:

[0053] Step 5.1, use AdamW optimizer to optimize the parameters in the content prompt generation network, task prompt generation network and image restoration network;

[0054] Step 5.2, when training the content hint generation network, define a loss function, which compares the content hint obtained in Step 2 with the clear image;

[0055] use The loss function optimizes the content hint generation network and is defined as:

[0056] ;

[0057] in, Indicates content hint. Indicates a clear image. express Loss function.

[0058] Step 5.3, when training the task prompt generation network, define a loss function, which compares the task prompt obtained in Step 3 with the clear image;

[0059] use The loss function optimization task prompts the generation network, which is defined as:

[0060] ;

[0061] in, Indicates task prompts. Indicates a clear image. express Loss function.

[0062] Step 5.4, define a loss function when training the image restoration network, which compares the restored image obtained in Step 4 with the clear image;

[0063] use The loss function optimizes the image restoration network and is defined as:

[0064] ;

[0065] in, Represents the restored image, Indicates a clear image. express Loss function.

[0066] The beneficial effects of the present invention are:

[0067] 1. Comprehensive restoration of image degradation under different weather conditions: This integrated image restoration method can comprehensively restore image degradation under different weather conditions through the integrated design of content prompts and task prompts. The content prompt generation network uses high-frequency and low-frequency information extraction branches to extract important features in the image, while the task prompt generation network captures information related to specific tasks, allowing the network to fully understand the degradation of images under different weather conditions, thereby achieving more comprehensive image restoration.

[0068] 2. Enhance the accuracy and robustness of image restoration: The strategy of multi-source information fusion combined with the design of content prompt and task prompt generation network helps to improve the accuracy and robustness of image restoration. The content prompt embedding module and the task prompt embedding module can effectively embed content prompts and task prompts into image features, allowing the network to more accurately understand the features and structure of the input image, thereby producing clearer and more realistic restoration results.

[0069] 3. Improve the efficiency and speed of the network: This method can improve the efficiency and speed of the network by adopting preprocessing and data enhancement methods, as well as optimizing the loss function and optimizer during the training process. The preprocessing step includes random cropping and normalization operations on the input image, the data enhancement method increases the diversity of data samples, and the optimized training process further improves the convergence speed of the network, making the image restoration process more efficient.

[0070] 4. Enhanced reliability and stability of practical applications: By training the content prompt generation network, task prompt generation network and image restoration network, they can adapt to image degradation under different weather conditions, further enhancing the reliability and stability of practical applications. This means that the method has wider applicability and more reliable image restoration effects in actual scenarios, providing a more reliable solution for practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the process structure of the method of the present invention;

[0072] Figure 2 Generate a network structure diagram for the content prompt of the method of the present invention;

[0073] Figure 3 Generate a network structure diagram for the task prompt of the method of the present invention;

[0074] Figure 4 A structural diagram of a content prompt embedding module of the method of the present invention;

[0075] Figure 5 It is a structural diagram of the task prompt embedding module of the method of the present invention;

[0076] Figure 6 This is a comparison diagram of the effects of the image restoration methods of different methods according to the present invention. DETAILED DESCRIPTION

[0077] Example 1: Figure 1-Figure 6 As shown, an integrated image restoration method based on content prompts and task prompts, the specific steps of the method are as follows:

[0078] Step 1: Get the degraded image of RGB three channels and preprocess the degraded image: randomly crop the degraded image of RGB three channels to The size is then normalized before being input into the content prompt generation network, task prompt generation network, and image restoration network; in addition, data enhancement methods such as random cropping, horizontal flipping, and contrast enhancement are used to increase the diversity of data samples to improve the robustness and generalization ability of the network. The normalization operation is calculated as follows:

[0079]

[0080] in is the normalized value, is the value before normalization, is the input sequence. represents the mean value of the input sequence, Represents the standard deviation of the input sequence.

[0081] Step 2: Construct a content hint generation network and input the preprocessed degraded image into the content hint generation network. The content hint generation network includes a high-frequency information extraction branch and a low-frequency information extraction branch, which are used to extract high-frequency hints and low-frequency hints, and then merge them into content hints;

[0082] In Step 2, extracting high-frequency cues includes the following:

[0083] For the high-frequency information extraction branch, the preprocessed degraded image is first converted to the frequency domain through Fourier transform. This transform can better capture the high-frequency information in the image. Then, the features in the frequency domain are processed using a gating operation consisting of convolution and activation functions to obtain high-frequency weights. This gating operation can dynamically adjust the network's attention to high-frequency information, allowing the network to more flexibly screen out high-frequency features that are beneficial to image restoration. Then, the obtained high-frequency weights are multiplied by the high-frequency features in the frequency domain to highlight useful high-frequency information. Finally, the generated high-frequency cues are obtained through the inverse Fourier transform, which provides key guidance and auxiliary information for the image restoration network. Such a design enables the network to more effectively extract and utilize high-frequency information in different image scenes. The entire process is defined as:

[0084] ;

[0085] ;

[0086] ;

[0087] in, represents the preprocessed degraded image, represents the Fourier transform, Represents high frequency features, represents the convolution operation, represents the activation function, represents the high frequency weight, represents the inverse Fourier transform, represents the dot product operation, Indicates a high frequency prompt.

[0088] In Step 2, extracting low-frequency cues includes the following:

[0089] For the low-frequency information extraction branch, first, the preprocessed degraded image is subjected to a convolution operation to reduce the resolution to obtain low-frequency features related to the image contour. Next, the low-frequency features in the spatial domain are processed using a gating operation consisting of a depthwise separable convolution and an activation function to obtain low-frequency weights. This gating operation can dynamically adjust the network's attention to low-frequency information, allowing the network to more flexibly screen out low-frequency features that are beneficial to image restoration. Then, the obtained low-frequency weights are multiplied by the low-frequency features in the frequency domain to highlight useful low-frequency information. Finally, the generated low-frequency cues are obtained through a depthwise separable convolution, which provides key guidance and auxiliary information for the image restoration network. Such a design enables the network to more effectively extract and utilize low-frequency information in different image scenes. The entire process is defined as:

[0090] ;

[0091] ;

[0092]

[0093] in, represents the preprocessed degraded image, Represents low-frequency characteristics, represents the convolution operation, represents a depth-wise separable convolution operation, represents the activation function, represents the low frequency weight, represents the dot product operation, Indicates low frequency prompt.

[0094] In the Step 2, merging the high-frequency prompts and the low-frequency prompts into the content prompts includes the following steps: splicing the high-frequency prompts and the low-frequency prompts on the channel by a splicing operation, and merging the high-frequency prompts and the low-frequency prompts into the content prompts by a convolution operation. The whole process is defined as:

[0095] ;

[0096] in, Indicates low frequency prompt, Indicates high frequency prompts, Represents a splicing operation, represents the convolution operation, Indicates content hint.

[0097] Step 3: Build a task prompt generation network and input the preprocessed degraded image into the task prompt generation network. The network uses a set of randomly initialized learnable vectors that are used to capture information related to a specific task. During the forward propagation process, the task prompt generation network uses these learned vectors to obtain task prompts;

[0098] The specific operation steps of the task prompt generation network in Step 3 are as follows:

[0099] First, a set of randomly initialized learnable vectors At the same time, the preprocessed degraded image is convolved, global average pooled, and convolved to obtain shallow features, and then the task prompt weights are obtained through the Softmax operation. Finally, these weights are adjusted in the learnable vector, and finally the task prompt is obtained through convolution. The whole process is defined as:

[0100] ;

[0101]

[0102] in, represents the preprocessed degraded image, represents the convolution operation, represents global average pooling, Represents shallow features, represents the activation function, represents the dot product operation, represents a set of randomly initialized learnable vectors, Indicates task prompts.

[0103] Step 4: Build an image restoration network and input the preprocessed degraded image into the image restoration network. The network performs shallow feature extraction through the forward propagation process to understand the features and structure of the input image. At the same time, through the content prompt embedding module and the task prompt embedding module, the content prompts generated by the content prompt generation network and the task prompts generated by the task prompt generation network are embedded into the shallow features. In this way, the network can better utilize content and task-related information to guide the image restoration process, thereby producing clearer and more realistic restored images;

[0104] The specific operation steps of the image restoration network in Step 4 are as follows:

[0105] For the content hint embedding module, first, the shallow feature A and the content hint are extracted through Unet respectively, and the extracted features are recorded as and It is worth noting that the weights of the two Unets are shared. Then the extracted features are concatenated in the channel dimension, and then the dimension is reduced through maximum pooling, MLP and ReLU, and then the content embedding weight is obtained through the Softmax operation. , and then the content embedding weights are respectively and After multiplication, we get the weighted features and ,Finally, the weighted features are added to obtain the content hint embedding features. The whole process is defined as:

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] in, represents the shallow feature A, It represents the features extracted by shallow feature A after Unet. Indicates content hint. Indicates the features extracted after the content prompts through Unet. represents the maximum pooling, represents the fully connected layer, represents the activation function, represents the dot product operation, represents the addition operation, represents the content embedding weight, Represents the weighted features , Represents the weighted features , Represents content hint embedding feature.

[0113] For the task prompt embedding module, first, the shallow feature A and the task prompt are extracted through Unet respectively, and the extracted features are recorded as and It is worth noting that the weights of the two Unets are shared. Then the extracted features are concatenated in the channel dimension, and then the dimension is reduced through maximum pooling, MLP and ReLU, and then the task embedding weight is obtained through the Softmax operation. , and then the task embedding weights are respectively and After multiplication, we get the weighted features and , and finally add the weighted features to get the task prompt embedding feature. The whole process is defined as:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] ;

[0120] in, represents the shallow feature A, It represents the features extracted by shallow feature A after Unet. Indicates task prompts. It represents the features extracted after the task prompt passes through Unet. represents the maximum pooling, represents the fully connected layer, represents the activation function, represents the dot product operation, represents the addition operation, represents the task embedding weight, Represents the weighted features , Represents the weighted features , Represents the task hint embedding feature.

[0121] Finally, through the fusion operation, the content hint embedding features and the task hint embedding features are concatenated on the channel, and then the restored image is obtained through convolution. The whole process is defined as:

[0122] ;

[0123] in, represents the task prompt embedding feature, represents the content hint embedding feature, Represents a splicing operation, represents the convolution operation, Restores the image.

[0124] Step 5: Train the content prompt generation network, task prompt generation network and image restoration network;

[0125] Step 5.1, use AdamW optimizer to optimize the parameters in the content prompt generation network, task prompt generation network and image restoration network;

[0126] Step 5.2, when training the content hint generation network, define a loss function, which compares the content hint obtained in Step 2 with the clear image;

[0127] use The loss function optimizes the content hint generation network and is defined as:

[0128] ;

[0129] in, Indicates content hint. Indicates a clear image. express Loss function.

[0130] Step 5.3, when training the task prompt generation network, define a loss function, which compares the task prompt obtained in Step 3 with the clear image;

[0131] use The loss function optimization task prompts the generation network, which is defined as:

[0132] ;

[0133] in, Indicates task prompts. Indicates a clear image. express Loss function.

[0134] Step 5.4, define a loss function when training the image restoration network, which compares the restored image obtained in Step 4 with the clear image;

[0135] use The loss function optimizes the image restoration network and is defined as:

[0136] ;

[0137] in, Represents the restored image, Indicates a clear image. express Loss function.

[0138] To verify the effectiveness of the method of the present invention, this embodiment uses Allweather as training data. The training data includes 9000 images sampled from Snow100K, 1069 from Raindrop, and 9000 from Outdoor-Rain. Snow100K provides synthetic images caused by snow, Raindrop contains real raindrop images, and Outdoor-Rain provides synthetic images caused by fog and raindrops. To evaluate our method, the Test1 dataset, RainDrop test dataset, and Snow100k-L test set were used. The algorithm of the present invention was developed based on the Pytorch framework and trained in an environment equipped with an NVIDIA GTX 4090 graphics card. During the model training process, the AdamW optimizer was selected to optimize the network parameters. In the training configuration, the Batch size was set to 16 and the initial learning rate was set to 0.00002. In order to effectively control the change of the learning rate, the cosine annealing algorithm was used for adjustment, and a warmup phase was set, accounting for 1 / 20 of the total training cycle. Under this configuration, after a total of 200 rounds of training, the algorithm of the present invention showed good performance. This training strategy not only ensures the convergence efficiency of the model, but also optimizes the adaptability of the model in complex environments and the effect of image restoration.

[0139] Furthermore, the present invention uses two key image quality evaluation criteria, PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index), when evaluating the image restoration effect. PSNR is a widely used image quality metric, which is mainly used to compare the similarity between the original image and the processed image. PSNR is measured in decibels (dB) and provides a quantitative method to evaluate the degree of image distortion. Generally speaking, a higher PSNR value indicates lower distortion, that is, higher image quality; conversely, a lower PSNR value usually represents higher distortion, that is, poor image quality. SSIM is another criterion for measuring image quality, which evaluates the similarity between two images by comparing their brightness, contrast and structure. SSIM specifically considers the perceptual characteristics of the human visual system (HVS) to images, so it not only focuses on the physical similarity of the images, but also simulates the subjective evaluation of the image quality by the human eye. The closer the SSIM value is to 1, the more similar the two images are visually, and the closer the image restoration effect is to the ideal state. Through these two indicators, the present invention can objectively evaluate the performance of its image restoration technology to ensure that the image processing results meet both technical requirements and the visual perception of the human eye.

[0140] Furthermore, on the Test1 test data set, the image restoration method of the present invention is compared with All-in-one, AirNet, TUM, Transweather and WGWS. The experimental results are shown in Table 1. According to the data, the peak signal-to-noise ratio (PSNR) of the present method reaches 29.84, and the structural similarity index (SSIM) is 0.936, which significantly proves the efficiency and superiority of its image restoration.

[0141] Similarly, the method of the present invention is compared with similar image restoration techniques on the RainDrop test data set. The experimental results shown in Table 2 show that the method of the present invention achieves a PSNR of 31.59 and a SSIM of 0.943. This result further confirms the effectiveness and practicality of the method of the present invention. These data emphasize the significant advantages of the present invention in image clarity and visual quality restoration.

[0142] Table 1 shows the objective evaluation comparison of different methods on the Test1 test set.

[0143] method PSNR↑ SSIM ↑ All-in-one 24.71 0.898 AirNet 23.12 0.837 TUM 23.92 0.855 Transweather 23.18 0.841 WGWS 25.31 0.901 Method of the present invention 29.84 0.936

[0144] Table 2 shows the objective evaluation comparison of different methods on the RainDrop test set.

[0145] method PSNR↑ SSIM ↑ All-in-one 31.12 0.927 AirNet 28.23 0.892 TUM 30.75 0.912 Transweather 28.98 0.902 WGWS 31.31 0.932 Method of the present invention 31.59 0.943

[0146] Furthermore, the present invention compares the visual effects of the proposed method with existing image restoration techniques such as All-in-one, AirNet, TUM, Transweather and WGWS. Figure 6 As shown in Figure 2, it can be seen that the proposed method can better restore the image, while the image restoration results of other methods are relatively rough.

[0147] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An integrated image restoration method based on content prompts and task prompts, characterized in that: The specific steps of the method are as follows: Step 1: Obtain the degraded image of RGB three channels and preprocess the degraded image; Step 2: Build a content hint generation network and input the preprocessed degraded image into the content hint generation network; The content prompt generation network includes a high-frequency information extraction branch and a low-frequency information extraction branch, which are used to extract high-frequency prompts and low-frequency prompts, and then merge the high-frequency prompts and the low-frequency prompts into content prompts; Step 3: Build a task prompt generation network and input the preprocessed degraded image into the task prompt generation network; The task cue generation network uses a set of randomly initialized learnable vectors that are designed to capture information relevant to a specific task; During the forward propagation process, the task prompt generation network uses these learned vectors to obtain task prompts; Step 4: Construct an image restoration network and input the preprocessed degraded image into the image restoration network; the image restoration network extracts shallow features through the forward propagation process; at the same time, through the content prompt embedding module and the task prompt embedding module, the content prompt generated by the content prompt generation network and the task prompt generated by the task prompt generation network are embedded into the shallow features, and the content and task-related information are used to guide the image restoration process, thereby restoring the image; Step 5: Train the content prompt generation network, task prompt generation network and image restoration network.

2. The integrated image restoration method based on content prompts and task prompts according to claim 1, characterized in that: In the Step 1, the degraded image of the RGB three channels is randomly cropped to The size is then normalized before being input into the content prompt generation network, the task prompt generation network, and the image restoration network. In addition, data enhancement methods such as random cropping, horizontal flipping, and contrast enhancement are used to increase the diversity of data samples.

3. The integrated image restoration method based on content prompts and task prompts according to claim 1 is characterized in that: In Step 2, extracting high-frequency cues includes the following: For the high-frequency information extraction branch, the preprocessed degraded image is first converted to the frequency domain through Fourier transform; then, the features in the frequency domain are processed using a gating operation consisting of convolution and activation functions to obtain high-frequency weights; then, the obtained high-frequency weights are multiplied by the high-frequency features in the frequency domain; finally, the generated high-frequency cues are obtained through inverse Fourier transform; the whole process is defined as: ; ; ; in, represents the preprocessed degraded image, represents the Fourier transform, Represents high frequency features, represents the convolution operation, represents the activation function, represents the high frequency weight, represents the inverse Fourier transform, represents the dot product operation, Indicates a high frequency prompt.

4. The integrated image restoration method based on content prompts and task prompts according to claim 1, characterized in that: In Step 2, extracting low-frequency cues includes the following: For the low-frequency information extraction branch, first, the preprocessed degraded image is convolved to reduce the resolution to obtain low-frequency features related to the image contour; then, the low-frequency features in the spatial domain are processed using a gating operation consisting of a depthwise separable convolution and an activation function to obtain low-frequency weights; then, the obtained low-frequency weights are multiplied by the low-frequency features in the frequency domain; finally, the generated low-frequency cues are obtained through depthwise separable convolution; the whole process is defined as: ; ; ; in, represents the preprocessed degraded image, Represents low-frequency characteristics, represents the convolution operation, represents a depth-wise separable convolution operation, represents the activation function, represents the low frequency weight, represents the dot product operation, Indicates low frequency prompt.

5. The integrated image restoration method based on content prompts and task prompts according to claim 1, characterized in that: In the Step 2, merging the high-frequency prompts and the low-frequency prompts into content prompts includes the following: The high-frequency cues and low-frequency cues are concatenated on the channel by the concatenation operation, and the high-frequency cues and low-frequency cues are merged into content cues by the convolution operation; the whole process is defined as: ; in, Indicates low frequency prompt, Indicates high frequency prompts, Represents a splicing operation, represents the convolution operation, Indicates content hint.

6. The integrated image restoration method based on content prompts and task prompts according to claim 1, characterized in that: In Step 3, the specific operation process of the task prompt generation network is as follows: First, a set of randomly initialized learnable vectors At the same time, the preprocessed degraded image is convolved, global average pooled, and convolved to obtain shallow features, and then the task prompt weights are obtained through the Softmax operation. Finally, these weights are adjusted in the learnable vector, and finally the task prompt is obtained through convolution; the whole process is defined as: ; ; in, represents the preprocessed degraded image, represents the convolution operation, represents global average pooling, Represents shallow features, represents the activation function, represents the dot product operation, represents a set of randomly initialized learnable vectors, Indicates task prompts.

7. The integrated image restoration method based on content prompts and task prompts according to claim 1, characterized in that: In Step 4, the specific operations of the image restoration network include: For the content hint embedding module, first, the shallow feature A and the content hint are extracted through Unet respectively, and the extracted features are recorded as and , then the extracted features are concatenated in the channel dimension, and then the dimension is reduced through maximum pooling, MLP and ReLU, and then the content embedding weight is obtained through the Softmax operation , and then the content embedding weights are respectively and After multiplication, we get the weighted features and ,Finally, the weighted features are added to obtain the content hint embedding features. The whole process is defined as: ; ; ; ; ; ; in, represents the shallow feature A, It represents the features extracted by shallow feature A after Unet. Indicates content hint. Indicates the features extracted after the content prompts through Unet. represents the maximum pooling, represents the fully connected layer, represents the activation function, represents the dot product operation, represents the addition operation, represents the content embedding weight, Represents the weighted features , Represents the weighted features , Represents content hint embedding feature.

8. The integrated image restoration method based on content prompts and task prompts according to claim 1, characterized in that: In the Step 4, the specific operation of the image restoration network also includes: For the task prompt embedding module, first, the shallow feature A and the task prompt are extracted through Unet respectively, and the extracted features are recorded as and It is worth noting that the weights of the two Unets are shared. Then the extracted features are concatenated in the channel dimension, and then the dimension is reduced through maximum pooling, MLP and ReLU, and then the task embedding weight is obtained through the Softmax operation. , and then the task embedding weights are respectively and After multiplication, we get the weighted features and , and finally add the weighted features to get the task prompt embedding features. The whole process is defined as: ; ; ; ; ; ; in, represents the shallow feature A, It represents the features extracted by shallow feature A after Unet. Indicates task prompts. It represents the features extracted after the task prompt passes through Unet. represents the maximum pooling, represents the fully connected layer, represents the activation function, represents the dot product operation, represents the addition operation, represents the task embedding weight, Represents the weighted features , Represents the weighted features , Represents the task prompt embedding features; Finally, through the fusion operation, the content hint embedding features and the task hint embedding features are concatenated on the channel, and then the restored image is obtained through convolution. The whole process is defined as: ; in, represents the task prompt embedding feature, represents the content hint embedding feature, Represents a splicing operation, represents the convolution operation, Restores the image.

9. The integrated image restoration method based on content prompts and task prompts according to claim 1, characterized in that: The Step 5 specifically includes: Step 5.1, use AdamW optimizer to optimize the parameters in the content prompt generation network, task prompt generation network and image restoration network; Step 5.2, when training the content hint generation network, define a loss function, which compares the content hint obtained in Step 2 with the clear image; use The loss function optimizes the content hint generation network and is defined as: ; in, Indicates content hint. Indicates a clear image. express Loss function; Step 5.3, when training the task prompt generation network, define a loss function, which compares the task prompt obtained in Step 3 with the clear image; use The loss function optimization task prompts the generation network, which is defined as: ; in, Indicates task prompts. Indicates a clear image. express Loss function; Step 5.4, define a loss function when training the image restoration network, which compares the restored image obtained in Step 4 with the clear image; use The loss function optimizes the image restoration network and is defined as: ; in, Represents the restored image, Indicates a clear image. express Loss function.

Citation Information

Patent Citations

  • A device and method for image processing

    WO2022096123A1