A method, apparatus, device, and medium for image restoration against various types of interference.

Abstract

This invention provides an image restoration method, apparatus, device, and medium for handling various types of interference, relating to the field of computer vision technology. It includes at least the following steps: processing a degraded sample image using a prompt generation network to generate content prompts, style prompts, and prompts to be learned; fusing and encoding multiple prompts using the prompt generation network to obtain a target prompt; combining the target prompt with the feature layer of the image restoration network to obtain a restored sample image during image restoration; training the image restoration model based on the restored sample image and a clean sample image to obtain a trained image restoration model; and inputting the image to be restored into the trained image restoration model to obtain the restored image, thereby improving the image restoration effect and performance against various types of interference.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image restoration method, apparatus, device, and medium for various types of interference. Background Technology

[0002] Computer vision (CV) is playing an increasingly important role in various fields of real life, widely applied in recognition, detection, and other scenarios, and has become an indispensable key technology for building smart cities, autonomous driving systems, and intelligent security systems. However, in practical applications, the quality of image acquisition has a decisive impact on the accuracy of subsequent processing tasks. Factors such as fog, rain, snow, camera shake, thermal noise, shadow occlusion, low lighting conditions, and image missing can significantly reduce image quality, generating damaged images. This, in turn, severely affects the performance of downstream tasks. Image restoration technology was developed to address these problems, aiming to restore damaged or degraded images to near their original state through algorithms. Against this backdrop, how to efficiently handle various types of image corruption problems to achieve high-quality image restoration has become a key research topic in the field of computer vision. With continuous technological advancements and increasingly diverse application scenarios, this challenge has prompted researchers to continuously explore more advanced solutions, striving to further improve the overall performance and reliability of image processing.

[0003] Image restoration technology is a comprehensive approach combining denoising and enhancement. It utilizes mathematical models and computer vision algorithms to optimize image quality and improve detail clarity. This technology aims to recover clearer and more accurate information from images affected by noise, rain, snow, fog, or degraded image quality due to low light, shadows, and other issues. By improving the overall quality of the image, image restoration not only enhances the visual effect of the image itself but also significantly improves the performance of subsequent processing steps such as semantic segmentation, object detection, and tracking, thereby greatly enhancing the accuracy and stability of results when analyzing these images. Therefore, in the face of complex and ever-changing interference factors, developing high-performance image restoration algorithms has become an indispensable key link in advancing the field of computer vision.

[0004] Currently, image restoration typically focuses on solving only one type of degradation problem, and its performance is often unsatisfactory when faced with multiple interfering factors. Furthermore, processing various types of damaged images requires deploying multiple independent models, which significantly increases the complexity in practical applications. Although multi-task image restoration algorithms have gained increasing attention in recent years, the types of tasks they can effectively handle remain limited, and the restoration results are often unsatisfactory. Therefore, how to develop an image restoration method that can flexibly cope with multiple interfering factors to improve the image restoration effect against various interferences is a technical problem that this invention urgently needs to solve. Summary of the Invention

[0005] Based on the above-mentioned technical problems, the present invention provides an image restoration method, apparatus, device and medium for various types of interference, aiming to improve the image restoration effect and performance for various types of interference.

[0006] The first aspect of the present invention provides an image restoration method for various types of interference, the method comprising:

[0007] The degraded images of the samples are respectively input into the image restoration network to be trained and the prompt generation network to be trained in the image restoration model;

[0008] The degraded sample image is processed by the prompt generation network to be trained, generating content prompt information, style prompt information, and learning prompt information corresponding to the degraded sample image. The content prompt information represents the image content prompt corresponding to the degraded sample image, the style prompt information represents the degradation type prompt corresponding to the degraded sample image, and the learning prompt information represents the semantic prompt information corresponding to the degraded sample image, which is of a higher dimension than the content prompt information and the style prompt information.

[0009] The target prompt information is obtained by fusing and encoding the content prompt information, the style prompt information, and the prompt information to be learned through the prompt generation network to be trained;

[0010] During the process of the image restoration network to be trained restoring the sample degraded image, the target prompt information is combined with the feature layer of the image restoration network to obtain the sample restored image output by the image restoration network.

[0011] Based on the restored sample image and the clean sample image corresponding to the degraded sample image, the image restoration model to be trained is trained until a trained image restoration model is obtained.

[0012] The image to be restored is input into the trained image restoration model to obtain the restored image.

[0013] A second aspect of the present invention provides an image restoration apparatus for various types of interference, the apparatus comprising:

[0014] The image input module is used to input the degraded sample images into the image restoration network and the prompt generation network to be trained in the image restoration model to be trained, respectively.

[0015] The prompt generation module is used to process the sample degraded image through the prompt generation network to be trained, and generate content prompt information, style prompt information and learning prompt information corresponding to the sample degraded image respectively; the content prompt information represents the image content prompt corresponding to the sample degraded image, the style prompt information represents the degradation type prompt corresponding to the sample degraded image, and the learning prompt information represents the semantic prompt information corresponding to the sample degraded image, which is of a higher dimension than the content prompt information and the style prompt information;

[0016] The prompt encoding module is used to fuse and encode the content prompt information, the style prompt information, and the prompt information to be learned through the prompt generation network to be trained, so as to obtain the target prompt information;

[0017] An image output module is used to combine the target prompt information with the feature layer of the image restoration network during the image restoration process of the image restoration network to be trained, so as to obtain the sample restoration image output by the image restoration network.

[0018] The model training module is used to train the image restoration model to be trained based on the sample restored image and the clean image corresponding to the sample degraded image until a trained image restoration model is obtained.

[0019] The image restoration module is used to input the image to be restored into the trained image restoration model to obtain the restored image.

[0020] A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image restoration method for various types of interference as described in the first aspect of the present invention.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image restoration method for multiple types of interference according to the first aspect of the present invention.

[0022] In the image restoration method for multiple types of interference provided by this invention, an image restoration model based on multiple cues is trained. This model comprises two parts: an image restoration network and a cue generation network. The image restoration network performs the image restoration task, while the cue generation network generates content cues, style cues, and cue information to be learned, representing the content cues, degradation type cues, and higher-dimensional semantic cues of the degraded image, respectively. These multiple cues are integrated to guide the image restoration network and assist in image restoration. Thus, the image restoration method of this invention uses multiple cues to implement style information cues, content information cues, and cue information to be learned, and integrates these multiple cues with the image restoration network to assist in image restoration. This achieves image restoration for multiple types of interference (i.e., multiple tasks), solving more types of image degradation problems at once. It can flexibly respond to multiple interference factors for image restoration, improving the image restoration effect and performance for multiple types of interference. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the steps of an image restoration method for various types of interference according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram illustrating the training of a content-aware processor according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram illustrating the training of a style perceptron according to an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of a perceptron to be learned according to an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of a prompt encoder according to an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the overall network structure of an image restoration model according to an embodiment of the present invention;

[0030] Figure 7 This is a flowchart illustrating the training process of an image restoration model according to an embodiment of the present invention;

[0031] Figure 8 This is a flowchart illustrating the reasoning process of an image restoration model according to an embodiment of the present invention;

[0032] Figure 9 This is a comparison chart of the visualization effects under three task scenarios proposed in an embodiment of the present invention;

[0033] Figure 10 This is a comparison chart of the visualization effects under five task scenarios proposed in an embodiment of the present invention;

[0034] Figure 11 This is a structural block diagram of an image restoration device for various types of interference provided in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of an image restoration method for various types of interference, as shown in an embodiment of the present invention. Figure 1 As shown, the image restoration method for various types of interference provided in this embodiment includes at least the following steps:

[0037] Step S11: Input the degraded sample images into the image restoration network to be trained and the prompt generation network to be trained in the image restoration model to be trained, respectively.

[0038] In this embodiment, the image restoration model to be trained consists of two parts: an image restoration network to be trained and a prompt generation network to be trained. The image restoration network performs the image restoration task, while the prompt generation network generates various prompts to assist the image restoration network in completing the image restoration. This embodiment provides a sample dataset for training the image restoration model. This sample dataset includes: degraded sample images and corresponding clean sample images. The clean sample images are the labels corresponding to the degraded sample images, representing the clear images after removing interference.

[0039] The degraded images in this embodiment are degraded image samples used for training the image restoration model. In this embodiment, the degraded images refer to images with one or more interferences, such as damaged images generated due to interference factors such as noise, haze, rain, snow, occlusion and / or low light. This is not limited. The sample dataset includes multiple degraded images with different interference factors.

[0040] In this embodiment, the degraded sample images are input into the image restoration network to be trained and the prompt generation network to be trained, respectively. The degraded sample images are processed in parallel by the image restoration network to be trained and the prompt generation network to be trained.

[0041] Step S12: The degraded sample image is processed by the prompt generation network to be trained, and content prompt information, style prompt information and learning prompt information corresponding to the degraded sample image are generated respectively.

[0042] In this embodiment, after inputting the degraded sample image into the prompt generation network to be trained, the network processes the image to generate content prompts, style prompts, and learnable prompts corresponding to the image. The content prompt represents the image content, the style prompt represents the degradation type, and the learnable prompt represents a higher-dimensional semantic prompt than the content and style prompts.

[0043] Step S13: The content prompt information, the style prompt information, and the prompt information to be learned are fused and encoded through the prompt generation network to be trained to obtain the target prompt information.

[0044] In this embodiment, after generating content prompts, style prompts, and learning prompts corresponding to the sample degraded image, the prompt generation network to be trained will further integrate these prompts by fusing and encoding them to obtain target prompts. These target prompts are used to guide the image restoration network in completing the image restoration task.

[0045] Step S14: During the process of image restoration of the sample degraded image by the image restoration network to be trained, the target prompt information is combined with the feature layer of the image restoration network to obtain the sample restored image output by the image restoration network.

[0046] In this embodiment, after the degraded sample image is input into the image restoration network to be trained, the image restoration network to be trained performs image restoration on the degraded sample image. During the image restoration process, the target prompt information output by the prompt generation network to be trained is combined with the feature layer in the image restoration network to be trained to achieve the restoration of a clear image and obtain the sample restored image output by the image restoration network to be trained.

[0047] Step S15: Based on the restored sample image and the clean sample image corresponding to the degraded sample image, train the image restoration model to be trained until a trained image restoration model is obtained.

[0048] In this embodiment, the image restoration model to be trained can be trained based on the sample restoration image corresponding to the sample degradation image and the sample clean image corresponding to the sample degradation image until a trained image restoration model is obtained.

[0049] In one alternative implementation, a loss between the sample restored image and the sample clean image can be calculated (e.g., L1 loss, etc.), and the model parameters of the image restoration model to be trained can be updated based on the calculated loss until a trained image restoration model is obtained.

[0050] Step S16: Input the image to be restored into the trained image restoration model to obtain the restored image.

[0051] In this embodiment, the trained image restoration model is used to restore degraded images with various types of interference. During the application of the trained image restoration model, the image to be restored can be input into the trained model to obtain the restored image output by the model. Here, the image to be restored is the image that needs to be restored to address image interference, and the restored image is the image after restoration of the image to be restored. This embodiment does not impose any restrictions on the interference factors involved in the image to be restored. The trained image restoration model in this embodiment can achieve image restoration under both single and combined interference conditions.

[0052] In this embodiment, a prompt generation network employs multiple prompts to implement style information prompts, content information prompts, and learning prompts. These multiple prompts are then fused with an image restoration network to assist in image restoration. This achieves image restoration for multiple types of interference (i.e., multiple tasks), addressing more types of image degradation problems simultaneously. It can flexibly handle various interference factors, further improving the image restoration effect and performance against multiple interferences. This also avoids designing and training dedicated models for each specific degradation case, reducing computational resource requirements and storage space consumption, and alleviating the workload of maintaining multiple independent systems.

[0053] In conjunction with the above embodiments, in one implementation, the present invention also provides an image restoration method for multiple types of interference, wherein the prompt generation network to be trained includes at least: a pre-trained content perceptron, a pre-trained style perceptron, and a perceptron to be trained; in this method, step S12 may specifically include steps S21 to S23:

[0054] Step S21: Input the degraded sample image into the pre-trained content awareness device to obtain the content prompt information.

[0055] In this embodiment, the pre-trained content perceptron and pre-trained style perceptron in the prompt generation network to be trained are perceptrons that have been pre-trained and have fixed parameters. They are not integrated with the image restoration network to be trained to participate in the training of the entire image restoration model. Moreover, in this embodiment, both the pre-trained content perceptron and the pre-trained style perceptron are obtained by retraining the image encoder in the pre-trained visual text network.

[0056] After a degraded image is input into the prompt generation network to be trained, the three perceptrons in the network (pre-trained content perceptron, pre-trained style perceptron, and the learner perceptron to be trained) will generate corresponding prompts: content prompts, style prompts, and learner prompts. Specifically, the degraded image is input into the pre-trained content perceptron to obtain the content prompts output by the pre-trained content perceptron.

[0057] Step S22: Input the degraded image of the sample into the pre-trained style perceptron to obtain the style cue information.

[0058] In this embodiment, after the degraded sample image is input into the prompt generation network to be trained, the degraded sample image is input into the pre-trained style perceptron to obtain the style prompt information output by the pre-trained style perceptron.

[0059] Step S23: Input the degraded image of the sample into the perceptron to be trained to obtain the learning prompt information.

[0060] Since the content awareness and style awareness are trained separately and are not integrated with the image restoration network to be trained in the training of the whole image restoration, in order to provide learnable parameters for the network in the whole image restoration process, this embodiment proposes a learnable awareness to infer that there are similar conditions between some scenes in the image, such as rainy days often have foggy conditions, in order to prompt the generating network to find the connection between scenes.

[0061] After inputting the degraded sample image into the prompt generation network to be trained, the degraded sample image is then input into the perceptron to be trained, and the prompt information to be learned is obtained from the output of the perceptron to be trained.

[0062] It should be noted that this embodiment does not impose any restrictions on the execution order of steps S21 to S23. For example, steps S21 to S23 can be executed in any order or at the same time.

[0063] In conjunction with the above embodiments, in one implementation, the present invention also provides an image restoration method for multiple types of interference, in which the training step of the pre-trained content-aware processor may include steps S31 to S34:

[0064] Step S31: Input the clean image corresponding to the sample degraded image into the pre-trained graph-text multimodal network to obtain the clean image description text corresponding to the sample degraded image.

[0065] Since the sample datasets used to train image restoration models often lack large-scale text-image pairs, this embodiment generates text information for clean images by inputting these images into a pre-trained graph-text multimodal network to generate content descriptions.

[0066] Specifically, in this embodiment, the clean image corresponding to the degraded image is input into a pre-trained image-text multimodal network to obtain the clean image description text corresponding to the degraded image output by the pre-trained image-text multimodal network. The image-text multimodal network may include a BLIP network, which is a multimodal encoder-decoder architecture capable of generating corresponding text based on an image. In this embodiment, the degraded and clean images used to train the content-aware processor are the same as those used to train the image restoration model, belonging to the same sample dataset.

[0067] Step S32: Input the clean image description text into the text encoder in the pre-trained visual text network to obtain the first text feature.

[0068] Due to the limited size of image restoration datasets, especially the small number of degraded image-clean image descriptions, directly training a network using this data is difficult to converge. To address this issue, this embodiment uses a pre-trained visual-text network. This pre-trained visual-text network relies on a large amount of image-text pairs for pre-training, achieving alignment between image and text features, and already contains rich image and text encoding information. This pre-trained visual-text network includes a text encoder and an image encoder. During the training of the content-aware perceptron, the text encoder of this pre-trained visual-text network is frozen, and only the image encoder of the pre-trained visual-text network is retrained.

[0069] In this embodiment, after obtaining the clean image description text corresponding to the sample degraded image, the clean image description text can be input into the text encoder in the pre-trained visual text network to obtain the first text feature output by the text encoder in the pre-trained visual text network. The first text feature is the text feature output by the text encoder when training the content-aware device.

[0070] Step S33: Input the degraded image of the sample into the image encoder in the pre-trained visual text network to obtain the first image features.

[0071] In this embodiment, the degraded sample image is also input into the image encoder in the pre-trained visual-text network to obtain the first image feature output by the image encoder in the pre-trained visual-text network. This first image feature is the image feature output by the image encoder during content-aware processor training. Specifically, during the training of the image encoder, the weights (network parameters) of the image encoder obtained from the pre-trained visual-text network are used as the initial weights (network parameters) of the image encoder.

[0072] Step S34: Compare and learn the first text features and the first image features to obtain a first contrastive learning loss. Update the network parameters of the image encoder based on the first contrastive learning loss until the updated first image encoder is obtained. Use the updated first image encoder as the pre-trained content awareness device.

[0073] In this embodiment, the content awareness uses a contrastive learning approach for training: after obtaining the first text features and the first image features, the first text features and the first image features are compared and learned to obtain the first contrastive learning loss. Based on the first contrastive learning loss, the network parameters of the image encoder in the pre-trained visual text network are updated until the updated first image encoder is obtained. The updated first image encoder is then used as the trained content awareness, i.e., the pre-trained content awareness.

[0074] In other words, this embodiment uses input sample degraded images - clean images to describe text, in order to train the image encoder in the pre-trained visual text network to obtain an image encoder for determining image content, which serves as a pre-trained content perceiver.

[0075] In one alternative implementation, the visual text network can be a CLIP network. CLIP (Contrastive Language-Image Pretraining) is a model trained using a contrastive learning method that can map images and text to the same latent space. By jointly training image and text data, it enables the model to understand the relationship between text and images and is widely used in tasks such as image generation, search, and classification.

[0076] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the training of a content-aware processor according to an embodiment of the present invention. Figure 2 In this context, the image-text multimodal network is a BLIP network, and the visual-text network can be a CLIP network. Figure 2 The left side shows the clean image corresponding to the degraded image. The clean image is input into the pre-trained BLIP network to obtain the clean image description text corresponding to the degraded image. Then, the clean image description text is input into the text encoder of the pre-trained CLIP network to obtain the first text feature. Figure 2 The image on the right is a degraded sample image. This degraded sample image is input into the pre-trained CLIP network's image encoder to obtain the first text feature. Based on the first text feature and the first image feature, comparative learning is performed to update the network parameters of the pre-trained CLIP network's image encoder until a well-trained CLIP network image encoder is obtained, which serves as the pre-trained content-aware sensor.

[0077] In conjunction with the above embodiments, in one implementation, the present invention also provides an image restoration method for multiple types of interference. In this embodiment, the training steps of the pre-trained style perceptron may include steps S41 to S44:

[0078] Step S41: Determine the dataset corresponding to the sample degraded image, and determine the scene description text corresponding to the dataset.

[0079] For generating style-related prompt text, this embodiment adopts a scenario description creation approach based on dataset context: The dataset corresponding to the sample degraded image is determined, and the scene description text corresponding to that dataset is determined based on the dataset corresponding to the sample degraded image; that is, the scene description text corresponding to the sample degraded image. In this embodiment, the sample degraded image used to train the style perceptron is the same as the sample degraded image used to train the image restoration model, belonging to the same sample dataset. The dataset corresponding to this sample degraded image is the sample dataset. It can be understood that the scene description text is the degradation type description text corresponding to the sample degraded image, such as the description text for degradation types like fog, noise, and occlusion.

[0080] In this embodiment, each dataset corresponds to a scene description. For example, in a fog dataset, the description is "A photo taken on a hazy day". Example scene description texts for each scene dataset are shown in Table 1:

[0081] Table 1. Correspondence between Scenes and Scene Description Text

[0082]

[0083] Step S42: Input the scene description text into the text encoder in the pre-trained visual text network to obtain the second text feature.

[0084] This embodiment trains the same system as the content perceptron, using a pre-trained visual-text network. This pre-trained visual-text network relies on a large amount of image-text pairs for pre-training, achieving alignment between image and text features, and already contains rich image and text encoding information. This pre-trained visual-text network includes a text encoder and an image encoder. When training the style perceptron, the text encoder of this pre-trained visual-text network is frozen, and only the image encoder of the pre-trained visual-text network is retrained.

[0085] In this embodiment, after obtaining the scene description text corresponding to the sample degraded image, the scene description text can be input into the text encoder in the pre-trained visual text network to obtain the second text feature output by the text encoder in the pre-trained visual text network. The second text feature is the text feature output by the text encoder when training the style perceptron.

[0086] Step S43: Input the degraded sample image into the image encoder in the pre-trained visual text network to obtain the second image features.

[0087] In this embodiment, the degraded sample image is also input into the image encoder in the pre-trained visual-text network to obtain the second image feature output by the image encoder in the pre-trained visual-text network. This second image feature is the image feature output by the image encoder during style perceptron training. Specifically, during the training of the image encoder, the weights (network parameters) of the image encoder obtained from the pre-trained visual-text network are used as the initial weights (network parameters) of the image encoder.

[0088] Step S44: Compare and learn the second text features and the second image features to obtain the second contrastive learning loss. Update the network parameters of the image encoder based on the second contrastive learning loss until the updated second image encoder is obtained. Use the updated second image encoder as the pre-trained style perceptron.

[0089] In this embodiment, the style perceptron is also trained using a contrastive learning approach: after obtaining the second text features and the second image features, the second text features and the second image features are compared and learned to obtain a second contrastive learning loss. Based on the second contrastive learning loss, the network parameters of the image encoder in the pre-trained visual text network are updated until the updated second image encoder is obtained. The updated second image encoder is then used as the trained style perceptron, i.e., the pre-trained style perceptron.

[0090] In other words, this embodiment trains an image encoder in a pre-trained visual-text network by inputting a degraded image sample and scene description text, resulting in an image encoder used to determine the type of image degradation, which serves as a pre-trained style perceptron. In an optional implementation, the visual-text network can be a CLIP (Contrastive Language-Image Pretraining) network.

[0091] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the training of a style perceptron according to an embodiment of the present invention. Figure 3 In this context, the image-text multimodal network is a BLIP network, and the visual-text network can be a CLIP network. Figure 3 The left side shows the scene description text corresponding to the sample degraded image, which belongs to datasets of different dashed line types (i.e., squares of different dashed line types in the figure). Then, the scene description text is input into the text encoder of the pre-trained CLIP network to obtain the second text feature. Figure 3 The image on the right is a degraded sample image. This degraded sample image is input into the pre-trained CLIP network's image encoder to obtain the second text feature. Based on the second text feature and the second image feature, comparative learning is performed to update the network parameters of the pre-trained CLIP network's image encoder until a well-trained CLIP network image encoder is obtained, which serves as the pre-trained style perceptron.

[0092] In conjunction with the above embodiments, in one implementation, the present invention also provides an image restoration method for multiple types of interference. In this embodiment, the perceptron to be trained includes at least: a pre-trained visual Transformer structure, a first multilayer perceptron to be trained, a second multilayer perceptron to be trained, a self-attention layer to be trained, and a cross-attention layer to be trained; step S23 may specifically include steps S51 to S55:

[0093] Step S51: Input the degraded sample image into the pre-trained visual Transformer structure to obtain multiple image feature vectors.

[0094] In this embodiment, after the degraded sample image is input into the perceptron to be trained, it is first fed into a pre-trained Vision Transformer (ViT) structure to obtain multiple image feature vectors. During this stage, the pre-trained Vision Transformer structure remains frozen, and no gradient propagation or parameter updates occur. For example, the pre-trained Vision Transformer structure in this embodiment can be a ViT-b16 model pre-trained on MAGENET1K.

[0095] Step S52: Input the multiple image feature vectors into the first multilayer perceptron to be trained for processing to obtain multiple first feature vectors, and perform self-attention operation between the multiple first feature vectors through the self-attention layer to be trained to obtain self-attention results.

[0096] In this embodiment, multiple image feature vectors output by the pre-trained visual Transformer structure are input into the first multilayer perceptron to be trained for processing to obtain multiple first feature vectors. Then, through the self-attention layer to be trained, a self-attention operation is performed between these multiple first feature vectors to obtain the self-attention result output by the self-attention layer to be trained, and the parameters of the first multilayer perceptron are updated in the process.

[0097] Step S53: Initialize multiple learning vectors, and perform cross-attention operation between the multiple learning vectors and the self-attention result through the cross-attention layer to be trained, to obtain the cross-attention result.

[0098] In this embodiment, before the training of the image restoration model to be trained begins, multiple learning vectors are randomly initialized. Then, the multiple learning vectors and the obtained self-attention results are input into the cross-attention layer to be trained. Through the cross-attention layer to be trained, the multiple learning vectors and the self-attention results are subjected to cross-attention operation to obtain the cross-attention result output by the cross-attention layer to be trained.

[0099] Step S54: Input the cross-attention result into the second multilayer perceptron to be trained for processing to obtain the updated learning vector.

[0100] In this embodiment, after obtaining the cross-attention result, the cross-attention result is input into the second multilayer perceptron to be trained for processing, so as to obtain the updated learning vector output by the second multilayer perceptron to be trained.

[0101] It should be noted that in the subsequent model training process, the updated learning vector is used as the initial learning vector and cross-attention is performed with the subsequent self-attention result to obtain the subsequent cross-attention result. Also, the first and second multilayer perceptrons in this embodiment are different multilayer perceptrons, each corresponding to different network parameters and not sharing weights. A multilayer perceptron (MLP) is also called an artificial neural network. During the training of the learning perceptron, the parameters of the first, second, self-attention, and cross-attention layers are updated, and each of these layers has independent parameters to be trained, without sharing weights.

[0102] Step S55: Based on the updated learning vector and the cross-attention result, obtain the learning prompt information.

[0103] In this embodiment, after obtaining the updated learning vector, the learning prompt information output by the perceptron to be trained can be obtained based on the updated learning vector and the cross-attention result. In an optional example, the learning prompt information can be obtained by performing a residual connection based on the updated learning vector and the cross-attention result.

[0104] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a perceptron to be learned according to an embodiment of the present invention. Figure 4 In the middle, the left side shows the degraded sample image. The degraded sample image is first processed through a pre-trained visual Transformer structure (i.e., Figure 4 The ViT layer in the first multilayer perceptron (MLP) is used to process the image features, generating 196 image feature vectors, each with a dimension of 768. During this stage, the ViT layer remains frozen, and no gradient propagation or parameter updates occur. These 196 image feature vectors are then processed by the first multilayer perceptron (MLP). Figure 4 The MLP in the top row (see the previous row) yields 32 first feature vectors. Self-attention is performed among these 32 first feature vectors to obtain the self-attention result, and the parameters within the first multilayer perceptron are updated during this process. Before training begins, 32 learnable vectors are randomly initialized, each with a dimension of 768. These learnable vectors undergo cross-attention with the self-attention result to obtain the cross-attention result. Then, the second multilayer perceptron (… Figure 4The MLP (Multi-Level Processing) in the lower row processes the cross-attention results, generating a set of 32 updated learnable vectors, each with 768 dimensions. Finally, element-wise addition is performed on these updated learnable vectors and the cross-attention results to obtain a 768*1 vector, which serves as the final learning cue.

[0105] Using prompts to guide another network in generating results has become a common practice. However, there is no universal formula for combining prompts with network feature values. Generally, the method for combining prompts with the original network's feature values ​​depends on the amount of information contained in the prompt. When the prompt is a scalar with limited information, such as using time T to guide a diffusion model, T is typically embedded directly with the network feature values ​​via multiplication. When the prompt contains more information, such as using the output of an object detection network, an embedding layer is used to align the prompt with the shape of the network feature map. After alignment, a connection operation is used to combine the prompt and the network feature values. When using feature values ​​from other backbone networks, encoders, or other neural networks as encoded information, the information content is the richest. In this case, an embedding layer can be used to align the prompt with the shape of the network's feature map, and then a self-attention layer can be used to connect them.

[0106] The embodiments of this invention use three types of prompt information, each with different information densities: content prompt information contains the most information, prompt information to be learned is second, and style prompt information contains the least information. How to combine these three types of prompt information with the feature mapping in the image restoration network becomes a problem. To solve this problem, in conjunction with the above embodiments, in one implementation, this invention also provides an image restoration method for multiple types of interference. In this method, the prompt generation network to be trained further includes: a prompt encoder to be trained, the prompt encoder to be trained including at least: three third-layer perceptrons to be trained and three fourth-layer perceptrons to be trained; and step S13 above specifically includes steps S61 to S63:

[0107] Step S61: Input the content prompt information, the style prompt information, and the prompt information to be learned into their respective third multilayer perceptrons to be trained for processing and then perform linear transformation to obtain the first prompt feature, the second prompt feature, and the third prompt feature corresponding to the content prompt information, the style prompt information, and the prompt information to be learned, respectively.

[0108] In this embodiment, three third-level multilayer perceptrons and three fourth-level multilayer perceptrons are used to process content prompts, style prompts, and prompts to be learned, respectively. That is, each type of prompt corresponds to one third-level multilayer perceptron and one fourth-level multilayer perceptron. In this embodiment, the third-level and fourth-level multilayer perceptrons are different multilayer perceptrons, and the six multilayer perceptrons in the prompt encoder to be trained correspond to different network parameters and do not share weights.

[0109] In this embodiment, the content prompt information, style prompt information, and learning prompt information are respectively input into their respective third multilayer perceptrons to be trained for processing and then linear transformation is performed to obtain the first prompt feature, the second prompt feature, and the third prompt feature corresponding to the content prompt information, style prompt information, and learning prompt information, respectively.

[0110] Step S62: Input the first prompt feature, the second prompt feature and the third prompt feature into their respective fourth multilayer perceptrons to be trained for processing, to obtain the fourth prompt feature, the fifth prompt feature and the sixth prompt feature corresponding to the content prompt information, the style prompt information and the prompt information to be learned, respectively.

[0111] In this embodiment, the first prompt feature, the second prompt feature, and the third prompt feature are then input into their respective corresponding fourth multilayer perceptrons to be trained for processing, so as to obtain the fourth prompt feature, the fifth prompt feature, and the sixth prompt feature corresponding to the content prompt information, the style prompt information, and the prompt information to be learned, respectively.

[0112] Step S63: After connecting the fourth cue feature, the fifth cue feature, and the sixth cue feature, the target cue information is obtained by sequentially processing through self-attention and feedforward networks, and then output by the cue encoder to be trained.

[0113] In this embodiment, after obtaining the fourth, fifth, and sixth cue features, the fourth, fifth, and sixth cue features are connected to obtain the connection features. Then, the connection features are processed through self-attention to obtain the seventh cue feature. Finally, the seventh cue feature is processed through a feedforward neural network (FFN) to obtain the target cue information output by the cue encoder to be trained.

[0114] In this embodiment, a prompt encoder is proposed to effectively integrate prompt information. This is an effective fusion module for multiple prompts, which can integrate multiple prompts and be used by downstream modules.

[0115] In one embodiment, such as Figure 5 As shown, Figure 5This is a schematic diagram of the structure of a prompt encoder according to an embodiment of the present invention. Figure 5 In this process, each prompt (style prompt, content prompt, and learning prompt) first passes through a third-level multilayer perceptron (MLP). Figure 5 The second-to-last row of the MLP is processed, then a linear transformation is performed, and then it is passed through the fourth multilayer perceptron (MLP). Figure 5 The fourth-to-last row of the MLP (Multi-Level Processing) is used to reduce the dimensions from the original 768 to 256. Then, the three 256-dimensional cues are concatenated to form a new 768-dimensional cue. This final 768-dimensional cue is then processed sequentially through a self-attention and feedforward network to obtain the final target cue information, which is then input into the image restoration network.

[0116] In related technologies, image restoration methods have shown good performance on some datasets, but they have not yet overcome all dataset challenges. When faced with more complex datasets, their performance is often unsatisfactory and they cannot effectively restore damaged images. To address this problem, in one embodiment, based on the above embodiments, the present invention also provides an image restoration method for multiple types of interference. In this method, the image restoration network to be trained is a diffusion model to be trained; and step S14 above specifically includes steps S71 and S72:

[0117] Step S71: The degraded sample image is sequentially forward-diffused and backward-diffused using the image restoration network to be trained.

[0118] In this embodiment, the image restoration network to be trained first performs forward diffusion on the sample degraded image to add noise to the sample degraded image, and then performs backward diffusion on the noisy sample image to remove noise from the noisy sample degraded image, so as to restore the image of the sample degraded image.

[0119] Step S72: During the backdiffusion process of the backdiffusion, the target cue information is combined with the feature layer in the backdiffusion process through a cross-attention mechanism to obtain the sample restoration image.

[0120] In this embodiment, during the backdiffusion process of the image restoration network to be trained, a cross-attention mechanism is used to combine the target prompt information output by the prompt generation network to be trained with the feature layer of the image restoration network during the backdiffusion process, thereby obtaining the sample restored image output by the image restoration network to be trained. In an optional embodiment, the diffusion model to be trained in this embodiment can be obtained by improving upon a Stable Diffusion-based structure.

[0121] To address the issue of insufficient absolute performance in related technologies when dealing with complex datasets, this embodiment utilizes a diffusion model to solve this problem. The diffusion model has excellent capabilities in the field of image generation and, after modification, is very suitable for solving this problem.

[0122] In one embodiment, to more effectively address the problem of restoring degraded images in multi-task scenarios, this embodiment proposes an image restoration model based on a cueing and diffusion model, constituting a new paradigm in the field of image generation based on pre-trained large-scale visual text models, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of the overall network structure of an image restoration model according to an embodiment of the present invention. Figure 6 In this model, the image restoration model consists of an Image Restoration Part (IRP) and a Prompt Generation Part (PGP). The IRP handles the image restoration task, while the PGP generates prompts.

[0123] The cue generation network comprises four components: a content perceptron, a style perceptron, a learning perceptron, and a cue encoder. When a degraded image is input into the network, the three perceptrons in the cue generation network generate corresponding cues: content cues, style cues, and learning cues. These cues are then uniformly encoded by the cue encoder to produce a unified cue used by the image restoration network. Finally, the unified cue is combined with the feature layers in the image restoration network to achieve the restoration of a clear image. Experimental results show that this method outperforms many mainstream methods under both single and complex interference conditions. Thus, addressing the problem that most related technologies can only solve a small number of images, this embodiment uses a multi-cue information approach to solve this problem; facing different degraded images, the image restoration model of this embodiment can solve more types of network degradation problems at once, making it a more adaptable image restoration model.

[0124] In conjunction with the above embodiments, in one implementation, the present invention also provides an image restoration method for multiple types of interference. In this method, step S15 specifically includes the following steps:

[0125] Based on the restored sample image and the clean sample image corresponding to the degraded sample image, the image restoration network to be trained, the perceptron to be trained, and the cue encoder to be trained are trained until a trained image restoration network, a trained perceptron to be trained, and a trained cue encoder are obtained; based on the trained image restoration network, the pre-trained content perceptron, the pre-trained style perceptron, the trained perceptron to be trained, and the trained cue encoder, the trained image restoration model is obtained.

[0126] In an optional embodiment, the image restoration method for various types of interference shown in any of the foregoing embodiments can be deployed as a preprocessing part of a vision system to assist in completing subsequent vision tasks. It is applicable to a variety of different vision application scenarios, including but not limited to: autonomous vehicles, security monitoring systems, industrial automation inspection, and medical image analysis. For example, it has broad application prospects in autonomous driving, where accurate road sign recognition and obstacle detection are crucial. By deploying an image restoration preprocessing module (i.e., the image restoration model proposed in this invention), vision systems in these fields can obtain more accurate data support, thereby making more informed decisions.

[0127] Image restoration is often integrated into the overall visual network as part of image preprocessing. For example, it can serve as an upstream step in object detection and tracking tasks, assisting them in achieving better detection and tracking. One practical scenario is its use as a preprocessing module in real-time detection tasks, reading in damaged images and restoring them. The system deployment is divided into training and inference phases. The flowchart for the training phase is shown below. Figure 7 As shown, Figure 7 This is a flowchart illustrating the training process of an image restoration model according to an embodiment of the present invention. The training process is as follows:

[0128] Dataset Construction: For the One-By-One image dehazing task (a way to evaluate the network's restoration capabilities by training a separate model for each specific restoration task), the RESIDE dataset is used. For the One-By-One image deraining task, the Rain100L dataset is used, which contains 200 pairs of clean rain images for training and 100 pairs for testing. For the One-By-One image denoising task, the BSD400 dataset is used. BSD400 contains 400 training images. In this embodiment, corresponding noisy versions of these images are generated by adding different levels of Gaussian noise σ∈(15,25,50), and then the denoising task is evaluated on the BSD68 dataset. In the All-In-One setting (a way to evaluate the network's restoration capabilities by training a single model to restore images of multiple degradation types), a single model is trained on a combined set of the above training datasets and tested on multiple restoration tasks. To further illustrate the excellent performance of the network proposed in this embodiment, a deblurring task is also added using the GoPro dataset. Low-light recovery task, using the LOL dataset.

[0129] Read image data: Read the corresponding image data and its real label data from the training dataset. The batch size of the training image data loaded per GPU is 1.

[0130] Training parameters were configured: The model was built using PyTorch 2.0 and trained on a system with four NVIDIA A100 GPUs. Specific training parameters are shown in Table 2. First, the content-aware and style-aware processors were trained on the aforementioned dataset. Image-text pairs were generated using the BLIP-ViT-L network. The CLIP network, implemented based on OpenCLIP, and the ViT-B-32 model trained on the laion2b_s34b_b79k pre-training data were used. During training, CLIP completely froze the text encoder and text embeddings, training only the image encoder. The batch size was 256, and the learning rate was 1e-4. Subsequently, the content-aware and style-aware processors were completely frozen during training.

[0131] Model output prediction result: The image restoration result obtained by the model inference is the same RGB image as the input degraded image.

[0132] Backpropagation after loss calculation: During training, the loss between the ground truth label image corresponding to the input degraded image and the output restored image is calculated. L1 loss is used to calculate the difference between the model output and the expectation.

[0133] Save training results: After training, the final image restoration model is obtained and saved offline, such as in HDF5, TensorFlow SavedModel, ONNX, etc., and exported through the corresponding API or tools. The model can also be validated on a validation set.

[0134] Table 2 System Hardware Configuration and Software Version Information

[0135]

[0136] The flowchart of the reasoning stage is as follows Figure 8 As shown, Figure 8 This is a flowchart illustrating the inference process of an image restoration model according to an embodiment of the present invention. The inference process is as follows:

[0137] Loading Models: Load offline image restoration models using tools such as ONNX and TensorRT. Using tools like ONNX and TensorRT can significantly improve the efficiency of model loading and inference, and support cross-platform flexibility.

[0138] Data Reading: The program reads and decodes images from the camera in real time. During data reading, it initializes the camera connection, captures image frames in real time, and decodes them, converting them into analyzable digital information for subsequent image processing and analysis algorithms. The camera is initialized using OpenCV or a similar computer vision library. Then, the camera's API is used to set parameters such as frame rate and resolution, and real-time image stream capture begins. After each frame is captured, the built-in decoder converts the raw image data (e.g., H.264 encoded) into matrix data in RGB or grayscale format for subsequent processing. This process is typically executed continuously in a loop to ensure the continuity and real-time nature of the image data.

[0139] The read image is fed into the model for inference: the output of the image restoration model is the restored image. The restored image is then fed into the next step to complete subsequent visual tasks.

[0140] Furthermore, in another embodiment, the image restoration model for multiple types of interference proposed in the above embodiments can be deployed and implemented as part of image processing software. Whether it's professional graphic design software or lightweight applications for general users, it can provide users with a powerful tool to improve the quality of photographs and other visual content. Especially in fields such as photographic post-processing, historical document preservation, and medical image analysis, image restoration functionality will play an irreplaceable role. At this point, the deployment and inference phase of the image restoration model is as follows:

[0141] When loading a model, there are some specific considerations when deploying an image restoration model as part of image processing software: the trained model and its inference logic can be seamlessly integrated into the existing image processing software framework to ensure good collaboration with other functional modules.

[0142] Data Loading: Real-time Data: Image frames are read from the camera, preprocessed (e.g., scaling, normalization), and then fed into the model. Custom Data: User-uploaded image or video files are automatically parsed and converted into a format acceptable to the model, ensuring the same preprocessing standards as real-time data.

[0143] Save results: Ensure that the inference results are presented to users in an intuitive way, and improve the overall user experience by displaying the detection results or classification tags through a graphical interface.

[0144] By implementing the above steps, the inference module can be effectively integrated into the image processing software, maintaining consistency with the training phase while fully considering the needs of different data sources and real-world application scenarios. This deployment method enhances the software's functionality and flexibility, making it more adaptable to diverse user needs.

[0145] In one embodiment, to verify the effectiveness of the image restoration method for various types of interference proposed in this invention, this embodiment strictly follows previous research and conducts experiments in two different environments: (1) an All-In-One task and (2) a One-By-One task. In the All-In-One setting, a single model is trained to restore images with multiple degradation types. In the One-By-One task setting, a separate model is trained for each specific restoration task.

[0146] I. For All-In-One Results: Three Types of Degradation

[0147] To ensure consistency with mainstream methods, three hybrid image restoration tasks—dehazing, dehazing, and denoising—were first tested. For comparison, this embodiment selected networks tested on the same tasks, including PromptIR, AirNet, and DL. Quantitative analysis results are shown in Table 3.

[0148] Table 3 compares the results with mainstream All-In-One networks on three tasks.

[0149]

[0150] As shown in Table 3, in this scenario across the three tasks, the average performance of multiple recovery tasks was improved, with varying degrees of improvement observed in specific tasks. Compared to PromptIR, the method proposed in this invention (Ours) improved the PSNR metric for the defogging task by 0.11 dB. Figure 9 As shown, Figure 9 This is a comparison of the visualization effects under three task scenarios proposed in an embodiment of the present invention. From Figure 9 The image restoration effect shown in the visualization can be seen, where GT represents the labeled image. The method proposed in this invention effectively eliminates the blur in the image and produces cleaner results than PromptIR.

[0151] II. Five Types of Degradation in All-In-One Results

[0152] To further illustrate the significant advantages of the proposed method in handling complex scenes, in addition to the three tasks mentioned above, more mixed task scenarios were implemented, including five scenarios: defogging, denoising, deblurring, and low-light image enhancement. This scenario setup is consistent with the contrast network, which also has five scenarios. Therefore, five datasets were used to train the network. These datasets include the datasets for the three tasks mentioned above, as well as other datasets: the GoPro dataset for motion deblurring and the LOL dataset for low-light image enhancement.

[0153] The results are shown in Table 4. Compared with networks that perform the same image restoration tasks for the five scenes described above, the method of this invention (Ours in Table 4) still maintains a leading position. Compared with IDR, the method of this invention leads by 1.08 dB on the average restoration task across the five scenes. It also shows varying degrees of leadership in each specific task. Furthermore, it can be observed that the use of a diffusion model provides a significant advantage in the denoising task, with an advantage of 2.73 dB. Figure 10 As shown, Figure 10 This is a comparison chart of the visualization effects under five task scenarios proposed in an embodiment of the present invention. Figure 10 The image restoration model proposed in this invention demonstrates its ability to restore damaged images in five mixed scenarios, where GT represents the labeled image.

[0154] Table 4. Analysis comparing it with mainstream All-In-One networks on five tasks.

[0155]

[0156] III. One-By-One Results

[0157] This embodiment also evaluates the performance of the proposed image restoration model in a one-to-one setting, where a separate model is trained for each different restoration task. A series of experiments show that the proposed method achieves good results in one-to-one scenarios. For the dehazing task, the proposed method was trained on the RESIDE OTS dataset and tested on the OTS-outdoor dataset. The experimental results are shown in Table 5. The results show that the PSNR of the proposed method is improved by 0.25 dB compared to PromptIR. It also has certain advantages compared to traditional single-task dehazing networks. For the rain removal task, the proposed method was trained and tested on the Rain100L dataset. It was compared with PromptIR and other networks. The experimental results are shown in Table 6. It can be observed that the proposed method has significant advantages. In this task, the proposed method outperforms PromptIR by 0.24 dB and 0.004 dB in PSNR and SSIM, respectively.

[0158] Table 5 shows the One-By-One results on the SOTS-Outdoor dataset.

[0159]

[0160] Table 6 shows the One-By-One results on the RAIN100L dataset.

[0161]

[0162] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0163] Based on the same inventive concept, one embodiment of the present invention provides an image restoration device for various types of interference. (Reference) Figure 11 , Figure 11 This is a structural block diagram of an image restoration device for various types of interference provided in an embodiment of the present invention. For example... Figure 11 As shown, the image restoration device for various types of interference in this embodiment may include:

[0164] The image input module is used to input the degraded sample images into the image restoration network and the prompt generation network to be trained in the image restoration model to be trained, respectively.

[0165] The prompt generation module is used to process the sample degraded image through the prompt generation network to be trained, and generate content prompt information, style prompt information and learning prompt information corresponding to the sample degraded image respectively; the content prompt information represents the image content prompt corresponding to the sample degraded image, the style prompt information represents the degradation type prompt corresponding to the sample degraded image, and the learning prompt information represents the semantic prompt information corresponding to the sample degraded image, which is of a higher dimension than the content prompt information and the style prompt information;

[0166] The prompt encoding module is used to fuse and encode the content prompt information, the style prompt information, and the prompt information to be learned through the prompt generation network to be trained, so as to obtain the target prompt information;

[0167] An image output module is used to combine the target prompt information with the feature layer of the image restoration network during the image restoration process of the image restoration network to be trained, so as to obtain the sample restoration image output by the image restoration network.

[0168] The model training module is used to train the image restoration model to be trained based on the sample restored image and the clean image corresponding to the sample degraded image until a trained image restoration model is obtained.

[0169] The image restoration module is used to input the image to be restored into the trained image restoration model to obtain the restored image.

[0170] Optionally, the prompt generation network to be trained includes at least: a pre-trained content perceptron, a pre-trained style perceptron, and a perceptron to be trained; the pre-trained content perceptron and the pre-trained style perceptron are obtained by retraining the image encoder in the pre-trained visual text network.

[0171] The prompt generation module includes:

[0172] The first prompt generation module is used to input the sample degraded image into the pre-trained content perceptron to obtain the content prompt information;

[0173] The second prompt generation module is used to input the sample degradation image into the pre-trained style perceptron to obtain the style prompt information;

[0174] The third prompt generation module is used to input the sample degradation image into the perceptron to be trained to obtain the prompt information to be learned.

[0175] Optionally, the image restoration device further includes: a first training module for training the pre-trained content-aware processor, the first training module comprising:

[0176] The first text generation module is used to input the clean image of the sample corresponding to the sample degraded image into a pre-trained graph-text multimodal network to obtain the clean image description text corresponding to the sample degraded image.

[0177] The first text processing module is used to input the clean image description text into the text encoder in the pre-trained visual text network to obtain the first text features;

[0178] The first image processing module is used to input the degraded sample image into the image encoder in the pre-trained visual text network to obtain the first image features;

[0179] The first perceptron training module is used to compare and learn the first text features and the first image features to obtain a first contrastive learning loss, update the network parameters of the image encoder based on the first contrastive learning loss, until the updated first image encoder is obtained, and use the updated first image encoder as the pre-trained content perceptron.

[0180] Optionally, the image restoration device further includes: a second training module for training the pre-trained style perceptron, the second training module comprising:

[0181] The second text generation module is used to determine the dataset corresponding to the sample degraded image and to determine the scene description text corresponding to the dataset.

[0182] The second text processing module is used to input the scene description text into the text encoder in the pre-trained visual text network to obtain the second text features;

[0183] The second image processing module is used to input the degraded sample image into the image encoder in the pre-trained visual text network to obtain the second image features;

[0184] The second perceptron training module is used to compare and learn the second text features and the second image features to obtain a second contrastive learning loss. Based on the second contrastive learning loss, the network parameters of the image encoder are updated until the updated second image encoder is obtained. The updated second image encoder is used as the pre-trained style perceptron.

[0185] Optionally, the perceptron to be trained includes at least: a pre-trained visual Transformer structure, a first multilayer perceptron to be trained, a second multilayer perceptron to be trained, a self-attention layer to be trained, and a cross-attention layer to be trained.

[0186] The third prompt generation module includes:

[0187] The first processing module is used to input the degraded image of the sample into the pre-trained visual Transformer structure to obtain multiple image feature vectors;

[0188] The second processing module is used to input the multiple image feature vectors into the first multilayer perceptron to be trained for processing to obtain multiple first feature vectors, and to perform self-attention operation between the multiple first feature vectors through the self-attention layer to be trained to obtain a self-attention result.

[0189] The third processing module is used to initialize multiple learning vectors, and through the cross-attention layer to be trained, to perform a cross-attention operation between the multiple learning vectors and the self-attention result to obtain the cross-attention result;

[0190] The fourth processing module is used to input the cross-attention result into the second multilayer perceptron to be trained for processing, so as to obtain the updated learning vector;

[0191] The fifth processing module is used to obtain the learning prompt information based on the updated learning vector and the cross-attention result.

[0192] Optionally, the prompt generation network to be trained further includes: a prompt encoder to be trained, wherein the prompt encoder to be trained includes at least: three third-layer perceptrons and three fourth-layer perceptrons to be trained.

[0193] The prompt encoding module includes:

[0194] The sixth processing module is used to input the content prompt information, the style prompt information, and the prompt information to be learned into their respective corresponding third multilayer perceptrons to be trained for processing and then perform linear transformation to obtain the first prompt feature, the second prompt feature, and the third prompt feature corresponding to the content prompt information, the style prompt information, and the prompt information to be learned, respectively.

[0195] The seventh processing module is used to input the first prompt feature, the second prompt feature and the third prompt feature into their respective corresponding fourth multilayer perceptrons to be trained for processing, so as to obtain the fourth prompt feature, the fifth prompt feature and the sixth prompt feature corresponding to the content prompt information, the style prompt information and the prompt information to be learned, respectively;

[0196] The eighth processing module is used to connect the fourth, fifth, and sixth prompt features, and then process them sequentially through a self-attention and feedforward network to obtain the target prompt information output by the prompt encoder to be trained.

[0197] Optionally, the image restoration network to be trained is a diffusion model to be trained; the image output module includes:

[0198] The diffusion processing module is used to sequentially perform forward diffusion and reverse diffusion on the sample degraded image through the image restoration network to be trained;

[0199] The prompting combination module is used to combine the target prompting information with the feature layer in the backdiffusion process through a cross-attention mechanism during the backdiffusion process to obtain the sample restoration image.

[0200] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the image restoration method for multiple types of interference as described in any of the above embodiments of the present invention.

[0201] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the image restoration method for multiple interferences described in any of the above embodiments of the present invention.

[0202] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0203] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0204] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0205] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0206] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0207] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0208] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0209] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0210] The foregoing has provided a detailed description of an image restoration method, apparatus, device, and medium for various types of interference provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image restoration method for multiple types of interference, characterized in that, The method includes: The degraded images of the samples are respectively input into the image restoration network to be trained and the prompt generation network to be trained in the image restoration model to be trained; the prompt generation network to be trained includes at least: a pre-trained content perceptron, a pre-trained style perceptron and a perceptron to be trained and learned; the pre-trained content perceptron and the pre-trained style perceptron are obtained by retraining the image encoder in the pre-trained visual text network. The degraded sample image is processed by the training-to-be-trained prompt generation network to generate content prompt information, style prompt information, and learning prompt information corresponding to the degraded sample image, including: inputting the degraded sample image into the pre-trained content perceptron to obtain the content prompt information; inputting the degraded sample image into the pre-trained style perceptron to obtain the style prompt information; and inputting the degraded sample image into the training-to-be-trained perceptron to obtain the learning prompt information. The content prompt information represents the image content prompt corresponding to the degraded sample image, the style prompt information represents the degradation type prompt corresponding to the degraded sample image, and the learning prompt information represents the semantic prompt information corresponding to the degraded sample image, which is of a higher dimension than the content prompt information and the style prompt information. The target prompt information is obtained by fusing and encoding the content prompt information, the style prompt information, and the prompt information to be learned through the prompt generation network to be trained; During the process of the image restoration network to be trained restoring the sample degraded image, the target prompt information is combined with the feature layer of the image restoration network to obtain the sample restored image output by the image restoration network. Based on the restored sample image and the clean sample image corresponding to the degraded sample image, the image restoration model to be trained is trained until a trained image restoration model is obtained. The image to be restored is input into the trained image restoration model to obtain the restored image; The training steps for the pre-trained content awareness include: Input the clean image corresponding to the degraded image into a pre-trained graph-text multimodal network to obtain the clean image description text corresponding to the degraded image. The clean image description text is input into the text encoder in a pre-trained visual text network to obtain the first text feature; The degraded image of the sample is input into the image encoder in the pre-trained visual-text network to obtain the first image features; The first text features and the first image features are compared and learned to obtain a first contrastive learning loss. The network parameters of the image encoder are updated based on the first contrastive learning loss until the updated first image encoder is obtained. The updated first image encoder is used as the pre-trained content awareness device.

2. The image restoration method for multiple types of interference according to claim 1, characterized in that, The training steps for the pre-trained style perceptron include: Determine the dataset corresponding to the degraded sample image, and determine the scene description text corresponding to the dataset; The scene description text is input into the text encoder in a pre-trained visual text network to obtain the second text feature; The degraded image of the sample is input into the image encoder in the pre-trained visual-text network to obtain the second image features; The second text features and the second image features are compared and learned to obtain a second contrastive learning loss. The network parameters of the image encoder are updated based on the second contrastive learning loss until the updated second image encoder is obtained. The updated second image encoder is used as the pre-trained style perceptron.

3. The image restoration method for multiple types of interference according to claim 1, characterized in that, The perceptron to be trained includes at least: a pre-trained visual Transformer structure, a first multilayer perceptron to be trained, a second multilayer perceptron to be trained, a self-attention layer to be trained, and a cross-attention layer to be trained. The degraded sample image is input into the perceptron to be trained to obtain the learning prompt information, including: The degraded image of the sample is input into the pre-trained visual Transformer structure to obtain multiple image feature vectors; The multiple image feature vectors are input into the first multilayer perceptron to be trained for processing to obtain multiple first feature vectors. Then, a self-attention operation is performed between the multiple first feature vectors through the self-attention layer to be trained to obtain a self-attention result. Multiple learning vectors are initialized, and the multiple learning vectors are cross-attentioned with the self-attention result through the cross-attention layer to be trained, so as to obtain the cross-attention result; The cross-attention result is input into the second multilayer perceptron to be trained for processing to obtain the updated learning vector. Based on the updated learning vector and the cross-attention result, the learning prompt information is obtained.

4. The image restoration method for multiple types of interference according to claim 1, characterized in that, The prompt generation network to be trained further includes: a prompt encoder to be trained, wherein the prompt encoder to be trained includes at least: three third-layer perceptrons and three fourth-layer perceptrons to be trained. The target prompt information is obtained by fusing and encoding the content prompt information, the style prompt information, and the prompt information to be learned through the prompt generation network to be trained, including: The content prompt information, style prompt information, and learning prompt information are respectively input into their respective third multilayer perceptrons to be trained for processing and then linear transformation is performed to obtain the first prompt feature, the second prompt feature, and the third prompt feature corresponding to the content prompt information, the style prompt information, and the learning prompt information, respectively. The first prompt feature, the second prompt feature, and the third prompt feature are respectively input into their respective fourth multilayer perceptrons to be trained for processing, so as to obtain the fourth prompt feature, the fifth prompt feature, and the sixth prompt feature corresponding to the content prompt information, the style prompt information, and the prompt information to be learned, respectively. After concatenating the fourth, fifth, and sixth cue features, the cue information is obtained by sequentially processing the cue information through a self-attention and feedforward network, and then output by the cue encoder to be trained.

5. The image restoration method for multiple types of interference according to any one of claims 1 to 4, characterized in that, The image restoration network to be trained is a diffusion model to be trained; during the image restoration process of the image restoration network to restore the sample degraded image, the target prompt information is combined with the feature layer of the image restoration network to obtain the sample restored image output by the image restoration network, including: The image restoration network to be trained performs forward diffusion and backward diffusion sequentially on the sample degraded image; During the backdiffusion process, the target cue information is combined with the feature layer in the backdiffusion process through a cross-attention mechanism to obtain the sample restoration image.

6. An image restoration device for multiple types of interference, characterized in that, The device includes: An image input module is used to input degraded sample images into the image restoration network to be trained and the prompt generation network to be trained in the image restoration model to be trained, respectively; the prompt generation network to be trained includes at least: a pre-trained content perceptron, a pre-trained style perceptron, and a perceptron to be trained and learned; the pre-trained content perceptron and the pre-trained style perceptron are obtained by retraining the image encoder in the pre-trained visual text network. The prompt generation module is used to process the sample degraded image through the prompt generation network to be trained, and generate content prompt information, style prompt information and learning prompt information corresponding to the sample degraded image respectively; the content prompt information represents the image content prompt corresponding to the sample degraded image, the style prompt information represents the degradation type prompt corresponding to the sample degraded image, and the learning prompt information represents the semantic prompt information corresponding to the sample degraded image, which is of a higher dimension than the content prompt information and the style prompt information; The prompt encoding module is used to fuse and encode the content prompt information, the style prompt information, and the prompt information to be learned through the prompt generation network to be trained, so as to obtain the target prompt information; An image output module is used to combine the target prompt information with the feature layer of the image restoration network during the image restoration process of the image restoration network to be trained, so as to obtain the sample restoration image output by the image restoration network. The model training module is used to train the image restoration model to be trained based on the sample restored image and the clean image corresponding to the sample degraded image until a trained image restoration model is obtained. The image restoration module is used to input the image to be restored into the trained image restoration model to obtain the restored image; The prompt generation module includes: The first prompt generation module is used to input the sample degraded image into the pre-trained content perceptron to obtain the content prompt information; The second prompt generation module is used to input the sample degradation image into the pre-trained style perceptron to obtain the style prompt information; The third prompt generation module is used to input the sample degradation image into the perceptron to be trained to obtain the prompt information to be learned; The image restoration device further includes: a first training module for training the pre-trained content perceiver, the first training module comprising: The first text generation module is used to input the clean image of the sample corresponding to the sample degraded image into a pre-trained graph-text multimodal network to obtain the clean image description text corresponding to the sample degraded image. The first text processing module is used to input the clean image description text into the text encoder in the pre-trained visual text network to obtain the first text features; The first image processing module is used to input the degraded sample image into the image encoder in the pre-trained visual text network to obtain the first image features; The first perceptron training module is used to compare and learn the first text features and the first image features to obtain a first contrastive learning loss, update the network parameters of the image encoder based on the first contrastive learning loss, until the updated first image encoder is obtained, and use the updated first image encoder as the pre-trained content perceptron.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the image restoration method for multiple types of interference as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the image restoration method for multiple types of interference as described in any one of claims 1 to 5.

Images