Multi-weather image restoration method and device based on coding and decoding double-prompt learning

Abstract

The invention relates to the technical field of computer vision, and discloses a multi-weather image restoration method and device based on coding and decoding double-prompt learning. The method comprises the following steps: performing multi-scale down-sampling to generate a multi-scale degraded image; carrying out feature extraction to obtain image features, adaptively generating coding prompt information by adopting a multi-scale coding prompt method, carrying out weighted fusion on the image features and the coding prompt information to obtain fusion features, and carrying out image feature extraction on the fusion features to obtain deepest image features; according to the deepest image features, adaptively generating decoding prompt information, carrying out double-attention fusion on the decoding prompt information and the image features, carrying out splicing fusion on the decoding prompt information and the image features to obtain fusion information, carrying out feature reconstruction according to the fusion information, and carrying out image reconstruction on a reconstructed feature map to obtain a restored image. According to the method, the adaptive capacity to various degradation types and the processing capacity to complex degradation phenomena can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-weather image restoration method and apparatus based on dual-cue learning of encoding and decoding, as well as a machine-readable storage medium and processor, applicable to integrated image restoration tasks under various severe weather conditions. Background Technology

[0002] Due to limitations in image acquisition equipment performance and interference from adverse weather conditions such as fog, rain, snow, and raindrops, images acquired in outdoor scenes often suffer from quality degradation. This degradation not only reduces the clarity and comfort of human observation, potentially leading to biased human judgment, but also hinders subsequent intelligent tasks such as image analysis and target recognition, increasing the likelihood of decision-making errors. Image degradation caused by different weather conditions exhibits significant differences in distribution and characteristics. For example, foggy weather manifests as increased global grayscale and decreased contrast; rainy weather presents as global rain line interference and localized blurred patches formed by raindrops from the lens; and snowy weather is characterized by randomly scattered, dense snowflake particles that obscure details and disrupt pixel continuity.

[0003] To improve the model's ability to process degraded images under various severe weather conditions, it is necessary to comprehensively consider the diverse impacts of different weather conditions on images. Although single-weather image restoration algorithms have achieved some success, they require separate model training for different weather degradation types, resulting in high resource consumption and difficulty in deployment on mobile or edge devices. Existing integrated restoration frameworks mostly adopt a shared codec structure, which easily leads to cross-task feature conflicts, resulting in poor restoration performance in complex degradation scenarios. Therefore, a lightweight and efficient multi-weather image integrated restoration method is needed to improve the model's adaptability to various degradation types and its ability to handle complex degradation phenomena. Summary of the Invention

[0004] The purpose of this invention is to overcome the problem that existing technologies are prone to cross-task feature conflicts, resulting in poor recovery performance in complex degradation scenarios. This invention provides a multi-weather image recovery method based on dual-cue learning of encoding and decoding. This multi-weather image recovery method based on dual-cue learning of encoding and decoding is a lightweight and efficient integrated recovery method for multi-weather images, which improves the model's adaptability to various degradation types and its ability to handle complex degradation phenomena.

[0005] To achieve the above objectives, the present invention provides a method for restoring multiple weather images based on dual-cue learning of encoding and decoding, comprising: S1: Perform multi-scale downsampling on the degraded image to generate a multi-scale degraded image; S2: Extract features from the largest-scale degraded image in the multi-scale degraded image to obtain the first image feature; adaptively generate coded prompt information using a multi-scale coding prompt method; and weight and fuse the first image feature with the coded prompt information to obtain the first fused feature. S3: Extract features from the first fusion feature to obtain the second image feature; obtain the second coded prompt information of the corresponding scale from the multi-scale degraded image; fuse the second image feature with the second coded prompt information to obtain the second fusion feature; S4: Repeat step S3 until the Nth fusion feature of the smallest scale degraded image is obtained. Then, perform image feature extraction on the Nth fusion feature to obtain the deepest image feature. S5: Generate first decoding prompt information adaptively based on the deepest image features, fuse the first decoding prompt information with the deepest image features using dual attention to obtain first fusion information, and concatenate and fuse the first fusion information with the image features of the previous layer to obtain second fusion information, and perform feature reconstruction based on the second fusion information; S6: Repeat step S5 until the reconstructed feature map is the same size as the largest degraded image. S7: Perform image reconstruction on the reconstructed feature map obtained in S6 to obtain the restored image.

[0006] Preferably, the multi-scale coding hint method generates the coding hint information by calculating attention in the channel dimension through a coding hint generation module.

[0007] Preferably, the computational complexity of calculating attention in the channel dimension is O(NC²), where N is the size of the spatial dimension and C is the number of channels.

[0008] Preferably, the perception prompt generation step includes a decoding prompt information generation step where the prompt generation module (PGM) and the weather classification loss function work together to generate representative weather embedded prompt information as the decoding prompt information. Dual attention fusion step: The decoded prompt information is fused with the encoder-decoder features through bidirectional interaction of feature information attention and prompt information attention.

[0009] Preferably, the perceptual prompt generation step includes: performing global average pooling on the input features, passing them sequentially through a linear layer and an activation function, and then operating through a fully connected layer to generate random weights to adjust the importance of the features, and finally generating the decoded prompt information through a convolutional layer.

[0010] Preferably, in the dual attention fusion step: The feature information attention focuses on key degradation areas by introducing prompt information into the feature query vector; The cue information attention incorporates feature information into the cue key vector and value vector to mine contextual information related to the recovered features.

[0011] Preferably, the loss function is the total loss function, with the minimum absolute deviation loss L. rec Amplitude spectral loss L amp Phase spectrum loss L pha Weather-related losses L cls The overall composition of the loss function can be simply represented as follows: .

[0012] Secondly, the present invention provides a multi-weather image restoration device based on dual-cue learning of encoding and decoding. The device includes: a first processing module for performing multi-scale downsampling on degraded images to generate multi-scale degraded images;

[0013] The second processing module is used to extract features from the largest-scale degraded image in the multi-scale degraded image to obtain a first image feature; adaptively generate coded prompt information using a multi-scale coded prompt method; weightedly fuse the first image feature with the coded prompt information to obtain a first fused feature; extract features from the first fused feature to obtain a second image feature; obtain the second coded prompt information of the corresponding scale from the multi-scale degraded image; fuse the second image feature with the second coded prompt information to obtain a second fused feature; repeat the above steps until the Nth fused feature of the smallest-scale degraded image is obtained; extract image features from the Nth fused feature to obtain the deepest image feature; The third processing module is used to adaptively generate first decoding prompt information based on the deepest image features, perform dual attention fusion of the first decoding prompt information and the deepest image features to obtain first fusion information, and concatenate and fuse the first fusion information with the image features of the previous layer to obtain second fusion information. Feature reconstruction is performed based on the second fusion information. The above steps are repeated until the reconstructed feature map is the same size as the largest degraded image. The fourth processing module is used to reconstruct the image from the reconstructed feature map obtained in the third module, and obtain the restored image.

[0014] Thirdly, the present invention provides a machine-readable storage medium storing instructions that cause a machine to execute the multi-weather image restoration method based on encoding / decoding dual-hint learning as described in any of the preceding claims of this application.

[0015] Fourthly, the present invention provides a processor for running a program, wherein the program is executed to perform: the instructions are used to cause the machine to perform the multi-weather image restoration method based on encoding / decoding dual-cue learning as described in any of the preceding claims of this application.

[0016] This invention proposes an image restoration scheme that combines a dual-cue learning mechanism for encoding and decoding with a frequency domain enhancement loss function, achieving significant results in multi-weather image restoration tasks: In the encoding stage, a multi-scale cueing method is adopted, and attention fusion is used to effectively enhance the model's adaptability to complex degradation scenes; In the decoding stage, perceptual cue generation and dual-attention mechanism are integrated, resulting in excellent performance in detail restoration and noise suppression.

[0017] As a preferred implementation, this scheme jointly optimizes the amplitude and phase spectrum losses, constraining the image reconstruction process in both the spatial and frequency domains, balancing detail restoration and structural integrity. This design not only achieves adaptive integrated image restoration under various adverse weather conditions and exhibits excellent generalization performance, but also employs a lightweight structure to significantly reduce computational complexity, enabling deployment on mobile and edge devices and meeting real-time processing requirements. This scheme is applicable to integrated image restoration tasks under various adverse weather conditions, providing an efficient and reliable image enhancement solution for vision systems in complex environments. Attached Figure Description

[0018] Figure 1 This is a flowchart of an image restoration method based on dual prompts for encoding and decoding according to the present invention; Figure 2 This is an overall schematic diagram of an image restoration method based on dual prompts of encoding and decoding according to the present invention; Figure 3 This is a schematic diagram of the multi-scale encoding prompt generation module of the present invention; Figure 4 This is a schematic diagram of the perception decoding prompt generation module of the present invention; Figure 5 This is a schematic diagram of the dual attention cue fusion module of the present invention; Figure 6 These are the unit parameters of the multi-scale encoding prompt generation module of this invention; Figure 7 These are the unit parameters of the perception decoding prompt generation module of this invention; Figure 8 These are the unit parameters of the dual attention cue fusion module (De-pro1) of this invention; Figure 9 These are the unit parameters of the dual attention cue fusion module (De-pro2) of this invention; Figure 10 These are the unit parameters of the dual attention cue fusion module (De-pro3) of this invention. Detailed Implementation

[0019] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0020] Currently used multi-weather image restoration methods in the industry typically require separate model training for different weather degradation types, resulting in high resource consumption and difficulty in deployment on mobile or edge devices. Existing integrated restoration frameworks often employ a shared encoder-decoder structure, which easily leads to cross-task feature conflicts, resulting in poor restoration performance in complex degradation scenarios. While some methods attempt to unify processing, their high model complexity and slow inference speed make them unsuitable for real-time applications. To address this, this invention develops a novel multi-weather image restoration method based on encoder-decoder dual-cue learning. This method achieves adaptive restoration under various severe weather conditions while ensuring lightweight operation, short processing time, significant results, and good consistency across diverse degradation types.

[0021] Figure 1 This is a flowchart of an image restoration method based on dual-hint encoding and decoding according to the present invention. Figure 2 This is a schematic diagram of the overall method for image restoration based on dual prompts of encoding and decoding according to the present invention, as shown below. Figure 1 , Figure 2 As shown, Step 101 involves multi-scale downsampling of the degraded image to generate a multi-scale degraded image.

[0022] Specifically, corresponding Figure 2 The leftmost column is used to downsample the input degraded image at scales of 0.75×, 0.5×, and 0.25× to obtain a multi-scale degraded image.

[0023] Step 102: Extract features from the largest-scale degraded image in the multi-scale degraded image to obtain the first image feature; adaptively generate coded prompt information using a multi-scale coding prompt method; and weight and fuse the first image feature with the coded prompt information to obtain the first fused feature.

[0024] Step 103 extracts features from the first fusion feature to obtain the second image feature, obtains the second coded prompt information of the corresponding scale from the multi-scale degraded image, and fuses the second image feature with the second coded prompt information to obtain the second fusion feature.

[0025] Repeat steps 102-103 until the Nth fusion feature of the smallest scale degraded image is obtained. Then, perform image feature extraction on the Nth fusion feature to obtain the deepest image feature.

[0026] Specifically, this invention employs U-Net as the network architecture, incorporating a designed multi-scale encoding cueing method and a perceptual decoding cueing method in the encoding and decoding stages, respectively, forming a hierarchical information guidance system. This invention uses a four-layer network depth, with each level having a different number of Transformer blocks, specifically [4, 6, 6, 8] from level 1 to level 4, with a channel size C of 36. The channel spreading factor r in the feedforward network is set to 2.667. Setting the number of heads in the self-attention at different stages to {1, 2, 4, 8} allows for the use of one cue block between every two consecutive decoders, for a total of three cue blocks throughout the network. The total number of cue components is five. In the loss function, α can be set to 0.01, and β and γ are set to 0.05.

[0027] correspond Figure 2 On the left, the input degraded image is downsampled at scales of 0.75×, 0.5×, and 0.25× to obtain multi-scale degraded images. The largest-scale degraded image in the multi-scale degraded image is input into the attention module of the Transformer block to capture key degradation features of the degraded image, and then downsampled to obtain the first image features. The degraded image... The input encoding prompt generation module (En-proi) generates multi-scale encoded prompt information. The encoded prompt information is then multiplied element-wise with the first image features and summed to obtain the first fused feature, which integrates the prompt information and the features. The calculation formula can be expressed as follows: , The obtained first fused feature is then input again into the attention module of the Transformer block to capture key degradation features of the degradation image. This is followed by downsampling to obtain the second image feature, which is then multiplied element-wise with the corresponding scale's encoded cue information and summed to obtain the second fused feature. Repeating the above steps, this embodiment uses a 4-layer network depth, resulting in a fourth fused feature. This fourth feature is then input into the attention module of the Transformer block to capture key degradation features, yielding the deepest layer image feature.

[0028] Step 105: Adaptively generate first decoding prompt information based on the deepest image features; perform dual attention fusion of the first decoding prompt information and the deepest image features to obtain first fusion information; concatenate and fuse the first fusion information with the image features of the previous layer to obtain second fusion information; and perform feature reconstruction based on the second fusion feature information.

[0029] Step 106: Repeat step 105 until the reconstructed feature map is the same size as the degraded image with the highest resolution; perform image reconstruction on the obtained reconstructed feature map to obtain the restored image.

[0030] Specifically. Corresponding Figure 2 On the right side, the deepest image features obtained from the decoder are first input into the perceptual cue generation module to obtain the first cue information. Then, the dual-attention cue fusion module fuses the cue information with the deepest image feature information from the decoder to obtain the first fused information. This first fused information is then concatenated and fused with the image features from the previous layer of the encoder to obtain the second fused information. Feature reconstruction is then performed based on the second fused feature information. Step 105 is repeated until the reconstructed feature map has the same size as the degraded image with the highest resolution. The reconstructed feature map is then processed by the enhancement module to perform image reconstruction, resulting in the restored image. The perceptual cue generation module and the dual-attention cue fusion module are described below.

[0031] The Perceptual Cue Generation Module (PGM) is introduced into the image restoration task. In image restoration, cue generation methods provide additional information and guidance, helping models better cope with various complex situations and improving model performance and robustness. However, in real-world applications, especially in multi-weather restoration tasks, these methods still exhibit many limitations. Some existing cue generation methods rely primarily on the network's own parameter updates, which, while effective, are insufficient in discerning details of degradation types when facing complex and variable severe weather scenarios, thus affecting the final restoration result. Therefore, we introduce a loss constraint into the cue generation process, where the Cue Generation Module (PGM) and the weather classification loss work together to generate a representative weather embedding cue. We utilize global pooling, linear layers, and activation functions to generate the cue information. The cue components are learnable parameters that work together with features obtained from the encoder to generate the final cue information Pi. After generating cue Pi, we project this cue Pi onto the probability vector of the weather category classification using a fully connected layer, and refine this vector into a more accurate information representation during training using cross-entropy loss. This module optimizes the cues by incorporating constrained loss during training, enabling the generated cues to have strong weather perception capabilities without increasing the number of parameters, thus better guiding image reconstruction.

[0032] Dual-Attention Cue Fusion Module. How effectively the network utilizes cue information is equally important for the final restoration effect. The dual-attention cue fusion module proposed in this invention effectively captures the potential correlation between cue information and image features through bidirectional interaction of feature information attention and cue information attention, avoiding information loss during the fusion process. This design enables the model to make fuller use of cue information, improving the accuracy of localization and restoration of key degraded areas, especially in complex weather-dependent degradation scenarios, better addressing diverse image restoration needs. Specifically, Feature Information Attention (FAtten) introduces the features of cue information Pi into the restoration feature query vector Q, constructing a representative query Q to guide the model to focus on key degraded areas. This enhances the model's sensitivity to image degradation features, enabling it to more accurately capture the key information to be restored and avoid losing important details during feature fusion. Cue Information Attention (PAtten) introduces the restored feature information into the cue information K,V vectors, mining content more similar to the restored feature information and providing the model with richer contextual information. It helps the model find matching information in a broader feature space, enhances the model's understanding of global semantics, and enables the model to perform feature matching and recovery from a more comprehensive perspective when fusing prompt information.

[0033] like Figure 3 As shown, the multi-scale encoding hint method generates the encoded hint information by calculating attention in the channel dimension through an encoding hint generation module. Furthermore, the computational complexity of calculating attention in the channel dimension is O(NC²), where N is the spatial dimension size and C is the number of channels.

[0034] Specifically, input image First, the data is normalized using a normalization layer (Norm), then divided into three parallel branches. Each branch sequentially adjusts the number of channels using a 1×1 convolution and extracts local spatial features using a 3×3 dynamic convolution. Then, it utilizes... and Reshape the Q, K, and V obtained from the three branches into matrix form from H×W×C: Then, attention is calculated along the channel dimension: .in, Responsible for cross-channel pixel-level aggregation, This is responsible for encoding the spatial context within each channel. Afterwards, the attention-weighted features are adjusted for channel count using a 1×1 convolution, and then combined with the features from the original branch. Element-wise addition (residual connection) is performed to preserve the original information; finally, the fused features are reshaped to output the final Prompt features. The entire process achieves the fusion of local and global features through multi-branch feature extraction and transposed attention, providing guidance for subsequent tasks. Specific operational parameters are as follows: Figure 6 As shown.

[0035] like Figure 4 , Figure 5 As shown, the decoding prompt information generation steps are as follows: the prompt generation module (PGM) and the weather classification loss function work together to generate representative weather embedded prompt information as the decoding prompt information; the dual attention fusion step is as follows: through the bidirectional interaction of feature information attention and prompt information attention, the decoding prompt information is fused with image features. The decoding prompt information generation step includes: performing global average pooling on the input image features, passing them sequentially through a linear layer and an activation function, and then operating through a fully connected layer to generate random weights to adjust the feature importance, before generating the decoding prompt information through a convolutional layer. In the dual attention fusion step: the decoding prompt information is introduced into the feature query vector to obtain feature information attention; the image feature information is introduced into the prompt key vector and value vector to obtain prompt information attention.

[0036] Specifically, the perception prompt generation module of this invention, for input features First, global average pooling is applied across spatial dimensions to generate a feature vector v∈RC, effectively aggregating global contextual information, avoiding local noise interference, and allowing the model to focus on the overall degradation features of the image, providing a more robust feature foundation for subsequent prompt generation. Next, v is processed through linear layers and the Gelu activation function to obtain a denser feature vector. The linear layer performs a linear transformation on the features, adjusting the feature dimensions. The activation function introduces non-linearity into the model, enhancing its expressive power. Then, a fully connected layer operation is performed to generate random weights w∈RN. These weights are used to adjust the importance of the prompt information, and then a 3×3 convolutional layer is used to generate the final prompt information Pi. After generating prompt Pi, Pi is projected through a fully connected layer to become a probability vector for weather category classification, and the cross-entropy loss is calculated. The specific operational parameters of this module are as follows: Figure 7 As shown.

[0037] The dual-attention cue fusion module, at its core, combines two input features, Pi and Fi, and utilizes a fusion attention mechanism to achieve cross-feature information interaction and fusion, such as... Figure 4As shown. Overall, the input features Pi and Fi are fed into the fusion module. Within the fusion module, Fi and Pi are first subjected to layer normalization to standardize the feature distribution and reduce training instability. Then, they enter the fusion module to implement attention interaction between Fi and Pi. The features after attention interaction then undergo layer normalization and a feedforward network. The feedforward network uses linear transformations and activation functions to non-linearly enhance the features, capturing more complex feature patterns. The overall process can be represented as: For the fusion attention module, including feature information attention (F... Atten ) and attention to cue information (P) Atten The feature information attention process generates a cue query (Q), a key (K), and a value (V) from the feature information tensor Fi∈RH×W×C. This is achieved by applying a 1×1 convolution W. d (·) is used to aggregate cross-channel context information pixel by pixel, and a 3×3 depthwise convolution W is applied. p (·) is used to encode spatial context information channel by channel, thereby generating Q=WdWpPi, K=W d W p P i V=W d W p P i Similarly, we first perform layer normalization on the prompt feature Pi, and then convert it into a recovery query (…). ),key( ) and value ( )projection: =WdWpFi, =W d W p F i , =W d W p F i Then perform F respectively. Atten and P Atten The calculation of P. Atten In the middle, first Q and Perform a concat operation to obtain the overall query vector, and then use W. p (·) is used to encode spatial context information channel by channel, and then attention matrix is ​​calculated with the cue key K and value V. The calculation formula can be expressed as: For F Atten First, connect K with and V and Perform the Concat operation separately, and then calculate the attention matrix. :

[0038] here α is a learnable scaling parameter used to control the size of the dot product of K and Q before applying the softmax function. C(·) denotes the concat operation. and These represent the outputs of feature information attention and cue information attention, respectively. Finally, the two attention maps are fused to obtain the final output. The formula is as follows: The specific operating parameters for this module are as follows: Figures 7-10 As shown.

[0039] like Figure 2-5 As shown, in the above operation process, optimization is performed using a loss function, which includes: minimum absolute deviation loss L. rec Amplitude spectral loss L amp Phase spectrum loss L pha Weather-related losses L cls The overall composition of the loss function is: , where α, β, and γ are weighting coefficients.

[0040] Specifically, this invention uses multiple loss functions to jointly constrain and optimize training: minimum absolute bias loss Lrec, amplitude spectrum loss Lamp, phase spectrum loss Lpha, and weather classification loss Lcls. The overall composition of the loss functions can be simply represented as follows:

[0041] The L1 norm of the pixel-wise difference between the restored high-quality image Iout and the ground truth image Igt is used as the reconstruction loss, i.e., Lrec. , The weather classification loss between the weather probability vector and its corresponding true value is calculated based on cross-entropy loss. It is expressed as an equation: , Where M is the number of weather categories. ybc is the category label (0, 1, 2, 3). If the true category of sample B is equal to c, then it is set to 1; otherwise, it is set to 0. pbc is the predicted probability that B belongs to category C.

[0042] This invention calculates the L1 loss on the amplitude spectrum, expressed by the following formula: , , in, This represents the amplitude spectrum of the recovered image. Let F(u,v) represent the amplitude spectrum of the real image. F(u,v) represents the complex form of the image obtained by transforming it from the spatial domain to the frequency domain, where u and v are the frequency domain coordinates. Re(·) and Im(·) are the real and imaginary parts of the complex number in the frequency domain, respectively.

[0043] This invention calculates the L1 loss on the phase spectrum, expressed by the following formula: ,

[0044] in, This represents the phase spectrum of the recovered image. This represents the phase spectrum of the real image. This joint constraint of the spatial and frequency domains allows the model to simultaneously capture the frequency domain features of pixel-level differences and degradation patterns. In multiple degradation scenarios, it ensures both the targeted nature of detail restoration and the consistency of overall semantics, ultimately achieving better output in both the spatial and frequency domains.

[0045] like Figure 2 As shown, the process includes several learnable cue component modules, including training settings. During training, this invention uses the ADAM optimizer to train the network from scratch in an end-to-end manner. For all weather datasets, images are uniformly cropped to 128×128 pixels by the center, with 100 training iterations and a batch size of 4. The proposed frequency domain augmentation loss function is used for optimization. The initial learning rate for the first 30 epochs is 3e-4, gradually reduced to 1e-6 using cosine annealing over the remaining 70 epochs. We randomly apply horizontal and vertical flipping as data augmentation techniques to expand the diversity of training samples, thereby enhancing the model's adaptability to severe weather images from different directions and perspectives. After training and debugging, the network is ready for use.

[0046] The aforementioned multi-weather image restoration device based on dual-hint learning of encoding and decoding includes a processor and a memory. The first processing module, the second processing module, the third processing module, and the fourth processing module are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0047] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, existing technologies can overcome cross-task feature conflicts that lead to poor recovery performance in complex degradation scenarios. This paper proposes a multi-weather image restoration method based on dual-cue learning (encoder / decoder). This method is lightweight and efficient, integrating multiple weather images into a single restoration approach, improving the model's adaptability to various degradation types and its ability to handle complex degradation phenomena.

[0048] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0049] This invention provides a storage medium storing a program that, when executed by a processor, implements the multi-weather image restoration method based on dual-cue learning of encoding and decoding.

[0050] This invention provides a processor for running a program, wherein the program executes the multi-weather image restoration method based on codec dual-cue learning.

[0051] This invention provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: S1: Multi-scale downsampling of the degraded image to generate a multi-scale degraded image; S2: Extract features from the largest-scale degraded image in the multi-scale degraded image to obtain the first image feature; adaptively generate coded prompt information using a multi-scale coding prompt method; and weight and fuse the first image feature with the coded prompt information to obtain the first fused feature. S3: Extract features from the first fusion feature to obtain the second image feature; obtain the second coded prompt information of the corresponding scale from the multi-scale degraded image; fuse the second image feature with the second coded prompt information to obtain the second fusion feature; S4: Repeat step S3 until the Nth fusion feature of the smallest scale degraded image is obtained. Then, perform image feature extraction on the Nth fusion feature to obtain the deepest image feature. S5: Generate first decoding prompt information adaptively based on the deepest image features, fuse the first decoding prompt information with the deepest image features using dual attention to obtain first fusion information, and concatenate and fuse the first fusion information with the image features of the previous layer to obtain second fusion information, and perform feature reconstruction based on the second fusion information; S6: Repeat step S5 until the reconstructed feature map is the same size as the largest degraded image. S7: Perform image reconstruction on the reconstructed feature map obtained in S6 to obtain the restored image.

[0052] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0053] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program with the following initialization method steps: S1: performing multi-scale downsampling on the degraded image to generate a multi-scale degraded image; S2: Extract features from the largest-scale degraded image in the multi-scale degraded image to obtain the first image feature; adaptively generate coded prompt information using a multi-scale coding prompt method; and weight and fuse the first image feature with the coded prompt information to obtain the first fused feature. S3: Extract features from the first fusion feature to obtain the second image feature; obtain the second coded prompt information of the corresponding scale from the multi-scale degraded image; fuse the second image feature with the second coded prompt information to obtain the second fusion feature; S4: Repeat step S3 until the Nth fusion feature of the smallest scale degraded image is obtained. Then, perform image feature extraction on the Nth fusion feature to obtain the deepest image feature. S5: Generate first decoding prompt information adaptively based on the deepest image features, fuse the first decoding prompt information with the deepest image features using dual attention to obtain first fusion information, and concatenate and fuse the first fusion information with the image features of the previous layer to obtain second fusion information, and perform feature reconstruction based on the second fusion information; S6: Repeat step S5 until the reconstructed feature map is the same size as the largest degraded image. S7: Perform image reconstruction on the reconstructed feature map obtained in S6 to obtain the restored image.

[0054] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0055] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0056] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0059] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0060] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0061] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings; however, the present invention is not limited thereto. Within the scope of the inventive concept, various simple modifications can be made to the technical solutions of the present invention. For example, the U-Net infrastructure used in the preferred embodiments can be changed to other encoding and decoding network architectures. This includes combining various specific technical features in any suitable manner. For example, the downsampling ratios (0.75×, 0.5×, 0.25×) in the multi-scale coding cue method can be adjusted to other reasonable ratios such as 0.8×, 0.6×, 0.3×, or more downsampling scales such as 0.125×. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately. However, these simple modifications and combinations should also be considered as the content disclosed in the present invention and are all within the protection scope of the present invention.

Claims

1. A method for restoring multiple weather images based on dual-cue learning of encoding and decoding, characterized in that, include: S1: Perform multi-scale downsampling on the degraded image to generate a multi-scale degraded image; S2: Extract features from the largest-scale degraded image in the multi-scale degraded image to obtain the first image feature; adaptively generate coded prompt information using a multi-scale coding prompt method; and weight and fuse the first image feature with the coded prompt information to obtain the first fused feature. S3: Extract features from the first fusion feature to obtain the second image feature; obtain the second coded prompt information of the corresponding scale from the multi-scale degraded image; fuse the second image feature with the second coded prompt information to obtain the second fusion feature; S4: Repeat step S3 until the Nth fusion feature of the smallest scale degraded image is obtained. Then, perform image feature extraction on the Nth fusion feature to obtain the deepest image feature. S5: Generate first decoding prompt information adaptively based on the deepest image features, fuse the first decoding prompt information with the deepest image features using dual attention to obtain first fusion information, and concatenate and fuse the first fusion information with the image features of the previous layer to obtain second fusion information, and perform feature reconstruction based on the second fusion information; S6: Repeat step S5 until the reconstructed feature map is the same size as the largest degraded image. S7: Perform image reconstruction on the reconstructed feature map obtained in S6 to obtain the restored image.

2. The multi-weather image restoration method based on dual-cue learning according to claim 1, characterized in that, The multi-scale coding hint method generates the coding hint information by calculating attention in the channel dimension through the coding hint generation module.

3. The multi-weather image restoration method based on dual-cue learning according to claim 2, characterized in that, The computational complexity of calculating attention in the channel dimension is O(NC²), where N is the size of the spatial dimension and C is the number of channels.

4. The multi-weather image restoration method based on dual-cue learning according to claim 1, characterized in that, Decoding prompt information generation steps: The prompt generation module (PGM) and the weather classification loss function work together to generate representative weather embedded prompt information as the decoding prompt information; Dual attention fusion step: The decoded prompt information is fused with image features through bidirectional interaction between feature information attention and prompt information attention.

5. The multi-weather image restoration method based on dual-cue learning according to claim 4, characterized in that, The decoding prompt information generation step includes: performing global average pooling on the input image features, passing them sequentially through a linear layer and an activation function, and then operating through a fully connected layer to generate random weights to adjust the feature importance, before generating the decoding prompt information through a convolutional layer.

6. The multi-weather image restoration method based on dual-cue learning according to claim 4, characterized in that, In the dual attention fusion step: The decoded prompts are incorporated into the feature query vector to obtain feature information attention; Image feature information is incorporated into the cue key vector and value vector to obtain cue attention.

7. The multi-weather image restoration method based on dual-cue learning according to claim 1, characterized in that, In S2-S7, optimization is performed using a loss function, which includes: minimum absolute deviation loss L. rec Amplitude spectral loss L amp Phase spectrum loss L pha and weather classification loss L cls The overall composition of the loss function is: , where α, β, and γ are weighting coefficients.

8. A multi-weather image restoration device based on dual-cue learning of encoding and decoding, characterized in that, The device includes: The first processing module is used to perform multi-scale downsampling on the degraded image to generate a multi-scale degraded image; The second processing module is used to extract features from the largest-scale degraded image in the multi-scale degraded image to obtain a first image feature; adaptively generate coded prompt information using a multi-scale coded prompt method; weightedly fuse the first image feature with the coded prompt information to obtain a first fused feature; extract features from the first fused feature to obtain a second image feature; obtain the second coded prompt information of the corresponding scale from the multi-scale degraded image; fuse the second image feature with the second coded prompt information to obtain a second fused feature; repeat the above steps until the Nth fused feature of the smallest-scale degraded image is obtained; extract image features from the Nth fused feature to obtain the deepest image feature; The third processing module is used to adaptively generate first decoding prompt information based on the deepest image features, perform dual attention fusion of the first decoding prompt information and the deepest image features to obtain first fusion information, and concatenate and fuse the first fusion information with the image features of the previous layer to obtain second fusion information. Feature reconstruction is performed based on the second fusion information. The above steps are repeated until the reconstructed feature map is the same size as the largest degraded image. The fourth processing module is used to reconstruct the image from the reconstructed feature map obtained in the third module, and obtain the restored image.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to execute: the multi-weather image restoration method based on encoding / decoding dual-cue learning according to any one of claims 1-7.

10. A processor, characterized in that, For running a program, wherein the program is run to execute: the multi-weather image restoration method based on encoding / decoding dual-cue learning according to any one of claims 1-7.

Images