An integrated image restoration system and method based on multi-cue and multi-scale gating

Abstract

This invention belongs to the field of electronic information technology, specifically relating to an integrated image restoration system and method based on multi-dimensional prompts and multi-scale gating. The system includes: an encoding unit, used to acquire the image to be restored, extract local and global fusion feature information of the image to be restored, and sequentially perform deep convolution, gating learning, residual connection, and feature downsampling on the local and global fusion feature information to obtain encoded features; and a decoding unit, used to obtain a category representation vector related to the current degradation type through a degradation-aware classifier based on the encoded features, generate a global category representation vector based on global average pooling of the category representation vector, determine a dynamic weight matrix through the global category representation vector, and then decode the encoded features to obtain the restored image. The degradation-aware generator dynamically generates a global category representation vector based on the encoded features, and adjusts the weight parameters of the decoding and reconstruction network accordingly, enabling the system to adaptively handle different types of weather degradation factors.

Description

Technical Field

[0001] This invention belongs to the field of electronic information technology, specifically including an integrated image restoration system and method based on multi-dimensional prompts and multi-scale gating. Background Technology

[0002] In the environmental perception module of autonomous driving systems, high-quality visual input is crucial for ensuring safe vehicle operation. However, images in real-world traffic scenarios are often affected by adverse weather conditions such as rain, fog, snow, and low light, resulting in reduced contrast, edge texture occlusion, and blurred details, which in turn impacts the performance of visual task algorithms such as object detection and semantic segmentation. Therefore, researching an integrated traffic scene image restoration method that can adaptively handle various weather degradation types is of great significance for improving the perception reliability of autonomous driving systems in complex environments.

[0003] Deep learning-based methods primarily operate on spatial domain-based convolution and attention mechanisms, neglecting the differences in frequency domain representation among various image degradation types. This leads to the loss of high-frequency detail information during feature extraction. Although some recent studies have mitigated this problem to some extent by employing large-kernel convolution techniques, which better preserve high-frequency information while reducing computational complexity, existing methods still have significant limitations: on the one hand, they fail to establish differentiated feature learning mechanisms for different degradation types (such as rain streaks, noise interference, and low-light conditions); on the other hand, existing methods generally lack the ability to analyze features from multiple perspectives and generate learnable cue vectors.

[0004] Therefore, the key to solving the current integrated image restoration task lies in how to achieve the collaborative capture of high-frequency details and low-frequency features, establish a multi-angle degradation feature analysis mechanism, generate effective cue learning vectors, and achieve a balance between the number of model parameters and multi-task performance, so as to achieve dynamic adaptive adjustment of parameters for different degradation types. Summary of the Invention

[0005] The purpose of this invention is to provide an integrated image restoration method and system based on multi-dimensional prompts and multi-scale gating, so as to solve the technical problem in the prior art that it fails to achieve dynamic adaptive adjustment of parameters for different degradation types.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides an integrated image restoration system based on multi-dimensional prompts and multi-scale gating, comprising:

[0008] The encoding unit is used to acquire the image to be restored, extract the local and global fusion feature information of the image to be restored, and perform deep convolution, gate learning, residual connection and feature downsampling on the local and global fusion feature information in sequence to obtain the encoded features;

[0009] The decoding unit is used to obtain the category representation vector related to the current degradation type through the degradation-aware classifier based on the encoded features, generate a global category representation vector by global average pooling based on the category representation vector, determine the dynamic weight matrix through the global category representation vector, and decode the encoded features based on the dynamic weight matrix to obtain the restored image.

[0010] Preferably, the encoding unit includes:

[0011] The multi-scale gated frequency optimization module is used to extract local and global fusion feature information of the image to be restored. The local and global fusion feature information are sequentially subjected to deep convolution, gated learning and residual connection to obtain cascaded features.

[0012] The feature downsampling module is used to downsample the cascaded features using a combination of inverse subpixel convolution and pointwise convolution to obtain the encoded features.

[0013] Preferably, the multi-scale gating frequency optimization module includes:

[0014] A frequency-optimized transposed self-attention module is used to extract local and global fusion feature information of the image to be restored;

[0015] A multi-scale gated feedforward network is used to sequentially perform deep convolution, gated learning, and residual connections on local and global fused feature information to obtain cascaded features.

[0016] Preferably, in the frequency-optimized transpose self-attention module, extracting local and global fusion feature information of the image to be restored specifically includes:

[0017] The image to be restored is used as the input feature, and the input feature is grouped according to the channel dimension to obtain the first group and the second group;

[0018] The input features in the first group and the second group are normalized respectively to obtain the first group features and the second group features;

[0019] Use the first set of features sequentially Convolutional aggregation of pixel-level context information, using Depth-separable convolutions enhance local features, yielding first branch features, second branch features, and third branch features;

[0020] The feature dimensions of the first branch features, the second branch features, and the third branch features are reconstructed respectively. The feature similarity between different positions is calculated based on the feature dimensions of the first branch features and the second branch features. The feature similarity is used to generate the first transposed attention map through the Softmax function.

[0021] The second transposed attention map is obtained based on the feature dimensions of the first transposed attention map and the third branch features; the dimensions are reconstructed based on the second transposed attention map to obtain the first output feature;

[0022] The second set of features is divided into two branches; the first branch uses... The convolution aggregates pixel-level context information, and then uses... Convolution expands the receptive field and optimizes local feature information to obtain local feature information;

[0023] The second branch connects with local feature information through The second output feature is obtained by learning through convolution;

[0024] The first and second output features are concatenated along the channel dimension to obtain the fused feature. The fused feature is then processed... The convolution and residual connections yield local and global fused feature information.

[0025] Preferably, in the multi-scale gated feedforward network, cascaded features are obtained by sequentially performing deep convolution, gated learning, and residual connections on local and global fused feature information, specifically including:

[0026] By fusing local and global feature information Convolutions are used to exchange information along the channel dimension to obtain optimized features;

[0027] Based on the channel dimension, the optimized features are divided into four groups of features, and the corresponding features are applied in parallel to the four groups of features. Multi-level semantic extraction using depthwise convolution of varying sizes yields features at different scales.

[0028] Features at different scales are learned through gating with two linear mappings and element-wise dot products to obtain important features for different groups;

[0029] The key features of different groups are cascaded along the channel dimension and through After the convolutions perform channel-dimensional information exchange, residual connections are performed to obtain cascaded features.

[0030] Preferably, the decoding unit includes:

[0031] The degradation-aware classifier is used to obtain the category representation vector related to the current degradation type based on the encoded features, and to generate a global category representation vector based on the global average pooling of the category representation vector.

[0032] The dynamic weight decoding module is used to determine the dynamic weight matrix through the global category representation vector, and decode the encoded features according to the dynamic weight matrix to obtain the restored image.

[0033] In a second aspect, the present invention provides an integrated image restoration method based on multi-dimensional prompts and multi-scale gating, characterized in that it includes:

[0034] The image to be restored is obtained, and the local and global fusion feature information of the image to be restored is extracted. The local and global fusion feature information is then subjected to deep convolution, gated learning, residual connection and feature downsampling in sequence to obtain the encoded features.

[0035] Based on the encoded features, a degradation-aware classifier is used to obtain a category representation vector related to the current degradation type. A global category representation vector is generated by global average pooling based on the category representation vector. A dynamic weight matrix is ​​determined using the global category representation vector. The encoded features are then decoded based on the dynamic weight matrix to obtain the restored image.

[0036] Preferably, the step of obtaining a category representation vector related to the current degradation type through a degradation-aware classifier based on encoded features, and generating a global category representation vector by global average pooling of the category representation vectors, specifically includes:

[0037] Based on the resolution of the encoded feature, the composition of the first downsampling module is determined; if the resolution of the encoded feature is outside the range of the preset resolution value, the first downsampling module corresponding to the encoded feature includes a residual block and a downsampling block, and the encoded feature is processed by the first downsampling module to obtain intermediate features; otherwise, the first downsampling module corresponding to the encoded feature includes an attention block and a downsampling block, and the encoded feature is processed by the first downsampling module to obtain a category representation vector.

[0038] If an intermediate feature is obtained, it is determined whether the resolution of the intermediate feature is within the range of the preset resolution value. If it is, the second downsampling module corresponding to the intermediate feature includes an attention block and a downsampling block. The intermediate feature is processed by the second downsampling module to obtain a category representation vector. Otherwise, the second downsampling module corresponding to the intermediate feature includes a residual block and a downsampling block. A second downsampling module is added until the category representation vector is obtained.

[0039] After adjusting the channel dimension of the category representation vector, global feature compression is performed through global average pooling to obtain the global category representation vector.

[0040] In a third aspect, the present invention provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the integrated image restoration method based on multi-dimensional prompting and multi-scale gating as described in any one of the preceding claims.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium storing at least one instruction, which, when executed by a processor, implements the integrated image restoration method based on multi-dimensional prompting and multi-scale gating as described above.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] This invention proposes an integrated image restoration system based on multi-scale prompting and multi-scale gating. Its core components are a multi-scale gating frequency optimization module and a dynamic weight decoding module. Specifically, a multi-branch feedforward network incorporating parallel multi-scale convolutions and gating mechanisms is used to regulate the flow of feature information at different scales, thereby enhancing the nonlinear representation capability of the feedforward. Furthermore, a degradation-aware generator can dynamically generate category representation vectors based on encoded features and adjust the weight parameters of the decoding and reconstruction network accordingly, enabling the system to adaptively handle different types of weather degradation factors. Comparative experiments with existing advanced integrated image restoration methods, combined with component ablation analysis, show that the proposed algorithm performs better in integrated image restoration tasks under severe weather conditions, successfully addressing the shortcomings of existing methods such as insufficient nonlinear representation of multi-scale features and the inability to dynamically and adaptively adjust parameters according to degradation type. Attached Figure Description

[0044] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0045] Figure 1 This is a system block diagram of an embodiment of the present invention;

[0046] Figure 2 This is a system block diagram of the encoding and decoding units according to an embodiment of the present invention;

[0047] Figure 3 This is a block diagram of the frequency optimization module for multi-scale gating according to an embodiment of the present invention;

[0048] Figure 4 This is a flowchart of a method according to an embodiment of the present invention;

[0049] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0051] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0052] See Figure 1 In one embodiment of the present invention, an integrated image restoration system based on multi-dimensional prompts and multi-scale gating is provided, which realizes adaptive weight modulation for different degradation types, thereby improving the model's performance in removing specific degradation types. Specifically, it includes the following:

[0053] The encoding unit is used to acquire the image to be restored, extract the local and global fusion feature information of the image to be restored, and perform deep convolution, gate learning, residual connection and feature downsampling on the local and global fusion feature information in sequence to obtain the encoded features;

[0054] The decoding unit is used to obtain the category representation vector related to the current degradation type through the degradation-aware classifier based on the encoded features, generate a global category representation vector by global average pooling based on the category representation vector, determine the dynamic weight matrix through the global category representation vector, and decode the encoded features based on the dynamic weight matrix to obtain the restored image.

[0055] A multi-branch feedforward network incorporating parallel multi-scale convolutions and gating mechanisms is employed to regulate the flow of feature information at different scales, thereby enhancing the nonlinear representation capability of the feedforward. Furthermore, a degradation-aware generator dynamically generates category representation vectors based on encoded features and adjusts the weight parameters of the decoding and reconstruction network accordingly, enabling the system to adaptively handle different types of weather degradation factors. Comparative experiments with existing state-of-the-art integrated image restoration methods, combined with component ablation analysis, demonstrate that the proposed algorithm performs better in integrated image restoration tasks under severe weather conditions, successfully addressing the shortcomings of existing methods such as insufficient nonlinear representation of multi-scale features and the inability to dynamically and adaptively adjust parameters based on degradation type.

[0056] In some embodiments, see Figure 2 The encoding unit includes:

[0057] The Multi-scale Gating Frequency-optimized Transformer Block (MGFTB) is used to extract local and global fusion feature information of the image to be restored. The local and global fusion feature information are sequentially subjected to deep convolution, gate learning and residual connection to obtain cascaded features.

[0058] The feature downsampling module is used to downsample the cascaded features using a combination of inverse subpixel convolution and pointwise convolution to obtain the encoded features.

[0059] In some embodiments, see Figure 3 The multi-scale gating frequency optimization module includes:

[0060] The frequency-optimized transposed self-attention module (FTSM) is used to extract local and global fused feature information of the image to be restored.

[0061] Multi-scale Gating Feed-forward Network (MGFN) is used to sequentially perform deep convolution, gating learning, and residual connections on local and global fused feature information to obtain cascaded features.

[0062] A multi-branch parallel structure with different receptive fields is constructed by grouping depthwise separable convolution. Then, a Gaussian error linear unit activation function is introduced to perform nonlinear gating enhancement on the features of different branches to optimize information flow and enhance the model's generalization ability to different weather types.

[0063] In some embodiments, the extraction of local and global fusion feature information of the image to be restored in the frequency-optimized transpose self-attention module specifically includes:

[0064] The image to be restored is used as the input feature, and the input feature is grouped according to the channel dimension to obtain the first group and the second group;

[0065] The input features in the first group and the second group are normalized respectively to obtain the first group features and the second group features;

[0066] Use the first set of features sequentially Convolution aggregates pixel-level context information, using Depth-separable convolutions enhance local features, yielding first branch features, second branch features, and third branch features;

[0067] The feature dimensions of the first branch features, the second branch features, and the third branch features are reconstructed respectively. The feature similarity between different positions is calculated based on the feature dimensions of the first branch features and the second branch features. The feature similarity is used to generate the first transposed attention map through the Softmax function.

[0068] The second transposed attention map is obtained based on the feature dimensions of the first transposed attention map and the third branch features; the dimensions are reconstructed based on the second transposed attention map to obtain the first output feature;

[0069] The second set of features is divided into two branches; the first branch uses... The convolution aggregates pixel-level context information, and then uses... Convolution expands the receptive field and optimizes local feature information to obtain local feature information;

[0070] The second branch connects with local feature information through The second output feature is obtained by learning through convolution;

[0071] The first and second output features are concatenated along the channel dimension to obtain the fused feature. The fused feature is then processed... The convolution and residual connections yield local and global fused feature information.

[0072] In some embodiments, the specific implementation steps of the frequency-optimized transposed self-attention module are as follows:

[0073] The image to be restored is used as input features, given the input features. First, the input features are grouped into the first group according to the channel dimension. With the second group ,in , Then, the input features within each group are normalized, as shown in formulas (1) and (2). The normalization operation makes the features within each group have a similar distribution, thereby accelerating the training process of the model while improving the stability of the model.

[0074] (1)

[0075] (2)

[0076] in, For normalization layer; For input features; Represents the real number field; Indicates the height of the feature map; Indicates the width of the feature map; Indicates the number of channels in the feature map; Indicates the first group; Indicates the second group; Indicates the first set of features; This represents the second set of features.

[0077] Next, the first set of features is divided according to the channel dimension. Divided into three branches , and As shown in formula (3), using Convolution aggregates pixel-level context information, using Depth-separable convolutions enhance local features and encode channel-level spatial context information.

[0078] (3)

[0079] in, This represents the input features of the current branch, and the features of the first branch are taken respectively. Features of the second branch and the characteristics of the third branch ; express Depth-separable convolution; express The convolution.

[0080] Next, refactor The feature dimension is , The feature dimension is , The feature dimension is It calculates the feature similarity between different positions and generates the first transposed attention map using the Softmax function. As shown in formula (4):

[0081] (4)

[0082] in, Represents the real number field; This indicates transpose.

[0083] Then, the feature dimensions of the third branch feature. Multiplying them yields the second transposed attention map. As shown in formula (5). Subsequently, the first output feature is obtained by reconstructing the dimensions. As shown in formula (6):

[0084] (5)

[0085] (6)

[0086] In the formula, Represents matrix multiplication; Indicates feature dimension; Indicates the reconstructed dimension; Indicates the height of the feature map; Indicates the width of the feature map; This indicates the number of channels in the feature map.

[0087] Second set of features It is divided into two branches, the first branch uses The convolution aggregates pixel-level context information, and then uses... Convolution expands the receptive field and optimizes local feature information to obtain local feature information. As shown in formula (7):

[0088] (7)

[0089] In the formula, express Convolution;

[0090] Next, the second branch is combined with local feature information through... The convolution is used for learning, and the outputs of the two branches are multiplied element-wise to enhance high-frequency details, as shown in formula (8), to obtain the second output feature. .

[0091] (8)

[0092] In the formula, Represents the element-wise dot product; Indicates the second output feature; Represents local feature information; Indicates the second set of features;

[0093] Finally, the first and second output features are concatenated along the channel dimension to obtain the fused feature, and then... Convolution and residual connections yield fused local and global feature information. The calculation process is shown in formula (9):

[0094] (9)

[0095] In the formula, Represents the given input features; Indicates cascading.

[0096] In addition, when cascading the two sets of features after calculating transposed self-attention and large kernel convolutional attention, the following is used: The purpose of convolution is to achieve cross-channel information exchange and feature fusion.

[0097] In some embodiments, the multi-scale gated feedforward network sequentially performs deep convolution, gated learning, and residual connections on local and global fused feature information to obtain cascaded features, specifically including:

[0098] By fusing local and global feature information Convolutions are used to exchange information along the channel dimension to obtain optimized features;

[0099] Based on the channel dimension, the optimized features are divided into four groups of features, and the corresponding features are applied in parallel to the four groups of features. Multi-level semantic extraction using depthwise convolution of varying sizes yields features at different scales.

[0100] Features at different scales are learned through gating of the element-wise dot product of two linear mappings to obtain important features for different groups;

[0101] The key features of different groups are cascaded along the channel dimension and through After the convolutions perform channel-dimensional information exchange, residual connections are performed to obtain cascaded features.

[0102] In some embodiments, the specific implementation steps of the multi-scale gated feedforward network (MGFN) are as follows:

[0103] Fusing local and global feature information As input features, first through The module uses convolution to achieve information interaction of input features in the channel dimension, and then divides the optimized features into four groups. ,in This represents the four optimized sets of features. Subsequently, as shown in formula (10), this module is used in parallel. Size of depthwise convolution ( This enhances the model's ability to extract feature information at different scales.

[0104] (10)

[0105] In the formula, Represents features at different scales; express Depth convolution of size.

[0106] use , and By using depthwise convolutions of varying sizes, the module can achieve multi-level semantic extraction of input features, thereby enhancing the feedforward network's ability to express features. Features at different scales... go through function after and Perform element-wise dot products to achieve gating learning of features at different scales, as shown in formula (11). Decreasing the input value of a function increases the probability of an output value of 0. Therefore, the gating mechanism is designed as an element-wise dot product of two linear mappings, thus making the output focus more on the important information of the branch features.

[0107] (11)

[0108] In the formula, Representing features at different scales Features after gating learning; Represents the element-wise dot product; This represents a linear function of the Gaussian error.

[0109] After gating the features of different groups, the module concatenates the features of different groups along the channel dimension, and then... The convolutions enable information interaction of features along the channel dimension. Finally, the module uses residual connections to enhance the nonlinear representation capability of the feedforward network while preserving the original feature structure, as shown in equations (12) and (13):

[0110] (12)

[0111] (13)

[0112] This indicates the characteristics after information exchange; This represents the output of a multi-scale gated feedforward network; This represents the fusion of local and global features; Indicates cascading; This indicates that features (different groups) are concatenated after gating learning of features at different scales.

[0113] Through the above operations, the Multi-Scale Gated Feedforward Network (MGFN) first addresses the limitation that a single convolutional kernel cannot acquire information from different receptive fields when extracting features. Then, it uses a gating mechanism to solve the information redundancy problem caused by multiple different convolutional kernels extracting features simultaneously. Finally, it uses residual connections to avoid the gradient vanishing problem during training, thereby enhancing the image restoration model's ability to represent different types of severe weather degradation.

[0114] In some embodiments, see Figure 2The decoding unit includes a degradation-aware classifier and a dynamic weighting decoding module (DWDM). The degradation-aware classifier obtains a category representation vector related to the current degradation type based on encoded features, and generates a global category representation vector through global average pooling of the category representation vector. The dynamic weighting decoding module determines a dynamic weight matrix using the global category representation vector, and decodes the encoded features based on the dynamic weight matrix to obtain the restored image. During the decoding and reconstruction process, the dynamic weighting decoding module dynamically adjusts the convolutional layer parameters of the decoding unit according to the dynamic weight matrix generated by the degradation-aware classifier, achieving adaptive weight modulation for different degradation types, thereby improving the model's image restoration effect and adaptability for specific degradation types.

[0115] In some embodiments, the features extracted by the encoding unit are used to obtain a category representation vector related to the current degradation type through a degradation-aware classifier. The specific implementation steps of the degradation-aware classifier are as follows:

[0116] Assuming input image features resolution The value is determined by the resolution setting. If the value is outside the range, the downsampling module is set to contain only residual blocks (RB) and downsampled blocks (DB), excluding attention blocks (AB), i.e., the first downsampling module. Input image features The intermediate features are obtained after the first downsampling module. The operation performed by the downsampling module can be described by formula (14):

[0117] (14)

[0118] in, Represents the residual module. This indicates a downsampling convolution. Downsampling is not used in the last layer of the module, so the final result is still a residual module. This represents the input image features after the first downsampling module operation has finished. resolution Set resolution Within the range, the downsampling module adds an attention module and a downsampling module after each residual module and after the downsampling module to improve the feature interaction between intermediate layers, i.e., the second downsampling module. Specifically, the operations performed by the model can be described by formulas (15) and (16):

[0119] (15)

[0120] (16)

[0121] in For attention module, This represents the intermediate features of the second downsampling module. ; This represents the feature after the second downsampling module operation is completed. .

[0122] The strategy of dynamically adjusting the number of downsampling layers can enhance the interaction between intermediate layer features, ultimately achieving accurate classification through spatial compression, and generating class representation vectors for subsequent dynamic updates of model weight parameters. Subsequently, the degradation-aware classifier utilizes... Convolution is used to adjust the channel dimensions, followed by global average pooling (GAP) to compress global features, ultimately generating a global class representation vector. The corresponding execution flow is shown in formula (17):

[0123] (17)

[0124] in, This represents the dimension of the global category representation vector F.

[0125] Step 105, the Dynamic Weight Decoding (DWDM) module includes the following steps:

[0126] The category representation vector is obtained from step 104. Let it pass Activation functions perform feature classification, resulting in tensors. , , The formula for the number of dynamic convolution kernels is shown in (18):

[0127] (18)

[0128] Among them, the global average feature vector generated by the degradation-aware classifier is passed through a fully connected layer. The model connects to each neuron in each layer of the decoding and reconstruction stage, and uses the Softmax activation function to perform a non-linear mapping on the feature vector, thereby updating the weight parameters of the decoding and reconstruction stage and enhancing the model's ability to remove different weather degradations.

[0129] The dynamic weight generation module generates weights based on the input tensor. and fundamental parameter matrix The dynamic parameter matrix is ​​generated, and the process is shown in formula (19):

[0130] (19)

[0131] in, Represents matrix multiplication. The synthesized dynamic parameter matrix, It is the feature dimension.

[0132] Furthermore, the generated dynamic parameter matrix Divided into attention weights and feedforward network weights The weights are then embedded into the frequency-optimized transposed self-attention module and the multi-scale gated feedforward network module, respectively, replacing the original depthwise convolutional weight parameters in these modules, ultimately resulting in the Dynamic Weight Decoding (DWDM) module. The dynamic parameter matrix is ​​then... The process of dividing the weights into attention weights and feedforward network weights is shown in formula (20):

[0133] (20)

[0134] In the formula, Indicates attention weight; Indicates the feedforward network weights; Represents the partitioning of the dynamic parameter matrix .

[0135] In some embodiments, the aforementioned feature upsampling operation Up is implemented using pixel-shuffle operations and pointwise convolution. The implementation steps of the multi-dimensional general hint module MGPM are as follows:

[0136] Given input features First, feature map It is uniformly divided into three parts along the channel dimension, resulting in , and Subsequently, the model initializes three sets of learnable general cue vectors and initializes all learnable parameters to 1, with corresponding dimensions of [missing data]. , and .in and These represent the dimensions of the cue vector in height and width, respectively. This represents the number of channels in the cue vector. The three-dimensional input features are obtained through structural transformation. , and Subsequently, each element is convolved pointwise with the corresponding dimension's general cue. Through general feature enhancement learning across different dimensions, the module learns richer general cue information. Compared to commonly used single-dimensional general cue learning strategies, this approach better addresses the problem of single-dimensional overfitting, thereby improving the model's performance in integrated image restoration tasks. Indicates the number of batch processes; This indicates the width of the feature map.

[0137] Example 1

[0138] The specific algorithmic steps for constructing an integrated image restoration system based on multi-dimensional prompts and multi-scale gating are as follows:

[0139] Step 1: Begin integrated image restoration based on multi-dimensional cueing and multi-scale gating;

[0140] Step 2: Preprocess the input image data to obtain the training set and test set;

[0141] Step 201: Input the degraded-sharp image pair to be restored , N For the number of samples, this example uses, but is not limited to, 256×256 random cropped blocks to train the network.

[0142] Step 202: Augment the dataset by randomly flipping it horizontally and vertically, and randomly shuffle the dataset to divide it into training and test sets.

[0143] Step 203: End the preprocessing of the input image data to obtain the corresponding training set and test set.

[0144] Step 3: The input sample image is fed into the encoding part, and feature encoding is performed sequentially through the multi-scale gated frequency optimization block MGFTB and the feature downsampling operation Down;

[0145] Step 4: Construct a degradation-aware classifier and embed it into the Dynamic Weight Decoding (DWDM) module in the decoding and reconstruction network to guide the generation of category representation vectors related to the current degradation type;

[0146] Step 5: Construct a Dynamic Weight Decoding (DWDM) module to update the weight parameters according to the input degradation type, enabling the model to adaptively update parameters based on different weather degradation types.

[0147] Step 6: Construct the network decoding part, and perform feature decoding sequentially through the Multi Dynamic Weight Decoding Module (DWDM), the Multi-Dimensional General Hint Module (MGPM), and the Feature Upsampling Operation (Up).

[0148] Step 7: Construct an integrated network structure based on multi-dimensional prompts and multi-scale gating. Input training data to train the network;

[0149] Step 8: Input the test data into the trained network model to complete the image restoration based on multivariate prompts and multi-scale gating of the integrated network. Use the integrated network structure model based on multivariate prompts and multi-scale gating trained in Step 7 to perform image restoration on the test set constructed in Step 2.

[0150] The loss function used in the network training process is the Mean Absolute Error (MAE) loss function, i.e., the L1 loss function, and its corresponding mathematical expression is formula (21).

[0151] (twenty one)

[0152] in, For the sample The true value, For the sample The predicted value, The number of samples; The actual value; These are predicted values.

[0153] Example 2

[0154] See Figure 4 Based on the same inventive concept as the above embodiments, the present invention also provides an integrated image restoration method based on multi-dimensional prompts and multi-scale gating, comprising:

[0155] S1: Obtain the image to be restored, extract the local and global fusion feature information of the image to be restored, and perform deep convolution, gated learning, residual connection and feature downsampling on the local and global fusion feature information in sequence to obtain the encoded features;

[0156] S2: Based on the encoded features, obtain the category representation vector related to the current degradation type through the degradation-aware classifier. Generate a global category representation vector by global average pooling based on the category representation vector. Determine the dynamic weight matrix through the global category representation vector. Decode the encoded features based on the dynamic weight matrix to obtain the restored image.

[0157] In some embodiments, the step of obtaining a category representation vector related to the current degradation type through a degradation-aware classifier based on encoded features, and generating a global category representation vector by global average pooling of the category representation vectors, specifically includes:

[0158] Based on the resolution of the encoded feature, the composition of the first downsampling module is determined; if the resolution of the encoded feature is outside the range of the preset resolution value, the first downsampling module corresponding to the encoded feature includes a residual block and a downsampling block, and the encoded feature is processed by the first downsampling module to obtain intermediate features; otherwise, the first downsampling module corresponding to the encoded feature includes an attention block and a downsampling block, and the encoded feature is processed by the first downsampling module to obtain a category representation vector.

[0159] If an intermediate feature is obtained, it is determined whether the resolution of the intermediate feature is within the range of the preset resolution value. If it is, the second downsampling module corresponding to the intermediate feature includes an attention block and a downsampling block. The intermediate feature is processed by the second downsampling module to obtain a category representation vector. Otherwise, the second downsampling module corresponding to the intermediate feature includes a residual block and a downsampling block. A second downsampling module is added until the category representation vector is obtained.

[0160] After adjusting the channel dimension of the category representation vector, global feature compression is performed through global average pooling to obtain the global category representation vector.

[0161] Example 3

[0162] See Figure 5 The present invention also provides an electronic device 100 for implementing an integrated image restoration method based on multi-dimensional prompts and multi-scale gating;

[0163] The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0164] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of embodiment 2 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0165] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0166] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0167] The memory 101 in the electronic device 100 stores multiple instructions to implement an integrated image restoration method based on multi-dimensional cueing and multi-scale gating, and the processor 102 can execute multiple instructions to achieve the following:

[0168] The image to be restored is obtained, and the local and global fusion feature information of the image to be restored is extracted. The local and global fusion feature information is then subjected to deep convolution, gated learning, residual connection and feature downsampling in sequence to obtain the encoded features.

[0169] Based on the encoded features, a degradation-aware classifier is used to obtain a category representation vector related to the current degradation type. A global category representation vector is generated by global average pooling based on the category representation vector. A dynamic weight matrix is ​​determined based on the category representation vector. The encoded features are then decoded based on the dynamic weight matrix to obtain the restored image.

[0170] Example 4

[0171] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0172] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An integrated image restoration system based on multi-dimensional prompts and multi-scale gating, characterized in that, include: The encoding unit is used to acquire the image to be restored, extract the local and global fusion feature information of the image to be restored, and perform deep convolution, gate learning, residual connection and feature downsampling on the local and global fusion feature information in sequence to obtain the encoded features; The decoding unit is used to obtain the category representation vector related to the current degradation type through the degradation-aware classifier based on the encoded features, generate a global category representation vector by global average pooling based on the category representation vector, determine the dynamic weight matrix through the global category representation vector, and decode the encoded features based on the dynamic weight matrix to obtain the restored image. The process of obtaining a category representation vector related to the current degradation type through a degradation-aware classifier based on encoded features, and generating a global category representation vector by global average pooling of the category representation vectors, specifically includes: Based on the resolution of the encoded feature, the composition of the first downsampling module is determined; if the resolution of the encoded feature is outside the range of the preset resolution value, the first downsampling module corresponding to the encoded feature includes a residual block and a downsampling block, and the encoded feature is processed by the first downsampling module to obtain intermediate features; otherwise, the first downsampling module corresponding to the encoded feature includes an attention block and a downsampling block, and the encoded feature is processed by the first downsampling module to obtain a category representation vector. If an intermediate feature is obtained, it is determined whether the resolution of the intermediate feature is within the range of the preset resolution value. If it is, the second downsampling module corresponding to the intermediate feature includes an attention block and a downsampling block. The intermediate feature is processed by the second downsampling module to obtain a category representation vector. Otherwise, the second downsampling module corresponding to the intermediate feature includes a residual block and a downsampling block. A second downsampling module is added until the category representation vector is obtained. After adjusting the channel dimension of the category representation vector, global feature compression is performed through global average pooling to obtain the global category representation vector.

2. The integrated image restoration system based on multi-dimensional prompts and multi-scale gating according to claim 1, characterized in that, The encoding unit includes: The multi-scale gated frequency optimization module is used to extract local and global fusion feature information of the image to be restored. The local and global fusion feature information are sequentially subjected to deep convolution, gated learning and residual connection to obtain cascaded features. The feature downsampling module is used to downsample the cascaded features using a combination of inverse subpixel convolution and pointwise convolution to obtain the encoded features.

3. The integrated image restoration system based on multi-dimensional prompts and multi-scale gating according to claim 2, characterized in that, The multi-scale gating frequency optimization module includes: A frequency-optimized transposed self-attention module is used to extract local and global fusion feature information of the image to be restored; A multi-scale gated feedforward network is used to sequentially perform deep convolution, gated learning, and residual connections on local and global fused feature information to obtain cascaded features.

4. The integrated image restoration system based on multi-dimensional prompts and multi-scale gating according to claim 3, characterized in that, In the frequency-optimized transpose self-attention module, extracting local and global fusion feature information of the image to be restored specifically includes: The image to be restored is used as the input feature, and the input feature is grouped according to the channel dimension to obtain the first group and the second group; The input features in the first group and the second group are normalized respectively to obtain the first group features and the second group features; Use the first set of features sequentially Convolution aggregates pixel-level context information, using Depth-separable convolutions enhance local features, yielding first branch features, second branch features, and third branch features; The feature dimensions of the first branch features, the second branch features, and the third branch features are reconstructed respectively. The feature similarity between different positions is calculated based on the feature dimensions of the first branch features and the second branch features. The feature similarity is used to generate the first transposed attention map through the Softmax function. The second transposed attention map is obtained based on the feature dimensions of the first transposed attention map and the third branch features; the dimensions are reconstructed based on the second transposed attention map to obtain the first output feature; The second set of features is divided into two branches; the first branch uses... The convolution aggregates pixel-level context information, and then uses... Convolution expands the receptive field and optimizes local feature information to obtain local feature information; The second branch connects with local feature information through The second output feature is obtained by learning through convolution; The first and second output features are concatenated along the channel dimension to obtain the fused feature. The fused feature is then processed... The convolution and residual connections yield local and global fused feature information.

5. The integrated image restoration system based on multi-dimensional prompts and multi-scale gating according to claim 3, characterized in that, In the multi-scale gated feedforward network, cascaded features are obtained by sequentially performing deep convolution, gated learning, and residual connections on local and global fused feature information, specifically including: By fusing local and global feature information Convolutions are used to exchange information along the channel dimension to obtain optimized features; Based on the channel dimension, the optimized features are divided into four groups of features, and the corresponding features are applied in parallel to the four groups of features. Multi-level semantic extraction using depthwise convolution of varying sizes yields features at different scales. Features at different scales are learned through gating with two linear mappings and element-wise dot products to obtain important features for different groups; The key features of different groups are cascaded along the channel dimension and through After the convolutions perform channel-dimensional information exchange, residual connections are performed to obtain cascaded features.

6. The integrated image restoration system based on multi-dimensional prompts and multi-scale gating according to claim 1, characterized in that, The decoding unit includes: A degradation-aware classifier is used to obtain a category representation vector related to the current degradation type based on encoded features, and to generate a global category representation vector based on the global average pooling of the category representation vector. The dynamic weight decoding module is used to determine the dynamic weight matrix through the global category representation vector, and decode the encoded features according to the dynamic weight matrix to obtain the restored image.

7. An integrated image restoration method based on multi-dimensional prompts and multi-scale gating, characterized in that, include: The image to be restored is obtained, and the local and global fusion feature information of the image to be restored is extracted. The local and global fusion feature information is then subjected to deep convolution, gated learning, residual connection and feature downsampling in sequence to obtain the encoded features. Based on the encoded features, a degradation-aware classifier is used to obtain a category representation vector related to the current degradation type. A global category representation vector is generated by global average pooling based on the category representation vector. A dynamic weight matrix is ​​determined based on the global category representation vector. The encoded features are then decoded based on the dynamic weight matrix to obtain the restored image. The process of obtaining a category representation vector related to the current degradation type through a degradation-aware classifier based on encoded features, and generating a global category representation vector by global average pooling of the category representation vectors, specifically includes: Based on the resolution of the encoded feature, the composition of the first downsampling module is determined; if the resolution of the encoded feature is outside the range of the preset resolution value, the first downsampling module corresponding to the encoded feature includes a residual block and a downsampling block, and the encoded feature is processed by the first downsampling module to obtain intermediate features; otherwise, the first downsampling module corresponding to the encoded feature includes an attention block and a downsampling block, and the encoded feature is processed by the first downsampling module to obtain a category representation vector. If an intermediate feature is obtained, it is determined whether the resolution of the intermediate feature is within the range of the preset resolution value. If it is, the second downsampling module corresponding to the intermediate feature includes an attention block and a downsampling block. The intermediate feature is processed by the second downsampling module to obtain a category representation vector. Otherwise, the second downsampling module corresponding to the intermediate feature includes a residual block and a downsampling block. A second downsampling module is added until the category representation vector is obtained. After adjusting the channel dimension of the category representation vector, global feature compression is performed through global average pooling to obtain the global category representation vector.

8. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the integrated image restoration method based on multi-cue and multi-scale gating as described in claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the integrated image restoration method based on multi-dimensional prompts and multi-scale gating as described in claim 7.

Images