Integrated image restoration method and device based on non-degradation perception feature modeling

Abstract

The invention discloses an integrated image restoration method and device based on non-degradation perception feature modeling. The method comprises the following steps: constructing a Dara model for image restoration; a linear congruence hash selector is adopted to replace an expert router in the dual-MoE system; inputting a to-be-restored image to the Dara model, so that the Dara model generates semantic restoration features; inputting the semantic restoration features into a specific degradation non-perceptual block so as to perform first fusion processing through a restoration prompt fusion module in the specific degradation non-perceptual block to obtain coarse-grained fusion features; inputting the coarse-grained fusion features into a TopK-Cross module in the specific degradation non-perception block for second fusion processing to obtain fine-grained fusion features; and injecting the fine-grained fusion features into a pixel-level restoration network to output a restored image. According to the method, explicit degradation perception feature modeling is not needed, various degradation modes including a shared degradation mode can be decoupled and independently learned, and high-quality restoration of low-quality images can be achieved.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and in particular to an integrated image restoration method and apparatus based on non-degradable perceptual feature modeling. Background Technology

[0002] Current image restoration methods have achieved impressive results by designing dedicated restoration solutions for each degradation type or reusing the same architecture across different degradations. Uniform image restoration aims to build a single model through a single training iteration, handling multiple degradation types simultaneously. This approach breaks free from the constraints of task-specific models, improving versatility and flexibility in complex scenarios. Compared to strategies that assign independent encoders or restoration heads to each degradation type, the mainstream "single encoder-single decoder" framework is more lightweight and maintains manageable computational and storage costs as the number of tasks increases. In this basic framework, the model typically learns degradation-aware features specific to a particular type and then provides intrinsic representations or external priors for different types of degradation processing.

[0003] However, schemes that encode highly discriminative degradation-aware features for different degradation types represent a high-level visual feature understanding based on human semantics. When the network is forced to learn highly discriminative degradation-aware features, the latent space is pushed toward semantic grouping of classes rather than fine-grained pixel cues, widening the gap between the degradation evidence required for representation and restoration. Common methods often rely on contrastive learning or task label-driven classification concepts, such as softmax activation and cross-entropy training strategies, to explicitly distinguish degradation types. However, softmax amplifies the maximum response, while cross-entropy forces the model to align with this maximum response, thus increasing the distance between classes. Although these operations can learn non-overlapping degradation representations based on human semantics, increasing research shows that some degradations exhibit overlap in pixel-level feature distributions, such as similar occlusion and brightness under adverse weather conditions, and shared global illumination patterns in dehazing and low-light enhancement. Related research further reveals shared patterns between different degradation features by calculating the softmax Shannon entropy of PromptIR (an all-around blind image restoration framework based on cue learning). Based on the above analysis, it is easy to see that existing integrated image restoration frameworks inevitably limit the model's ability to represent shared degradation patterns in the process of pursuing specific degradation-aware features. Therefore, designing a method that does not require explicit degradation-aware feature modeling, while constructing a restoration feature space, and can decouple and independently learn various degradation patterns, including shared degradation patterns, thus achieving a so-called degradation-free method, has become a crucial challenge. Summary of the Invention

[0004] This invention aims to at least partially address the technical problems in related technologies. Therefore, a first objective of this invention is to provide an integrated image restoration method based on non-degradation-aware feature modeling. This method eliminates the need for explicit degradation-aware feature modeling, decouples and independently learns various degradation patterns, including shared degradation patterns, and enables high-quality restoration of low-quality images.

[0005] The second objective of this invention is to provide an integrated image restoration device based on non-degradable perceptual feature modeling.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] An integrated image restoration method based on non-degradable perceptual feature modeling includes:

[0008] A pre-trained CLIP visual encoder was used as the backbone network, and a dual MoE system and specific degenerate non-perceptual blocks were integrated to obtain the Dara model for image restoration.

[0009] The expert router in the dual MoE system is replaced with a linear congruent hash selector;

[0010] Input the image to be restored into the Dara model so that the Dara model assigns an expert index to the degradation type of the image to be restored using a linear congruent hash selector;

[0011] The expert network in the dual MoE system corresponding to the expert index is activated to process the degraded image features extracted from the image to be restored by the CLIP visual encoder, and the outputs of the dual MoE system are fused to generate semantic restoration features.

[0012] The semantic restoration features are input into a specific degenerate non-perceptual block so that the restoration cue fusion module in the specific degenerate non-perceptual block can perform the first fusion process to obtain coarse-grained fusion features;

[0013] The coarse-grained fusion features are input into the TopK-Cross module in a specific degenerate non-perceptual block for a second fusion process to obtain fine-grained fusion features;

[0014] The fine-grained fusion features are injected into a pixel-level restoration network to output a restored image.

[0015] In one possible implementation, the dual MoE system is integrated, including:

[0016] Embed the projection hybrid expert system into the attention module of the transformer model in the backbone network;

[0017] An MLP module that embeds residual hybrid expert systems into the transformer model of the backbone network.

[0018] In one possible implementation, the expert index is assigned using a linear congruent hash selector as follows:

[0019]

[0020] Among them, c k The expert index is represented by a key-value pair, where A and B represent the first and second constants, d represents the degradation type, and V... dict (d) represents the dictionary lookup, indicating a dictionary lookup for the degenerate type d. mod represents the remainder operator for dividing two integers. N is the number of expert networks. This is a set of degenerate types.

[0021] In one possible implementation, a coarse-grained fusion feature is obtained by performing a first fusion process through a restoration cue fusion module in a specific degenerate, imperceptible block, including:

[0022] Learnable cue vectors that match the degradation type are selected from the cue pool;

[0023] The semantic reconstruction features, which are high-level semantic features, are multiplied element-wise with the learnable cue vector to generate a global guidance signal carrying global semantics.

[0024] The global guidance signal is concatenated with low-level visual features, and after convolutional splitting, it is fused through a gating mechanism to obtain coarse-grained fused features.

[0025] In one possible implementation, the coarse-grained fused features are input into the TopK-Cross module of a specific degenerate, non-perceptive block for a second fusion process to obtain fine-grained fused features, including:

[0026] Based on the sparse attention mechanism, multiple key location features are selected from low-level visual features;

[0027] Multiple key location features are combined with coarse-grained fusion features through cross-attention calculation to generate fine-grained fusion features.

[0028] In one possible implementation, before outputting the restored image through the Dara model, the method further includes training the Dara model, the training steps including:

[0029] Input a pair of paired images, the paired images comprising a low-quality input image and a high-quality target image;

[0030] The CLIP visual encoder with parameter freezing extracts high-level semantic features from high-quality target images as ground truth embeddings;

[0031] The Dara model receives a low-quality input image and learns to map the low-quality input image to a feature space to output a restored embedding, where the semantic features are similar to the high-level semantic features in the GT embedding.

[0032] In one possible implementation, the method further includes evaluating the similarity between the restored embedding and the GT embedding using a semantic restoration evaluation metric.

[0033] To achieve the above objectives, a second aspect of the present invention provides an integrated image restoration device based on non-degradable perceptual feature modeling, comprising:

[0034] The building block is used to integrate a dual MoE system and specific degenerate non-perceptual blocks to obtain a Dara model for image restoration, using a pre-trained CLIP visual encoder as the backbone network, and replacing the expert router in the dual MoE system with a linear congruent hash selector.

[0035] An input module is used to input the image to be restored into the Dara model, so that the Dara model assigns an expert index to the degradation type of the image to be restored using a linear congruent hash selector;

[0036] The activation module is used to activate the expert network in the dual MoE system corresponding to the expert index to process the degraded image features extracted from the image to be restored by the CLIP visual encoder.

[0037] The semantic restoration feature generation module is used to fuse the outputs of the dual MoE system to generate semantic restoration features;

[0038] The specific degenerate non-perceptual block includes a restoration cue fusion module and a TopK-Cross module, which are used to perform a first fusion processing operation on the semantic restoration features to obtain coarse-grained fusion features, and to perform a second fusion processing operation on the coarse-grained fusion features to obtain fine-grained fusion features, respectively.

[0039] The output module is used to inject fine-grained fusion features into the pixel-level restoration network to output the restored image.

[0040] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described above.

[0041] To achieve the above objectives, a fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method described above.

[0042] This invention has at least the following technical effects:

[0043] This invention provides an integrated image restoration method based on non-degenerate perceptual feature modeling. The first stage of this method initializes the Dara encoder using a CLIP visual encoder, including pre-trained parameters and weights. The pre-trained CLIP visual encoder serves as the backbone network, and a projection hybrid expert system is embedded into the attention module of the backbone network's transformer model, allowing for control over the attention module. Furthermore, a residual hybrid expert system is embedded into the MLP module of the backbone network's transformer model, enabling control over the MLP module. Thus, by integrating dual hybrid expert systems, synergistic semantic perception enhancement and detail restoration optimization can be achieved, while simultaneously improving computational efficiency and cross-task generalization ability. Additionally, the dual hybrid expert system, coupled with a training-free linear congruent hash selector, utilizes degenerate natural class partitioning instead of softmax routing, reducing computational cost and significantly improving computational efficiency. The second stage involves the selective-level deep fusion of semantic-level restoration information obtained from the Dara model—that is, semantic restoration features—into a pixel-level restoration network. The integrated specific degradation-agnostic block consists of a restoration cue fusion module and a TopK-Cross module, enabling pixel-level restoration networks to integrate semantic restoration priors with finer granularity. During processing, the specific degradation-agnostic block of this invention selects more specialized learnable cues from the cue pool for semantic-level restoration information. Each degradation task independently navigates its own restoration feature space, implicitly achieving differentiation between different degradation tasks. Hierarchical deep fusion can also seamlessly achieve the fusion of low-level restoration information from coarse to fine, thereby enabling high-quality restoration of low-quality images.

[0044] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] Figure 1 This is a flowchart of an integrated image restoration method based on non-degradable perceptual feature modeling, according to an embodiment of the present invention.

[0046] Figure 2 This is a flowchart illustrating the overall process of the integrated image restoration framework according to an embodiment of the present invention.

[0047] Figure 3 This is a visual comparison diagram of the method of this invention and the most advanced technology in the integrated restoration task.

[0048] Figure 4 This is a visual comparison diagram of the present invention with the latest technology on the AllWeather dataset, based on an embodiment of the invention.

[0049] Figure 5 This is a structural block diagram of an integrated image restoration device based on non-degradable perceptual feature modeling according to an embodiment of the present invention. Detailed Implementation

[0050] The following describes this embodiment in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0051] The integrated image restoration method based on non-degradable perceptual feature modeling proposed in this embodiment employs an independent two-stage training framework. In stage A, a visual encoder modified by a dual MoE (Hybrid Expert System) system is trained to generate restoration embeddings. Then, in stage B, these restoration embeddings are hierarchically and deeply integrated and injected into a pixel-level restoration network. The integrated image restoration method and apparatus based on non-degradable perceptual feature modeling of this embodiment are described below with reference to the accompanying drawings.

[0052] Figure 1 This is a flowchart illustrating an integrated image restoration method based on non-degradable perceptual feature modeling, according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0053] Step S101: Using a pre-trained CLIP visual encoder as the backbone network, a dual MoE system and a specific degenerate non-perceptual block are integrated to obtain the Dara model for image restoration.

[0054] In this embodiment, ViTs (a deep learning model) can be used as the backbone of the CLIP (a multimodal pre-trained model) visual encoder. Based on the pre-trained CLIP visual encoder, a scalable architecture, the Dara model, is then built using a dual MoE system. This architecture can freely expand as the number of degradation types processed increases. The Dara model also integrates specific degradation-aware blocks. In this embodiment, the MoE layer typically contains a set of parallel expert subnets or fine-tuning instruction sets. Where N is the number of expert networks, E i Let be the i-th expert network. Given an input feature x, it is typically assigned only to the activated expert networks for processing. The weight vector of the activated expert is... Typically generated by an expert router R(·) participating in the training, which can be a simple gated network, where... Let be an N-dimensional vector space over the real number field. Only the components with the largest TopK values ​​(a sparsity operation selecting K components) are chosen, and their contributions are determined through softmax probability normalization (a general mathematical transformation method). The MoE output is represented as follows:

[0055]

[0056] Where MoE(x) represents the function representation of the traditional MoE calculation process, representing the overall output of MoE to input x. The calculation method is to perform a weighted sum of the outputs of several expert networks, w i E represents the i-th component of the weight vector, i.e., the probability vector assigned to the i-th expert network, used to weight the expert network output. i (x) represents the i-th expert network, softmax(TopK(R(x))) represents the softmax probability normalization of the selected K scores to obtain the weight vector w, TopK(R(x)) represents the selection of the K largest scores from the N scores of R(x), and R(x) represents the router function that gives a score to each expert network.

[0057] In this embodiment, the traditional MoE expert router relies on a trainable routing gating network and softmax probability normalization.

[0058] Q, K, V projection is a key component of the Transformer model (a deep learning model based entirely on attention mechanisms) for capturing global semantic information. Therefore, this embodiment proposes a dual MoE mechanism, embedding two independent MoE systems, P-MoE (Projection Hybrid Expert System) and R-MoE (Residual Hybrid Expert System), into the attention module and MLP module of the Transformer model, respectively. Specifically, P-MoE is placed in the attention module of the Transformer model, replacing the traditional Q, K, V projection layers and output projection layers. By sparsely mapping the hash-like keys of the current sample, a one-hot (encoding method in deep learning) activation signal is obtained, which can decouple the complex degenerate feature space into a low-rank, trainable subspace through LoRA (Low-Rank Adaptive). This allows the projection MoE, i.e., P-MoE, to adaptively adjust attention weights and enhance the model's ability to represent high-level semantic features. Similarly, the residual MoE, i.e., R-MoE, is inserted as a bypass between the input and output of the MLP module, enabling the model to perform finer feature transformations in residual connections. This improves the model's ability to handle different degenerate tasks and enhances its ability to recover and generate complex features.

[0059]

[0060] Where Q, K, V, O represent the projection layers corresponding to the attention modules. E represents the weight vector obtained by the linear congruent hash selector and sparse mapping. i (x Q ,x K ,x V ,x O ) represents the i-th projection expert, x Q x K x V x O Let g(c) represent the inputs to the Q, K, and V linear projection layers, and the input to the output projection after attention calculation, respectively. k ) represents a sparse mapper, c k Let Y be the expert index represented by key-value pairs, Y be the final output of the transformer model after incorporating the R-MoE residuals, Z be the output of the MLP branch, and E be the expert index represented by key-value pairs. i (x out Let x represent the i-th residual expert. out This is the output of the attention module's output projection. Among them, P-MoE stabilizes the attention pattern through deterministic routing, while R-MoE enhances fine-grained recovery capabilities through diversity.

[0061] Step S102: Replace the expert router in the dual MoE system with a linear congruent hash selector.

[0062] Step S103: Input the image to be restored into the Dara model so that the Dara model assigns an expert index to the degradation type of the image to be restored using a linear congruent hash selector.

[0063] To further optimize the traditional MoE routing mechanism, this embodiment proposes a linear congruent hash selector, which replaces the trainable expert router while preserving the strict MoE sparsity. The linear congruent hash selector in... In time, known degradation labels (i.e., degradation types) are deterministically mapped to individual experts and converted into one-hot routing weights, thereby decoupling expert routers from gradient-based optimization and avoiding the instability of soft routing. Specifically, the degradation text label (i.e., degradation type) is denoted as... Here, "snow" means snow and "rain" means rain. Then, a fixed vocabulary is used, i.e., a lookup dictionary V. dict Normalize each degenerate text tag, and then apply a linear congruential hash selector to generate class key c. k The expert index is assigned using a linear congruent hash selector as follows:

[0064]

[0065] Where A and B represent the first and second constants, A and N are coprime, d represents the degeneration type, and V dict (d) is a dictionary lookup, indicating a dictionary lookup for the degenerate type d, and mod is the operator for finding the remainder after dividing two integers. For a set of degenerate types. Then the sparse mapper uses c. k As input, generate a weight vector:

[0066]

[0067] in, Let δ(·) represent the i-th component of the weight vector obtained by the sparse mapper, and let δ(·) be the Kronecker function, using the frozen V. dict Adding a new index is conflict-free as long as the number of different indexes assigned does not exceed N, and each new index maps to an unused remaining class; at a fixed N, existing assignments will not change.

[0068] Step S104: Activate the expert network in the dual MoE system corresponding to the expert index to process the degraded image features extracted from the image to be restored by the CLIP visual encoder, and fuse the outputs of the dual MoE system to generate semantic restoration features.

[0069] It should be noted that the Dara model can be trained before generating semantic restoration features and outputting the restored image. Specifically, a pair of paired images are input, consisting of a low-quality input image and a high-quality target image; a parameter-frozen CLIP visual encoder extracts high-level semantic features from the high-quality target image as a GT (high-quality target image) embedding; the Dara model receives the low-quality input image and learns to map the low-quality input image to a feature space to output the restored embedding, where the semantic features are similar to the high-level semantic features in the GT embedding.

[0070] Specifically, the input consists of a standard supervised image reconstruction pair (LQ, GT), where LQ is a low-quality input image and GT is a high-quality target image. The frozen CLIP visual encoder receives the GT to obtain the GT embedding, while the Dara model receives the LQ to learn how to map the LQ to a semantic space close to the GT, i.e., the feature space. During this process, the Dara model suppresses and corrects degradation factors in the LQ. After training, the output of the Dara model integrates the prediction and compensation of GT semantic features, implicitly carrying reconstruction priors for clear structure and semantic consistency, providing global semantic guidance for pixel-level reconstruction networks. Based on Peak Signal-to-Noise Ratio (PSNR), this embodiment also proposes a semantic reconstruction evaluation metric, E_PSNR, to evaluate the optimization degree of the high-dimensional embedding, i.e., the similarity between the reconstructed embedding and the GT embedding.

[0071]

[0072] Where, N b N represents the number of samples in the batch. d e represents the feature dimension of the embedding. i,j This represents the j-th dimension component of the input (LQ) embedding vector of the i-th sample. Let represent the component of the reference (GT) embedding vector corresponding to the i-th sample in the j-th dimension.

[0073] After training the Dara model, the expert network in the dual MoE system of the Dara model corresponding to the expert index can be activated to process the degraded image features extracted from the image to be restored by the CLIP visual encoder, and the outputs of the dual MoE system can be fused to generate semantic restoration features.

[0074] To bridge the gap between semantic-level and pixel-level reconstruction, this embodiment proposes a selective hierarchical deep fusion strategy. First, a selective cue pool is constructed using the concept of a linear congruent hash selector. Then, at the high-level semantic features... and optional learnable cue vectors Element-wise multiplication is performed between them, and then the data is passed through a set of depthwise separable convolutions and low-level visual features. A first coarse-grained fusion is performed. To further reduce feature domain differences, suppress noise, and inject semantic restoration priors into the pixel-level restoration network, a second fine-grained fusion is performed using a TopK-Cross module based on sparse attention (a feature fine-tuning module based on sparse attention). These represent three sets of real tensor matrices.

[0075] like Figure 2 As shown, through the above selective hierarchical deep fusion process, selective and adaptive learning from semantic level to pixel level restoration can be achieved.

[0076] Step S105: Input the semantic restoration features into a specific degenerate non-perceptual block so that the restoration prompt fusion module in the specific degenerate non-perceptual block can perform the first fusion process to obtain coarse-grained fusion features.

[0077] In one embodiment of the present invention, a coarse-grained fusion feature is obtained by performing a first fusion process through a restoration cue fusion module in a specific degenerate non-perceptive block, including:

[0078] Learnable cue vectors matching the degradation type are selected from the cue pool; semantic restoration features, which are high-level semantic features, are multiplied element-wise with the learnable cue vectors to generate a global guidance signal carrying global semantics; the global guidance signal is concatenated with low-level visual features, split by convolution, and then fused through a gating mechanism to obtain coarse-grained fused features.

[0079] Regarding the hint pool, to enable the restoration model to navigate the parameter space of a specific task more effectively and handle different degradation factors efficiently, this embodiment inherits the zero-overhead selection mechanism of the linear congruent hash selector from the Dara model. Class key c k Selecting a more specialized hint from the hint pool is similar to the basic principle of MoE.

[0080]

[0081] Where F0 is the global guidance signal weight, F h For high-level semantic features, P i Let be the i-th learnable cue vector in the cue pool.

[0082] Regarding the Restoration Cue Fusion Module (RPFM), since the goal is to achieve a coarse-to-fine hierarchical fusion of high-level semantic features and low-level visual features, RPFM can perform coarse fusion of the F0 carrying global semantics through a combined depthwise separable convolution.

[0083] (F1,F2)=chunk(Conv 1×1 (Cat(F l ,F0))) (7)

[0084] As shown in equation (7), the low-level visual features F l Connected to F0, initial processing is performed using a 1×1 convolution to obtain the first feature F1 and the second feature F2. Next, after the second feature F2 is convolved with a 3×3 convolution, a gating mechanism is used to activate the first feature F1. Then, element-wise multiplication is performed with the first feature F1 for fusion. Finally, pixel-level convolution is used to obtain the coarse-grained fused feature F after the first fusion. P As shown below:

[0085] F P ′=Mish(DwConv 3×3 (F2))⊙DwConv 5×5 (F1) (8)

[0086] F P =Conv P (F P ′) (9)

[0087] Among them, F P′ represents an intermediate feature of the first coarse-grained fusion feature, Mish(DwConv 3×3 (F2) represents the activation function, DwConv 3×3 (F2) indicates a depthwise separable convolution with a kernel size of 3×3, DwConv 5×5 (F1) indicates a depthwise separable convolution with a kernel size of 5×5. P (F P ′) represents pixel convolution.

[0088] Step S106: Input the coarse-grained fusion features into the TopK-Cross module in a specific degenerate non-perceptual block for a second fusion process to obtain fine-grained fusion features.

[0089] In one embodiment of the present invention, the coarse-grained fusion features are input into the TopK-Cross module in a specific degenerate non-perceptual block for a second fusion process to obtain fine-grained fusion features, including: selecting multiple key position features from low-level visual features based on a sparse attention mechanism; and performing cross-attention calculation between the multiple key position features and the coarse-grained fusion features to generate fine-grained fusion features.

[0090] To achieve a second fine-grained fusion of semantic restoration features and better modulate the composite low-level visual features, this embodiment designs a TopK-Cross module based on adaptive sparse attention. Sparse cross-attention is based on the TopK (returning the top K values ​​and their corresponding index functions) adaptive selection operation, effectively preserving the highest response of 'Q@K' (representing the calculation of the dot product similarity matrix between Q and K), achieving more efficient feature aggregation. In this embodiment, each encoder uses downsampling skip connections on the original degraded image to focus on learning the residuals caused by various degradation types.

[0091] Step S107: Inject fine-grained fusion features into the pixel-level restoration network to output the restored image.

[0092] To demonstrate the effectiveness of the proposed method in ensemble image restoration tasks, this embodiment evaluates it in three ensemble settings covering a wide range of degradation types. All reported results are the average of three runs (with different random number seeds).

[0093] Table 1 is a self-comparison table of PSNR for 6 tasks (a subset of 10 tasks) and 10 tasks.

[0094]

[0095] In this text, ↓ indicates a decrease, ↑ indicates an increase, and the loss of accuracy is indicated in bold.

[0096] Table 2 is a self-comparison table of 6 tasks (a subset of 10 tasks) and 10 tasks on SSIM.

[0097]

[0098]

[0099] SSIM stands for Structural Similarity Index, where ↓ indicates a decrease and ↑ indicates an increase, and the accuracy loss is represented in bold.

[0100] The 10-task setup: Technical personnel have increased the number of degradation types handled by a single model to 10. To comprehensively evaluate the model's ability to handle various degradations within a unified framework, this embodiment uses a dataset integrating 10 tasks, including dehazing, raindrop removal, snowfall, deblurring, low-light enhancement, JPEG artifact removal, denoising, shadow removal, and image inpainting.

[0101] 6 Task Settings: A subset of 10 Task Settings, including Defogging, Dehazing, Raindrop Removal, Snow, Deblurring, and Low Light Enhancement.

[0102] 4. Task Setup: This embodiment uses the AllWeather dataset for training to evaluate the model's integrated image restoration capabilities under various, even mixed, extreme weather degradation factors. This dataset includes Snow100K-L, Snow100K-S, Outdoor-Rain, and Raindrop.

[0103] For training in phase A, the Dara model is implemented in PyTorch (a machine learning framework) and trained on a single NVIDIA 4090 GPU (graphics processing unit). The AdamW optimizer (an improved version of the Adam optimizer) is used, with an initial learning rate of 2×10⁻⁶. -4 The batch size is 16×n, where n represents the number of tasks. Training for Phase B is also deployed on PyTorch and trained on a single NVIDIA L20 GPU. The entire training process is incremental, with an initial patch size of 128 and an initial batch size of 8. The AdamW optimizer is also used, with an initial learning rate of 3×10⁻⁶. -4 It is combined with a cosine annealing scheduling strategy.

[0104] The following is a comparison with advanced methods.

[0105] 10. Task Setup: To fully explore the potential of this method for uniform restoration across a wide range of degradation factors, it was compared with state-of-the-art methods. Figure 3 The visual comparison shown further confirms the effectiveness of the method presented in this paper.

[0106] 6. Task Setup: To evaluate the stability of the proposed method in terms of degradation type scalability, this setup was arranged for self-comparison. As shown in Tables 1 and 2, when scaling from 6 tasks to 10 tasks, the method effectively avoided a significant decrease in metrics, with an average PSNR decrease of only 1.8% and an average SSIM decrease of only 2.6%.

[0107] 4. Task Setup: To evaluate the ability of this method to restore clean images from various degraded observations, this experiment was conducted, comparing it with state-of-the-art integrated image restoration methods such as AdaIR and GridFormer. As shown in Table 3, the proposed method achieved state-of-the-art performance across all benchmarks and metrics. Qualitative results are as follows: Figure 4 As shown, the order is as follows: desnow, defog & rain, deraining raindrops. Compared with competing methods that leave weather residuals or introduce excessive illusory details, the proposed method achieves the best balance, providing comprehensive desiccation of severe weather while faithfully preserving scene details.

[0108] Table 3 compares the current state-of-the-art methods across four tasks.

[0109]

[0110]

[0111] The optimization of the Dara model primarily supports the B-stage, thus forming a complete non-degradational perception method. Compared with other CLIP-based methods, besides achieving a decoupled restoration feature space to handle all degradation modes, this design also solves key issues such as stability, scalability, and minimizing computational cost. The complete DaCLIP (an image restoration framework) generates two embedding spaces to fit the downstream network, resulting in the reuse of the visual encoder throughout the computation flow, while DG-Router (a routing architecture) uses a larger-scale encoder for better results. Dara achieves lightweight operation by utilizing a smaller visual encoder, requiring only one forward pass for the entire computation flow, as shown in Table 4.

[0112] Table 4 compares the computational overhead with other CLIP-based methods.

[0113]

[0114] The complete degeneracy-free perceptual framework follows an independent two-stage process. Therefore, this embodiment also conducted separate ablation studies to verify the contribution of each Dara component and SDAF (Specific Degeneracy-Free Perceptual Block) block to the integrated image restoration. Stage A: Remove the core modules P-MoE and R-MoE, and evaluate semantic restoration using E-PSNR. Similar to PSNR, a higher E-PSNR corresponds to a lower MSE (mean squared error), indicating that the embedding is closer to its GT semantic space. Stage B: Remove key components of the SDAF block, including RPFM, TopK-Cross, and the cue pool in RPFM. RPFM is crucial for injecting semantic-level information, and removing it directly reveals the benefit of the semantic features generated by the Dara model to the restoration network. Retaining only the TopK-Cross module restricts it to self-interaction of low-level visual features generated by the preceding decoder layers; essentially,

[0115] This is equivalent to simply adding an extra standard backbone block. As shown in Tables 5 and 6, the complete framework consistently outperforms all reduction variants across all benchmarks and metrics. Removing MoE in Phase A reduces E-PSNR, while removing RPFM or TopK-Cross in Phase B leads to a decrease in PSNR (e.g., 0.9 dB and 1.4 dB respectively for the desnow task). These results confirm the complementarity and necessity of the components.

[0116] Table 5 compares the ablation methods after applying this approach in the 6-task configuration.

[0117]

[0118] Table 6 compares the different methods after six-task ablation experiments of the Dara model on E-PSNR.

[0119]

[0120] Furthermore, the present invention also provides an integrated image restoration device based on non-degradable perceptual feature modeling. For example... Figure 5 As shown, the integrated image restoration device based on non-degradation-aware feature modeling includes a construction module, an input module, an activation module, a semantic restoration feature generation module, a specific degradation-aware block, and an output module connected in sequence.

[0121] The system comprises the following modules: a construction module, which uses a pre-trained CLIP visual encoder as the backbone network, integrates a dual MoE system and specific degradation-agnostic blocks to obtain a Dara model for image restoration, and replaces the expert router in the dual MoE system with a linear congruent hash selector; an input module, which inputs the image to be restored into the Dara model so that the Dara model assigns an expert index to the degradation type of the image to be restored using a linear congruent hash selector; an activation module, which activates the expert network in the dual MoE system corresponding to the expert index to process the degradation image features extracted from the image to be restored by the CLIP visual encoder; a semantic restoration feature generation module, which fuses the output of the dual MoE system to generate semantic restoration features; specific degradation-agnostic blocks, including a restoration cue fusion module and a TopK-Cross module, which respectively perform a first fusion processing operation on the semantic restoration features to obtain coarse-grained fusion features and a second fusion processing operation on the coarse-grained fusion features to obtain fine-grained fusion features; and an output module, which injects the fine-grained fusion features into the pixel-level restoration network to output the restored image.

[0122] It should be noted that the specific implementation of the integrated image restoration method based on non-degradational sensing feature modeling in this embodiment can be found in the specific implementation of the integrated image restoration device based on non-degradational sensing feature modeling described above. To avoid redundancy, it will not be repeated here.

[0123] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method.

[0124] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. An integrated image restoration method based on non-degradable perceptual feature modeling, characterized in that, include: A pre-trained CLIP visual encoder was used as the backbone network, and a dual MoE system and specific degenerate non-perceptual blocks were integrated to obtain the Dara model for image restoration. The expert router in the dual MoE system is replaced with a linear congruent hash selector; Input the image to be restored into the Dara model so that the Dara model assigns an expert index to the degradation type of the image to be restored using a linear congruent hash selector; The expert network in the dual MoE system corresponding to the expert index is activated to process the degraded image features extracted from the image to be restored by the CLIP visual encoder, and the outputs of the dual MoE system are fused to generate semantic restoration features. The semantic restoration features are input into a specific degenerate non-perceptual block so that the restoration cue fusion module in the specific degenerate non-perceptual block can perform the first fusion process to obtain coarse-grained fusion features; The coarse-grained fusion features are input into the TopK-Cross module in a specific degenerate non-perceptual block for a second fusion process to obtain fine-grained fusion features; The fine-grained fusion features are injected into a pixel-level restoration network to output a restored image.

2. The method as described in claim 1, characterized in that, The integrated dual MoE system includes: Embed the projection hybrid expert system into the attention module of the transformer model in the backbone network; An MLP module that embeds residual hybrid expert systems into the transformer model of the backbone network.

3. The method as described in claim 1, characterized in that, The allocation of expert indexes using a linear congruential hash selector is represented as follows: Among them, c k The expert index is represented by a key-value pair, where A and B represent the first and second constants, d represents the degradation type, and V... dict (d) represents the dictionary lookup, indicating a dictionary lookup for the degenerate type d. mod represents the remainder operator for dividing two integers. N is the number of expert networks. This is a set of degenerate types.

4. The method as described in claim 1, characterized in that, The first fusion process, performed by the restoration cue fusion module within a specific degenerate, imperceptible block, yields coarse-grained fusion features, including: Learnable cue vectors that match the degradation type are selected from the cue pool; The semantic reconstruction features, which are high-level semantic features, are multiplied element-wise with the learnable cue vector to generate a global guidance signal carrying global semantics. The global guidance signal is concatenated with low-level visual features, and after convolutional splitting, it is fused through a gating mechanism to obtain coarse-grained fused features.

5. The method as described in claim 1, characterized in that, The coarse-grained fused features are input into the TopK-Cross module in a specific degenerate, non-perceptual block for a second fusion process to obtain fine-grained fused features, including: Based on the sparse attention mechanism, multiple key location features are selected from low-level visual features; Multiple key location features are combined with coarse-grained fusion features through cross-attention calculation to generate fine-grained fusion features.

6. The method as described in claim 1, characterized in that, Before outputting the restored image through the Dara model, the method further includes training the Dara model, the training steps of which include: Input a pair of paired images, the paired images comprising a low-quality input image and a high-quality target image; The CLIP visual encoder with parameter freezing extracts high-level semantic features from high-quality target images as ground truth embeddings; The Dara model receives a low-quality input image and learns to map the low-quality input image to a feature space to output a restored embedding, where the semantic features are similar to the high-level semantic features in the GT embedding.

7. The method according to any one of claims 1 to 6, characterized in that, The method also includes evaluating the similarity between the restored embedding and the GT embedding using a semantic restoration evaluation metric.

8. An integrated image restoration device based on non-degradable perceptual feature modeling, characterized in that, include: The building block is used to integrate a dual MoE system and specific degenerate non-perceptual blocks to obtain a Dara model for image restoration, using a pre-trained CLIP visual encoder as the backbone network, and replacing the expert router in the dual MoE system with a linear congruent hash selector. An input module is used to input the image to be restored into the Dara model, so that the Dara model assigns an expert index to the degradation type of the image to be restored using a linear congruent hash selector; The activation module is used to activate the expert network in the dual MoE system corresponding to the expert index to process the degraded image features extracted from the image to be restored by the CLIP visual encoder. The semantic restoration feature generation module is used to fuse the outputs of the dual MoE system to generate semantic restoration features; The specific degenerate non-perceptual block includes a restoration cue fusion module and a TopK-Cross module, which are used to perform a first fusion processing operation on the semantic restoration features to obtain coarse-grained fusion features, and to perform a second fusion processing operation on the coarse-grained fusion features to obtain fine-grained fusion features, respectively. The output module is used to inject fine-grained fusion features into the pixel-level restoration network to output the restored image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images