A large multi-modal model guided adaptive image fog removal method

Abstract

The application discloses a large multi-modal model guided adaptive image fog removal method, belongs to the technical field of visual information processing, and is suitable for foggy scene image processing. The application realizes the method as follows: 1, obtaining degradation priori by using a pre-trained large multi-modal model; 2, inputting a foggy image and the degradation priori into a MoE-SSM model to perform training, so as to dynamically adjust model parameters and then perform dynamic perception defogging; 3, fusing shallow feature F s and optimized features to reconstruct a clean and fog-free image; 4, training a neural network by using a clean and fog-free image and a true value clear image by using a Charbonnier loss function shown in formula (8), so as to punish the deviation of the recovered image from the true value clear image and encourage consistent image gradients; compared with the prior art, the application solves the compromise processing problem of receptive field and calculation efficiency in image processing of the prior defogging model, simultaneously reduces the requirement for a large amount of labeled data, and then improves the image processing accuracy and stability.

Description

Technical Field

[0001] The present invention relates to an adaptive image fog removal method guided by a large multimodal model, belongs to the technical field of visual information processing, and is applicable to foggy scene image processing. Background Art

[0002] Image dehazing aims to restore a clean, fog-free image from a foggy image. This is crucial for advanced tasks that rely on visual information, such as autonomous driving and outdoor surveillance systems. Imaging in foggy weather can lead to severe image degradation, significantly reducing system performance. To meet the safety demands of all-weather services, developing an economical and effective image dehazing method has broad application value.

[0003] Foggy images can be modeled as a masked additive combination of a clean, fog-free image and the fog residual. In real-world scenes, especially outdoor scenes, obtaining the fog concentration and the accurate depth of the scene is a challenging task. Atmospheric scattering models are difficult to realistically represent all foggy scenes in the real world. Some works use deep neural networks based on end-to-end learning to handle degradation in all foggy scenes, but current image dehazing networks face an irreconcilable trade-off between receptive field size and computational efficiency. In addition, the fixed network architecture of existing methods limits the diversity and processing methods of learning different fog forms, such as dense fog and non-uniform fog images that have not been encountered before.

[0004] Currently, existing fog image processing has the following problems. (1) High computational complexity and resource requirements: Due to the trade-off between receptive field size and computational efficiency, existing methods may face significant limitations in resource-constrained devices or scenarios requiring real-time processing. (2) Limited generalization capability: The model requires a large amount of training data labeled with specific weather conditions, which is expensive to obtain or has limited data volume. When encountering new types of rain, fog, or other severe weather conditions that differ significantly from the training data, the performance of existing technologies is limited.

[0005] Therefore, how to solve the trade-off between receptive field and computational efficiency in the existing dehazing model image processing becomes an urgent problem to be solved. Summary of the Invention

[0006] The purpose of the present invention is to solve the technical problem of the trade-off between receptive field and computational efficiency in the image processing of existing defogging models, and to propose an adaptive image defogging method guided by a large multimodal model.

[0007] The principle of the present invention is as follows: the present invention proposes to use a large multimodal model (LMM) to generate a degradation prior for foggy images, and use the Mamba architecture to construct an image defogging network to adaptively perform image restoration. By inferring the diverse fog morphology knowledge in the feature space of the large multimodal model, the fog morphology in the picture is adaptively estimated. In addition, the potential of the Mamba architecture in image defogging tasks, the image defogging method based on the Mamba architecture is used to solve the trade-off between receptive field size and computational efficiency. At the same time, with the assistance of the mixed expert MoE model, the proposed method dynamically selects experts for restoration, thereby ensuring adaptive learning of specific fog knowledge, which is not limited by a fixed network architecture.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] The present invention provides an adaptive image haze removal method guided by a large multimodal model, which is applied to haze image restoration scenarios and includes the following steps:

[0010] Step 1: Obtain degradation prior using a pre-trained large multimodal model;

[0011] Step 1.1: Define the cue word Q of the evaluation image to evaluate the quality of the blurred image;

[0012] Step 1.2: Input the cue word Q of the evaluation image and the fog image into the large multimodal model, and output the probability distribution shown in Equation (1);

[0013] p=M(I h ,Q),p∈R L (1)

[0014] where M(·,·) is the large multimodal model, L is the length of all tokens in the large multimodal model, and p is the output probability distribution.

[0015] Step 1.3: Construct a set of candidate rating levels The mapping relationship between the image quality rating method and the text-defined rating level is established as shown in formula (2);

[0016]

[0017] where G(·) is a mapping from text-defined rating levels to probabilities, i∈[1,N] and j∈[1,L];

[0018] Step 1.4: Perform a closed set SoftMax operation on the output probability distribution p to obtain the degenerate prior as shown in Equation (3);

[0019]

[0020] wherein is the probability of the i-th level;

[0021] Step 2: input the fog image and the degradation prior into the MoE-SSM model for training, to dynamically adjust the model parameters, and then perform dynamic perception defogging;

[0022] Step 2.1: obtain the shallow feature F s through the convolution layer of the MoE-SSM for the fog image;

[0023] Step 2.2: use the LayerNorm layer to extract the long-distance dependency feature F ln from the shallow feature;

[0024] Step 2.3: based on the degradation prior, dynamically select the module index in the manner shown in formula (4);

[0025]

[0026] wherein v represents the selected index of the first K largest elements in

[0027] Step 2.4: use the selected index v obtained in step 2.3 to further extract the refined feature in the manner shown in formula (5):

[0028]

[0029] wherein the input feature is the i-th largest element in , SSB v(i) (·) is the v(i)-th SSB in all candidate SSM modules, SSB, i.e., SSM module, is the element-wise multiplication; SSB is used to model the long-distance pixel dependency. Each SSB successively contains a deep convolution layer, a SiLU activation function, a 2D-SSM layer and a LayerNorm layer; according to the degradation prior different SSBs are selected by using an adaptive strategy;

[0030] Step 2.5: obtain the fused feature in the manner shown in formula (6) by using the refined feature and the shallow feature:

[0031]

[0032] wherein TK(·,·) represents the top-K expert selection mechanism, Linear(·) and Residual connection and element-wise multiplication, respectively. Following the traditional Transformer architecture, we add F s and F d element-wise, and pass the result through LayerNorm, a convolutional layer, and an attention layer in order to obtain the output.

[0033] Step 2.6: Steps 2.1 to 2.5 are performed in a loop to obtain the optimized feature F

[0034]

[0035] where F s is the feature obtained from I h by a convolutional layer, δ is a learnable scaling factor, and H is the number of MM modules used; MM denotes a single module of MoE-SSM, i.e., a single module of the mixed expert-state space model; and MM(·,·,·) denotes the processing of multiple MM modules.

[0036] Step 3: The shallow feature F s and the optimized feature F are fused to reconstruct the clean and haze-free image.

[0037] Step 4: The clean and haze-free image and the ground truth clear image are used to train the neural network using the Charbonnier loss function as shown in equation (8) to penalize the deviation of the recovered image from the ground truth clear image and encourage consistent image gradients.

[0038]

[0039] where ∈ = 10-3 is a constant in all experiments.

[0040] Advantages:

[0041] The present application has the following advantages over the prior art:

[0042] 1. By combining a large multi-modal model with a mixed expert (MoE) model and a Mamba architecture, the present application dynamically selects appropriate experts for processing, solves the trade-off between receptive field and computational efficiency in image processing, significantly reduces the consumption of computing resources, and thus achieves the effects of speeding up processing and reducing computational complexity, making it particularly suitable for real-time processing scenarios of resource-limited devices such as autonomous vehicles.

[0043] 2. The present application uses adaptive learning of image degradation features under different fog conditions to greatly improve the generalization ability of the model, making it perform excellently under changing and extreme weather conditions, and significantly improving the effect of the prior art in processing unseen fog conditions.

[0044] 3、The present application does not rely on specific types of fog data, reduces the need for a large amount of labeled data, and achieves the effects of reducing data acquisition costs and improving practical application flexibility, especially suitable for scenes with limited data or high data costs.

[0045] 4、The present application significantly improves the accuracy and stability of image processing through advanced loss function design and recovery methods combining image features and prior knowledge, especially in rapidly changing fog scenes, ensuring image processing quality and consistency.

[0046] 5、The present application simplifies parameter adjustment and optimization process, uses dynamic selection experts and adaptive learning mechanism to reduce the need for complex parameter adjustment, making the model easy to apply in variable environments, reducing the technical use threshold and maintenance cost. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a schematic diagram of the method of the present application;

[0048] Figure 2 is a de-fogging effect diagram of the present application;

[0049] Figure 3 is a gain result diagram of the present application for different downstream tasks. DETAILED DESCRIPTION

[0050] In order to better illustrate the purpose and advantages of the present application, the following will further illustrate the content of the application combined with the drawings and examples. It should be pointed out that the implementation of the present application is not limited to the following examples, and any form of variation or change of the present application will fall within the scope of protection of the present application.

[0051] Given a degraded image I h affected by fog, our goal is to restore the clean and fog-free image I dh . The method is based on a pre-trained large multi-modal model to obtain image degradation prior adaptively restore the de-fogging image I dh . To obtain an image degradation severity representation We first set a text prompt Q, and then input the fogged image I h and Q into a large multi-modal model to query the image probability distribution p. Then we convert p to a specific rating level to obtain the image degradation prior We restore i h to i dh . Given i h , we first use a 3x3 convolution to obtain shallow features F s . Use F s and We use multiple MoE-SSM (MM) modules to obtain features Then, we use element-wise addition to combine the features and get F a Next, a 3×3 convolutional projection layer can be used to project the a Reconstructing the dehazed image I dh .

[0052] Example

[0053] like Figure 1 As shown, the present invention is a large-scale multimodal model guided adaptive image fog removal method, the specific implementation steps are as follows:

[0054] Step 1: Obtain degradation prior using a pre-trained large multimodal model;

[0055] Step 1.1: Define the cue word Q of the evaluation image to evaluate the quality of the blurred image;

[0056] Step 1.2: Input the cue word Q of the evaluation image and the fog image into the large multimodal model, and output the probability distribution shown in Equation (1);

[0057] p=M(I h ,Q),p∈R L (1)

[0058] where M(·,·) is the large multimodal model, L is the length of all tokens in the large multimodal model, and p is the output probability distribution.

[0059] Step 1.3: Construct a set of candidate rating levels The mapping relationship between the image quality rating method and the text-defined rating level is established as shown in formula (2);

[0060]

[0061] where G(·) is the index from the rating level defined by the text to the probability, j∈[1,N] and j∈

[0062] [1,L];

[0063] Step 1.4: Perform a closed set SoftMax operation on the output probability distribution p to obtain the degenerate prior as shown in Equation (3);

[0064]

[0065] in is the probability of the i-th level;

[0066] In the embodiment, {bad, poor, fair, good, excellent} is selected as the standard rating level, which is defined by ITU. In addition, in order to be suitable for the defogging task, an additional reference rating level {clear, fog, mist} describing the fog intensity is added.

[0067] Step 2: Input the fog image and degradation prior into the MoE-SSM model for training, which is used to dynamically adjust the model parameters and perform dynamic perception dehazing.

[0068] Step 2.1: Pass the fog image through the convolution layer of MoE-SSM to obtain the shallow feature F s ;

[0069] Step 2.2: Use LayerNorm layer to extract long-distance dependency features F from shallow features ln ;

[0070] Step 2.3: Dynamically select the module index based on the degradation prior using the method shown in formula (4);

[0071]

[0072] Among them, v represents Selected indices of the top K largest elements in ;

[0073] Step 2.4: Using the selected index v obtained in step 2.3, further refine features are extracted using the method shown in formula (5):

[0074]

[0075] Among them, the input features yes The i-th largest element in SSB v(i) (·) is the v(i)th SSB among all candidate SSM modules, SSB is the SSM module, It is an element-by-element multiplication; SSB is used to model long-distance pixel dependencies. Each SSB consists of a depthwise convolution layer, a SiLU activation function, a 2D-SSM layer, and a LayerNorm layer in sequence; according to the degradation prior Adopting adaptive strategies to select different SSBs;

[0076] Step 2.5: The refined features and shallow features are combined to obtain fusion features as shown in formula (6);

[0077]

[0078] Among them, TK(·,·) represents the top K expert selection mechanism, Linear(·) and They are the fully connected layer and element-by-element multiplication. According to the traditional Transformer architecture, we will s and F d Do element-wise addition and pass the result through LayerNorm, convolutional layer, and attention layer in sequence to get the output.

[0079] Step 2.6: Loop through steps 2.1 to 2.5 and use the formula (7) to obtain the optimized features.

[0080]

[0081] Among them, F s It is obtained from I through the convolution layer h The obtained features, δ is the learnable scaling factor, H is the number of MM modules used; MM represents a single module of MoE-SSM, that is, a single module of the hybrid expert-state space model; MM(·,·,·) represents the processing of multiple MM modules;

[0082] Step 3: Utilize shallow features F s and optimization features Perform fusion to reconstruct a clean and fog-free image;

[0083] Step 4: Use the clean, haze-free image and the ground-truth clear image to train the neural network using the Charbonnier loss function shown in formula (8), which is used to penalize the deviation between the restored image and the ground-truth clear image and encourage consistent image gradients;

[0084]

[0085] where ∈=10-3 is a constant in all experiments.

[0086] To verify the effectiveness of the present invention, the present invention was compared with 13 state-of-the-art dehazing methods. Specifically, DCP, DehazeNet, AODNet, GridDehazeNet, FFANet, AECRNet, Dehamer, DehazeFormer, SLP, MB-TaylorFormer, DEANet, MambaIR, and the all-in-one method LDR were used for comparative evaluation, and experiments were conducted on the LMHaze and Dense-Haze datasets. Peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learning-perceptual image block similarity (LPIPS) were used as evaluation indicators. In addition, LMM was also used to evaluate the dehazed images, and its quality assessment score (QAS) was selected as a supplement to the evaluation indicators. In terms of implementation details, the present invention is completely based on Pytorch implementation, and all experiments are performed on a single NVIDIA RTX 3090 GPU. Our MoE-Mamba adopts a four-stage encoder-decoder structure. From the first to the fourth level, the number of MM modules is {4, 6, 6, 8}, respectively, and the values ​​of parameters N and K are {14, 1, 1, 14} and {7, 1, 1, 7}, respectively. We use the AdamW optimizer for training, with a batch size of 4 and a total number of training iterations of 2×10^5. The learning rate decays from 2×10^-4 to 1×10^-5 using a cosine annealing strategy. During training, we randomly crop images to 256×256, but do not crop them during testing. For the LMM in the model, we use Q-Align, which is fine-tuned based on mPLUG-Owl2.

[0087] To further demonstrate the superiority of the present invention, the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learning-based perceptual image patch similarity (LPIPS), and quality assessment score (QAS) of the obtained dehazed image and the clean, haze-free image were calculated as evaluation indicators, and the effect of the present invention was verified by subjective and objective experiments.

[0088] (1) Comparison with SOTA methods. The comparison results on the LMHaze dataset are shown in Table 1 and Figure 2As shown. Compared with the second-ranked LDR method, we improved the PSNR / SSIM indicators by 2.9% / 6.7% respectively. Compared with the SOTA Mamba architecture image restoration method, we improved the PSNR / SSIM by 3.1% / 6.7% respectively. This result shows the advantage of our model in dynamic parameter selection by using image degradation prior. In addition, we also achieved the best results on the LMM-based evaluation index QAS, indicating that the dehazed images processed by our method are better in visual perception. The comparison results of the Dense-Haze dataset are shown in the right column of Table 2 (LMHaze is not used). The method of the present invention also achieved the best performance on the Dense-Haze dataset.

[0089] Table 1 Quantitative comparison of this patent and other dehazing methods on the LMHaze dataset

[0090]

[0091] Table 2 Quantitative comparison of this patent and other dehazing methods on the Dense-Haze dataset

[0092]

[0093] (2) Downstream task validation. One goal of image dehazing is to serve high-level vision tasks, so it is crucial to evaluate the performance of dehazing methods in these tasks. We selected three typical tasks and their related SOTA methods as benchmarks: object detection using DEQDet, image segmentation using SAM, and image description generation using LLaVA.

[0094] like Figure 3 As shown in Figure 2, current state-of-the-art baseline methods still perform poorly in dense fog, failing to detect bicycles. When we feed dehazed images into these baseline methods for downstream tasks, we observe significant performance gains. This finding highlights the importance of image dehazing as a preprocessing step. Notably, images dehazed using our MoE-Mamba framework show improved performance on all three downstream tasks.

[0095] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large-scale multimodal model-guided adaptive image haze removal method, characterized by: The following steps are included: Step 1: Obtain degradation prior using a pre-trained large multimodal model; Step 2: Input the fog image and degradation prior into the MoE-SSM model for training, which is used to dynamically adjust the model parameters and perform dynamic perception dehazing. Step 2.1: Pass the fog image through the convolution layer of MoE-SSM to obtain the shallow feature F s ; Step 2.2: Use LayerNorm layer to extract long-distance dependency features F from shallow features ln ; Step 2.3: Dynamically select the module index based on the degradation prior using the method shown in formula (4); Among them, v represents Selected indices of the top K largest elements in ; Step 2.4: Using the selected index v obtained in step 2.3, further refine features are extracted using the method shown in formula (5): Among them, the input features yes The i-th largest element in SSB v(i) (·) is the v(i)th SSB among all candidate SSM modules, SSB is the SSM module, It is element-by-element multiplication; SSB is used to model long-distance pixel dependencies; each SSB contains a depth convolution layer, a SiLU activation function, a 2D-SSM layer and a LayerNorm layer in sequence; according to the degradation prior Adopting adaptive strategies to select different SSBs; Step 2.5: The refined features and shallow features are combined to obtain fusion features as shown in formula (6); Among them, TK(·,·) represents the top K expert selection mechanism, Linear(·) and They are the fully connected layer and element-by-element multiplication; according to the traditional Transformer architecture, we will F s and F d Do element-wise addition and pass the result through LayerNorm, convolutional layer and attention layer in sequence to get the output; Step 2.6: Loop through steps 2.1 to 2.5 and use the formula (7) to obtain the optimized features. Among them, F s It is obtained from I through the convolution layer h The obtained features, δ is the learnable scaling factor, H is the number of MM modules used; MM represents a single module of MoE-SSM, that is, a single module of the hybrid expert-state space model; MM(·,·,·) represents the processing of multiple MM modules; Step 3: Utilize shallow features F s and optimization features Perform fusion to reconstruct a clean and fog-free image; Step 4: Use the clean and haze-free image and the ground-truth clear image to train the neural network using the Charbonnier loss function shown in formula (8), which is used to penalize the deviation between the restored image and the ground-truth clear image and encourage consistent image gradients; where ∈=10-3 is a constant in all experiments.

2. The large-scale multimodal model-guided adaptive image haze removal method according to claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Define the cue word Q of the evaluation image to evaluate the quality of the blurred image; Step 1.2: Input the cue word Q of the evaluation image and the fog image into the large multimodal model, and output the probability distribution shown in Equation (1); p=M(I h ,Q),p∈R L (1) Where M(·,·) is the large multimodal model, L is the length of all tokens in the large multimodal model, and p is the output probability distribution; Step 1.3: Construct a set of candidate rating levels The mapping relationship between the image quality rating method and the text-defined rating level is established as shown in formula (2); where G(·) is the mapping from the rating level defined by the text to the index of the probability, j∈[1,N] and j∈[1,L]; Step 1.4: Perform a closed set SoftMax operation on the output probability distribution p to obtain the degenerate prior as shown in Equation (3); in is the probability of the ith level.

Images