Self-adaptive image fog removal method guided by large-scale multi-modal model

Abstract

The invention discloses a self-adaptive image fog removal method guided by a large-scale multi-modal model, belongs to the technical field of visual information processing, and is applicable to the field of foggy day scene image processing. The implementation method of the invention comprises the following steps: 1, obtaining degradation prior by using a pre-trained large-scale multi-modal model; 2, inputting the fog image and the degradation priori into a MoE-SSM model for training, so as to dynamically adjust model parameters and further perform dynamic perception defogging; 3, fusing the shallow feature Fs and the optimized feature # imgabs0 #, and reconstructing a clean and fogless image; 4, training a neural network by using a clean and fogless image and a truth value clear image by using a Charbonier loss function shown in a formula (8), so as to punish the deviation between the recovered image and the truth value clear image and encourage a consistent image gradient; compared with the prior art, the problem of compromise processing of receptive fields and calculation efficiency in existing defogging model image processing is solved, meanwhile, the requirement for a large amount of annotation data is reduced, and then the image processing accuracy and stability are improved.

Description

Technical Field

[0001] The invention relates to an adaptive image fog removal method guided by a large multimodal model, belongs to the technical field of visual information processing, and is suitable for foggy scene image processing. Background Art

[0002] Image dehazing aims to restore clean and fog-free images from foggy images, which is crucial for advanced tasks that rely on visual information, such as autonomous driving and outdoor monitoring systems. Imaging in foggy weather will produce severe image degradation, which will significantly reduce the performance of the system. In order to meet the safety needs of all-weather services, developing an economical and effective image dehazing method has a wide range of application value.

[0003] A foggy image can be modeled as a masked additive combination of a clean fog-free image and a fog residual. In real-world scenes, especially outdoor scenes, obtaining the concentration of fog and the accurate depth of the scene is a challenging task. It is difficult for atmospheric scattering models to realistically present all foggy scenes in the real world. Some works use deep neural networks based on end-to-end learning to handle degradation in all foggy scenes, but current image dehazing networks face an irreconcilable trade-off between receptive field size and computational efficiency. In addition, the fixed network architecture of existing methods limits the diversity and processing methods of learning different fog forms, such as dense fog and non-uniform fog images that have not been encountered before.

[0004] At present, the existing fog image processing has the following problems. (1) High computational complexity and resource requirements: Due to the trade-off between receptive field size and computational efficiency, existing methods may face significant limitations in resource-limited devices or scenarios that require real-time processing. (2) Limited generalization ability: The model requires a large amount of training data labeled with specific weather conditions, which is expensive to obtain or the amount of data is limited. When encountering new types of rain, fog or other severe weather that are significantly different from the training data, the performance of existing technologies is limited.

[0005] Therefore, how to solve the trade-off between receptive field and computational efficiency in image processing of existing dehazing models has become an urgent problem to be solved. Summary of the invention

[0006] The purpose of the present invention is to solve the technical problem of the compromise between receptive field and computational efficiency in image processing of existing defogging models, and to propose an adaptive image defogging method guided by a large multimodal model.

[0007] The principle of the present invention is as follows: the present invention proposes to use a large multimodal model (LMM) to generate a degradation prior for foggy images, and use the Mamba architecture to construct an image defogging network to adaptively perform image restoration. By inferring the diverse fog morphology knowledge in the feature space of the large multimodal model, the fog morphology in the picture is adaptively estimated. In addition, the potential of the Mamba architecture in the image defogging task, the image defogging method based on the Mamba architecture is used to solve the trade-off between receptive field size and computational efficiency. At the same time, with the assistance of the mixed expert MoE model, the proposed method dynamically selects experts for restoration, thereby ensuring adaptive learning of specific fog knowledge, which is not limited by a fixed network architecture.

[0008] The objective of the present invention is achieved through the following technical solutions:

[0009] The present invention discloses an adaptive image fog removal method guided by a large multimodal model, which is applied to fog image restoration scenarios and includes the following steps:

[0010] Step 1: Obtain degradation prior using a pre-trained large multimodal model;

[0011] Step 1.1: Define the cue word Q of the evaluation image to evaluate the quality of the blurred image;

[0012] Step 1.2: Input the prompt word Q of the evaluation image and the fog image into the large multimodal model, and output the probability distribution shown in formula (1);

[0013] p=M(I h ,Q),p∈R L (1)

[0014] where M(·,·) is the large multimodal model, L is the length of all tokens in the large multimodal model, and p is the output probability distribution.

[0015] Step 1.3: Construct a set of candidate rating levels A mapping relationship between the image quality rating method and the text-defined rating level is established as shown in formula (2).

[0016]

[0017] where the mapping G(·) is an index from the rating level defined by the text to the probability, i∈[1,N] and j∈[1,L];

[0018] Step 1.4: Perform a closed set SoftMax operation on the output probability distribution p to obtain the degraded prior as shown in formula (3);

[0019]

[0020] in is the probability of the ith level;

[0021] Step 2: Input the fog image and degradation prior into the MoE-SSM model for training to dynamically adjust the model parameters and perform dynamic perception defogging;

[0022] Step 2.1: Pass the fog image through the convolution layer of MoE-SSM to obtain the shallow feature F s ;

[0023] Step 2.2: Use the LayerNorm layer to extract long-distance dependent features F from shallow features ln ;

[0024] Step 2.3: Dynamically select the module index based on the degradation prior using the method shown in formula (4);

[0025]

[0026] Among them, v represents Selected indices of the first K largest elements in ;

[0027] Step 2.4: Using the selected index v obtained in step 2.3, further refine the features by using the method shown in formula (5):

[0028]

[0029] Among them, the input features yes The i-th largest element in SSB v(i) (·) is the v(i)th SSB among all candidate SSM modules, where SSB is the SSM module. is an element-by-element multiplication; SSB is used to model long-distance pixel dependencies. Each SSB contains a deep convolution layer, a SiLU activation function, a 2D-SSM layer, and a LayerNorm layer in sequence; according to the degradation prior Adopt adaptive strategies to select different SSBs;

[0030] Step 2.5: The refined features and shallow features are combined to obtain fusion features as shown in formula (6);

[0031]

[0032] Among them, TK(·,·) represents the top K expert selection mechanism, Linear(·) and They are the fully connected layer and element-by-element multiplication. According to the traditional Transformer architecture, we will s and F d Do element-wise addition and pass the result through LayerNorm, convolutional layer, and attention layer sequentially to get the output.

[0033] Step 2.6: Loop through steps 2.1 to 2.5 to obtain the optimized features using the method shown in formula (7)

[0034]

[0035] Among them, F s It is passed through the convolutional layer from I h The obtained features, δ is the learnable scaling factor, H is the number of MM modules adopted; MM represents a single module of MoE-SSM, that is, a single module of the hybrid expert-state space model; MM(·,·,·) represents the processing of multiple MM modules;

[0036] Step 3: Using shallow features F s and optimization features Perform fusion to reconstruct a clean and fog-free image;

[0037] Step 4: Use the clean haze-free image and the true clear image to train the neural network using the Charbonnier loss function shown in formula (8), which is used to penalize the deviation between the restored image and the true clear image and encourage consistent image gradients;

[0038]

[0039] Here, ∈=10-3 is a constant in all experiments.

[0040] Beneficial effects:

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention combines a large multimodal model with a mixture of experts (MoE) model and the Mamba architecture to dynamically select appropriate experts for processing, thereby solving the trade-off between receptive field and computational efficiency in image processing and significantly reducing the consumption of computing resources, thereby achieving the effect of accelerating processing speed and reducing computational complexity. It is particularly suitable for real-time processing scenarios of resource-limited devices such as autonomous driving vehicles.

[0043] 2. The present invention utilizes adaptive learning of image degradation features under different fog forms, which greatly improves the generalization ability of the model, enabling it to perform well under changing and extreme weather conditions, and significantly improves the effect of existing technologies in processing unprecedented fog forms.

[0044] 3. The present invention does not need to rely on a specific type of fog data, reduces the need for a large amount of labeled data, and achieves the effect of reducing data acquisition costs and improving practical application flexibility. It is particularly suitable for scenarios with limited data or high data costs.

[0045] 4. The present invention significantly improves the accuracy and stability of image processing through advanced loss function design and a recovery method that combines image features and prior knowledge, especially in rapidly changing fog scenes, ensuring the quality and consistency of image processing.

[0046] 5. The present invention simplifies the parameter adjustment and optimization process, utilizes dynamic selection of experts and adaptive learning mechanisms, reduces the need for complex parameter adjustment, makes the model easy to apply in a changing environment, and reduces the threshold for use and maintenance costs of the technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the method of the present invention;

[0048] Figure 2 This is a defogging effect diagram of the present invention;

[0049] Figure 3 This is a graph showing the gain results of the present invention for different downstream tasks. DETAILED DESCRIPTION

[0050] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0051] Given a degraded image I affected by fog h , our goal is to restore the clean and haze-free image I dh This method obtains image degradation priors based on a pre-trained large multimodal model Adaptively restore the dehazed image I dh In order to obtain a representation of the severity of image degradation We first set a text prompt Q, and then the haze image I h and Q are input into a large multimodal model to query the image probability distribution p. We then transform p into a specific rating level to obtain the image degradation prior We use the following steps to h Restore to i dh Given i h , we first use 3×3 convolution to obtain the shallow feature F s . Using F s and We use multiple MoE-SSM (MM) modules to obtain features Then, we use element-wise addition to combine the features and get F a Next, a 3×3 convolutional projection layer can be used to project the a Reconstructing the dehazed image I dh .

[0052] Example

[0053] like Figure 1 As shown, the adaptive image fog removal method guided by a large multimodal model of the present invention has the following specific implementation steps:

[0054] Step 1: Obtain degradation prior using a pre-trained large multimodal model;

[0055] Step 1.1: Define the cue word Q of the evaluation image to evaluate the quality of the blurred image;

[0056] Step 1.2: Input the prompt word Q of the evaluation image and the fog image into the large multimodal model, and output the probability distribution shown in formula (1);

[0057] p=M(I h ,Q),p∈R L (1)

[0058] where M(·,·) is the large multimodal model, L is the length of all tokens in the large multimodal model, and p is the output probability distribution.

[0059] Step 1.3: Construct a set of candidate rating levels A mapping relationship between the image quality rating method and the text-defined rating level is established as shown in formula (2).

[0060]

[0061] where G(·) is the index from the rating level defined by the text to the probability, j∈[1,N] and j∈

[0062] [1,L];

[0063] Step 1.4: Perform a closed set SoftMax operation on the output probability distribution p to obtain the degraded prior as shown in formula (3);

[0064]

[0065] in is the probability of the ith level;

[0066] In the embodiment, {bad, poor, fair, good, Excellent} is selected as the standard rating level, which is defined by ITU. In addition, in order to be suitable for the defogging task, an additional reference rating level {clear, fog, mist} describing the fog intensity is added.

[0067] Step 2: Input the fog image and degradation prior into the MoE-SSM model for training to dynamically adjust the model parameters and perform dynamic perception defogging;

[0068] Step 2.1: Pass the fog image through the convolution layer of MoE-SSM to obtain the shallow feature F s ;

[0069] Step 2.2: Use the LayerNorm layer to extract long-distance dependent features F from shallow features ln ;

[0070] Step 2.3: Dynamically select the module index based on the degradation prior using the method shown in formula (4);

[0071]

[0072] Among them, v represents Selected indices of the first K largest elements in ;

[0073] Step 2.4: Using the selected index v obtained in step 2.3, further refine the features by using the method shown in formula (5):

[0074]

[0075] Among them, the input features yes The i-th largest element in SSB v(i) (·) is the v(i)th SSB among all candidate SSM modules, where SSB is the SSM module. is an element-by-element multiplication; SSB is used to model long-distance pixel dependencies. Each SSB contains a deep convolution layer, a SiLU activation function, a 2D-SSM layer, and a LayerNorm layer in sequence; according to the degradation prior Adopt adaptive strategies to select different SSBs;

[0076] Step 2.5: The refined features and shallow features are combined to obtain fusion features as shown in formula (6);

[0077]

[0078] Among them, TK(·,·) represents the top K expert selection mechanism, Linear(·) and They are the fully connected layer and element-by-element multiplication. According to the traditional Transformer architecture, we will s and F d Do element-wise addition and pass the result through LayerNorm, convolutional layer, and attention layer sequentially to get the output.

[0079] Step 2.6: Loop through steps 2.1 to 2.5 to obtain the optimized features using the method shown in formula (7)

[0080]

[0081] Among them, F s It is passed through the convolutional layer from I h The obtained features, δ is the learnable scaling factor, H is the number of MM modules adopted; MM represents a single module of MoE-SSM, that is, a single module of the hybrid expert-state space model; MM(·,·,·) represents the processing of multiple MM modules;

[0082] Step 3: Using shallow features F s and optimization features Perform fusion to reconstruct a clean and fog-free image;

[0083] Step 4: Use the clean haze-free image and the true clear image to train the neural network using the Charbonnier loss function shown in formula (8), which is used to penalize the deviation between the restored image and the true clear image and encourage consistent image gradients;

[0084]

[0085] Here, ∈=10-3 is a constant in all experiments.

[0086] To verify the effectiveness of the present invention, the present invention is compared with 13 latest dehazing methods, specifically DCP, DehazeNet, AODNet, GridDehazeNet, FFANet, AECRNet, Dehamer, DehazeFormer, SLP, MB-TaylorFormer, DEANet, MambaIR and All-in-One method LDR are used for comparative evaluation, and experiments are carried out on LMHaze and Dense-Haze datasets. Peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and learning perceptual image block similarity (LPIPS) are used as evaluation indicators. In addition, LMM is also used to evaluate the dehazed image, and its quality assessment score (QAS) is selected as a supplement to the evaluation indicators. In terms of implementation details, the present invention is completely based on Pytorch implementation, and all experiments are carried out on a single NVIDIA RTX 3090 GPU. Our MoE-Mamba adopts a four-level encoder-decoder structure. From the first to the fourth level, the number of MM modules is {4, 6, 6, 8}, and the values ​​of parameters N and K are {14, 1, 1, 14} and {7, 1, 1, 7}, respectively. We use the AdamW optimizer for training, with a batch size of 4, a total number of training iterations of 2×10^5, and a learning rate decay from 2×10^-4 to 1×10^-5 according to the cosine annealing strategy. During training, we randomly cropped the image to 256×256 size, while no cropping was performed during testing. For the LMM in the model, we used Q-Align, which is fine-tuned based on mPLUG-Owl2.

[0087] In order to further demonstrate the superiority of the present invention, the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learning perceptual image patch similarity (LPIPS) and quality assessment score (QAS) of the obtained dehazed image and the clean haze-free image are calculated as evaluation indicators, and the effect of the present invention is verified by subjective and objective experiments.

[0088] (1) Comparison with SOTA methods. The comparison results on the LMHaze dataset are shown in Tables 1 and Figure 2As shown. Compared with the second-ranked LDR method, we improved the PSNR / SSIM indicators by 2.9% / 6.7% respectively. Compared with the SOTA Mamba architecture image restoration method, we improved the PSNR / SSIM by 3.1% / 6.7% respectively. This result shows the advantage of our model in dynamic parameter selection by using image degradation prior. In addition, we also achieved the best results on the LMM-based evaluation indicator QAS, indicating that the dehazed images processed by our method are better in visual perception. The comparison results of the Dense-Haze dataset are shown in the right column of Table 2 (LMHaze is not used). The method of the present invention also achieved the best performance on the Dense-Haze dataset.

[0089] Table 1 Quantitative comparison of this patent and other dehazing methods on the LMHaze dataset

[0090]

[0091] Table 2 Quantitative comparison of this patent and other dehazing methods on the Dense-Haze dataset

[0092]

[0093] (2) Downstream task verification. One goal of image dehazing is to serve high-level visual tasks, so it is crucial to evaluate the performance of dehazing methods in these tasks. We selected three typical tasks and their related SOTA methods as benchmarks: object detection using DEQDet, image segmentation using SAM, and image description generation using LLaVA.

[0094] like Figure 3 As shown in Figure 3, current SOTA baseline methods still perform poorly in dense fog conditions, such as failing to detect bicycles. When we feed the dehazed images into these baseline methods for downstream tasks, we observe significant performance gains. This finding highlights the importance of image dehazing as a preprocessing step. Notably, images dehazed using our MoE-Mamba framework perform better in all three downstream tasks.

[0095] The specific description above further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large multimodal model-guided adaptive image haze removal method, characterized by: The following steps are included: Step 1: Obtain degradation prior using a pre-trained large multimodal model; Step 2: Input the fog image and degradation prior into the MoE-SSM model for training to dynamically adjust the model parameters and perform dynamic perception defogging; Step 2.1: Pass the fog image through the convolution layer of MoE-SSM to obtain the shallow feature F s ; Step 2.2: Use the LayerNorm layer to extract long-distance dependent features F from shallow features ln ; Step 2.3: Dynamically select the module index based on the degradation prior using the method shown in formula (4); Among them, v represents Selected indices of the first K largest elements in ; Step 2.4: Using the selected index v obtained in step 2.3, further refine the features by using the method shown in formula (5): Among them, the input features yes The i-th largest element in SSB v(i) (·) is the v(i)th SSB among all candidate SSM modules, where SSB is the SSM module. is an element-by-element multiplication; SSB is used to model long-distance pixel dependencies; each SSB contains a deep convolutional layer, a SiLU activation function, a 2D-SSM layer, and a LayerNorm layer in sequence; according to the degradation prior Adopt adaptive strategies to select different SSBs; Step 2.5: The refined features and shallow features are combined to obtain fusion features as shown in formula (6); Among them, TK(·,·) represents the top K expert selection mechanism, Linear(·) and They are the fully connected layer and element-by-element multiplication respectively; according to the traditional Transformer architecture, we will s and F d Do element-wise addition and pass the result through LayerNorm, convolutional layer, and attention layer in sequence to get the output; Step 2.6: Loop through steps 2.1 to 2.5 to obtain the optimized features using the method shown in formula (7) Among them, F s It is passed through the convolutional layer from I h The obtained features, δ is the learnable scaling factor, H is the number of MM modules adopted; MM represents a single module of MoE-SSM, that is, a single module of the hybrid expert-state space model; MM(·,·,·) represents the processing of multiple MM modules; Step 3: Using shallow features F s and optimization features Perform fusion to reconstruct a clean and fog-free image; Step 4: Use the clean haze-free image and the true clear image to train the neural network using the Charbonnier loss function shown in formula (8), which is used to penalize the deviation between the restored image and the true clear image and encourage consistent image gradients; Here, ∈=10-3 is a constant in all experiments.

2. The large multimodal model-guided adaptive image haze removal method according to claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Define the cue word Q of the evaluation image to evaluate the quality of the blurred image; Step 1.2: Input the prompt word Q of the evaluation image and the fog image into the large multimodal model, and output the probability distribution shown in formula (1); p=M(I h ,Q),p∈R L (1) Where M(·,·) is the large multimodal model, L is the length of all tokens in the large multimodal model, and p is the output probability distribution; Step 1.3: Construct a set of candidate rating levels A mapping relationship between the image quality rating method and the text-defined rating level is established as shown in formula (2). where the mapping G(·) is the index from the rating level defined by the text to the probability, j∈[1,N] and j∈[1,L]; Step 1.4: Perform a closed set SoftMax operation on the output probability distribution p to obtain the degraded prior as shown in formula (3); in is the probability of the ith level.

Images