Lightweight image defogging method based on cross-stage local connection

Abstract

The application provides a kind of light image defogging method based on cross-stage local connection, comprising the following steps: obtaining sample clear image data and the corresponding synthetic fog image of the sample clear image data as target training set;The data of the target training set is preprocessed to obtain an input feature image;Construct an image defogging network model based on cross-stage local connection, including local residual combination module, adaptive feature fusion module, deformable convolution module;The input feature image is extracted using the local residual combination module, then the fusion feature is output through the adaptive feature fusion module, and finally the sample network defogging image is reconstructed and output through the deformable convolution module;Train the initial image defogging network model, calculate the reconstruction loss and the contrast loss, and update the parameters of the initial image defogging network model by back propagation;Obtain the trained image defogging network model.

Description

Technical Field

[0001] This application relates to deep learning and image processing, and in particular to a lightweight image dehazing method based on cross-stage local connectivity. Background Technology

[0002] Digital images, as a medium for recording the human visual system, are already closely related to our daily lives. Computer vision has become an important research direction in the field of artificial intelligence in recent years, thus playing a vital role in many areas of our daily lives, such as security monitoring, mobile devices, healthcare, intelligent transportation, and aerospace. The process of digital image processing essentially involves converting the signals captured by imaging sensors into an image consistent with the current shooting scene.

[0003] However, due to various factors such as sensor quality, photographer's skill, shooting scene, and imaging environment, the final image often fails to clearly and accurately record the effective information in the scene during the imaging process. Such low-quality digital images not only fail to meet human visual needs intuitively but also negatively impact the processing performance of subsequent advanced computer vision tasks. Furthermore, in hazy or smoggy weather, suspended particles in the air (fog, haze, water-soluble substances, and dust, etc.) absorb light from objects, causing refraction and scattering. Therefore, images acquired in hazy or smoggy weather tend to be grayish-white, lack detail, and suffer from reduced contrast and recognizability, directly hindering their application and processing performance in many fields.

[0004] For example, in security monitoring and autonomous driving, foggy images can cause significant interference with information acquisition, hindering subsequent tasks such as target detection or facial recognition. In military reconnaissance, where image accuracy is paramount, the loss of information in foggy images can lead to substantial losses. A clear, high-quality image accurately records scene color contrast, background structure, and the boundaries of moving objects. However, in foggy images created by adverse weather conditions, the reduced contrast makes objects difficult to discern.

[0005] Therefore, it is of great practical significance to recover useful features from foggy images, eliminate the interference of weather factors, enhance the clarity of images, and improve the robustness of the visual system. Summary of the Invention

[0006] Exemplary embodiments of this application provide a lightweight image dehazing method based on cross-stage local connectivity, so as to at least achieve the technical effect of recovering useful features in the image based on information in the hazy image and enhancing the image clarity.

[0007] Exemplary embodiments of this application provide a lightweight image dehazing method based on cross-stage local connectivity. The method includes the following steps: acquiring clear sample image data and corresponding synthetic hazy images as a target training set; preprocessing the data in the target training set to obtain an input feature image; constructing an image dehazing network model based on cross-stage local connectivity, including a local residual combination module, an adaptive feature fusion module, and a deformable convolution module; using the local residual combination module to extract features from the input feature image, then outputting fused features through the adaptive feature fusion module, and finally reconstructing and outputting a sample network dehazed image through the deformable convolution module; training an initial image dehazing network model, calculating reconstruction loss and contrast loss, and updating the parameters of the initial image dehazing network model through backpropagation; obtaining the trained image dehazing network model and performing dehazing operations on the target image.

[0008] In one embodiment, the preprocessing of the target training set data includes image pairing, image cropping, and data augmentation; the image pairing involves matching clear sample image data with corresponding synthetic foggy images to form data pairs; the image cropping involves cropping the training samples of the target training set to a size of 240×240; the data augmentation involves normalizing the pixel values ​​of the training samples of the target training set to the range [-1, 1], while simultaneously augmenting the cropped images by randomly horizontally flipping and rotating them by 90°.

[0009] In one embodiment, the step of using the local residual combination module to extract features from the input feature image, then outputting fused features through the adaptive feature fusion module, and finally reconstructing and outputting the sample network dehazed image through the deformable convolution module includes: the local residual combination module contains three local residual combinations, configured such that each local residual combination consists of six cross-stage residual attention modules and one convolutional layer, connected in a residual manner; the cross-stage residual attention consists of a cross-stage local connection structure and a feature attention module, wherein the cross-stage local connection structure contains three residual blocks and two dilated convolutions, and the feature attention module contains channel attention and spatial attention; after the feature extraction stage, in each cross-stage residual attention, features are gradually separated through channel separation and finally fused across stages, while different weights are assigned to different channel and spatial features.

[0010] In one embodiment, the step of using the local residual combination module to extract features from the input feature image, then outputting fused features through the adaptive feature fusion module, and finally reconstructing and outputting the sample network dehazed image through the deformable convolution module includes: the adaptive feature fusion module is configured to assign different weights to the multi-level features output by different local residual combinations and perform adaptive fusion.

[0011] In one embodiment, the step of using the local residual combination module to extract features from the input feature image, then outputting fused features through the adaptive feature fusion module, and finally reconstructing and outputting the sample network dehazed image through the deformable convolution module includes: the deformable convolution module contains two deformable convolutional layers and two ordinary convolutional layers, configured to perform feature enhancement during the reconstruction stage, and then perform alignment summation with the input feature image in the final stage to connect them, ultimately outputting the sample network dehazed image.

[0012] In one embodiment, the reconstruction loss specifically includes: calculating the reconstruction loss using the L1 distance loss function based on the output dehazed image from the sample network and the corresponding haze-free image in the dataset; on the dehazed dataset, for the dehazed image output by the network... The loss function for the corresponding sharp baseline image J is expressed as:

[0013]

[0014] In one embodiment, the contrast loss specifically includes: using the output dehazed image of the sample network as an anchor point, the corresponding hazy image in the dataset as a positive sample, and the hazy image as a negative sample, to calculate the contrast loss:

[0015]

[0016] Among them G i Let i = 1, 2, ..., M be the hidden features extracted from the VGG19 pre-trained model in the latent feature space, D(x,y) be the L1 distance between x and y, and ω be the hidden features at the i-th layer. i These are the weight coefficients of different layers in the feature space. This loss function makes the dehazed image closer to the positive sample and farther away from the negative sample in the feature space.

[0017] In one embodiment, training the initial image dehazing network model involves calculating reconstruction loss and contrast loss, and then backpropagating to update the parameters of the initial image dehazing network model. This includes combining the reconstruction loss and contrast loss, with the total loss function being:

[0018] L = L rec +λL c ,

[0019] Where L rec L represents the reconstruction loss function. c Let λ represent the comparison loss function, and let λ represent the hyperparameter that balances the two loss functions. During the training iteration cycle, the network model parameters are updated by backpropagation by minimizing this total loss function.

[0020] This application offers the following advantages: To design a lightweight network structure and further improve dehazing performance, this invention provides a lightweight image dehazing method based on cross-stage local connections. This lightweight network model uses a cross-stage residual attention module as the basic unit of the dehazing network to improve model efficiency. This module consists of cross-stage local connections and feature attention. Subsequently, to address the issues of increased parameters and loss of influence of multi-level features during feature fusion, this invention provides an adaptive feature fusion method. Furthermore, considering the limited receptive field of traditional convolutional kernels with fixed grids, this invention employs deformable convolution for dynamic feature enhancement. Finally, this invention introduces a contrastive loss function, substituting the output image into the feature space to calculate its distance to positive and negative samples, further enriching the supervision information. The lightweight image dehazing method provided by this invention achieves superior dehazing performance on different datasets while maintaining a low parameter count. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a flowchart of the general implementation method in one embodiment of this application;

[0023] Figure 2 This is an overall network structure diagram in one embodiment of this application;

[0024] Figure 3 This is a structural diagram of the cross-stage residual attention module proposed in one embodiment of this application;

[0025] Figure 4 This is a structural diagram of the feature attention module proposed in one embodiment of this application;

[0026] Figure 5 This is a comparison diagram of image dehazing algorithms in one embodiment of this application. Detailed Implementation

[0027] Current research on image dehazing mainly falls into three categories: image enhancement-based, image restoration-based, and deep learning-based dehazing algorithms. Image enhancement-based dehazing algorithms primarily rely on the distribution of image pixels to reduce noise or increase contrast, but these methods are prone to causing color distortion. Image restoration-based methods, on the other hand, are based on the imaging principles of hazy weather conditions, using atmospheric scattering models to derive solutions for hazy images and ultimately output clear, haze-free images. Solving the model itself involves many unknown parameters (such as atmospheric light intensity and transmittance). Therefore, it is necessary to mine the prior knowledge of the image itself through statistical regularities to estimate the parameters and finally substitute them into the model to achieve the purpose of dehazing. However, when the foggy scene does not meet the above prior knowledge, the dehazing algorithm based on image restoration will lead to problems such as color distortion, fog residue, and halo effect at abrupt changes in depth of field. The dehazing method based on deep learning can estimate the parameters in the atmospheric scattering model or directly learn the mapping between foggy and fog-free images in an end-to-end manner to achieve dehazing. Relying on the powerful feature learning ability of convolutional neural networks, the deep learning method overcomes the shortcomings of traditional methods. The network can learn features by itself through training data, and has better dehazing effect and universality.

[0028] However, current advanced image dehazing networks often involve complex network structures and large model parameters, significantly impacting their operational efficiency. Furthermore, existing dehazing networks typically only consider clear, haze-free images as positive samples to guide network training, neglecting the importance of negative samples. Contrastive learning, a type of self-supervised learning, is characterized by its independence from manually labeled class information, directly utilizing the data itself as supervisory information to learn knowledge from sample data and apply it to downstream tasks. Contrastive learning is relatively less complex; its main idea is to compare data in the feature space between positive and negative samples, minimizing the distance between similar samples and maximizing the distance between negative samples, thereby learning the feature representation of the samples. Addressing the aforementioned issues, researching how to reduce model parameters and further improve dehazing performance has significant application value and practical implications.

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] like Figure 1 As shown, this application provides a lightweight image dehazing method based on cross-stage local connectivity, the method comprising:

[0031] S102, acquire clear sample image data and the corresponding synthetic foggy image as the target training set;

[0032] S104, preprocess the data of the target training set to obtain the input feature image;

[0033] S106, Construct an image dehazing network model based on cross-stage local connectivity, including a local residual combination module, an adaptive feature fusion module, and a deformable convolution module; use the local residual combination module to extract features from the input feature image, then use the adaptive feature fusion module to output fused features, and finally use the deformable convolution module to reconstruct and output the sample network dehazing image.

[0034] S108, Train the initial image dehazing network model, calculate the reconstruction loss and contrast loss, and update the parameters of the initial image dehazing network model through backpropagation;

[0035] S110: Obtain the trained image dehazing network model and perform dehazing operation on the target image.

[0036] It should be noted that this embodiment uses the ITS or OTS training set in RESIDE for training, where the dataset contains clear images and corresponding synthetic foggy image data pairs.

[0037] In one embodiment, the step of preprocessing the data of the target training set includes image pairing, image cropping, and data augmentation, specifically:

[0038] S1, the image pairing is to match the clear sample image data with the corresponding synthetic hazy image data one by one to form a data pair;

[0039] S2, the image cropping cropped the training samples of the target training set to a size of 240×240;

[0040] S3, the data augmentation is to normalize the pixel values ​​of the training samples in the target training set to the range [-1, 1], while simultaneously augmenting the cropped image by randomly horizontally flipping and rotating it by 90°.

[0041] It should be noted that, to prevent overfitting during training and improve the network's robustness, the training input samples will be cropped to a size of 240×240. This can be adjusted according to the actual situation. Normalizing pixel values ​​to the range [-1, 1] prevents excessive differences in input image pixel values, which could lead to difficulties in network convergence. Simultaneously, the cropped images are augmented through random horizontal flipping and 90° rotation to increase sample richness.

[0042] In one embodiment, the steps of extracting features from the input feature image using the local residual combination module, outputting fused features through the adaptive feature fusion module, and finally reconstructing and outputting the dehazed sample network image through the deformable convolution module include:

[0043] The local residual combination module includes three local residual combinations, configured such that each local residual combination consists of six cross-stage residual attention modules and one convolutional layer, connected in a residual manner. The cross-stage residual attention consists of a cross-stage local connection structure and a feature attention module. The cross-stage local connection structure contains three residual blocks and two dilated convolutions, while the feature attention module contains channel attention and spatial attention. During the feature extraction stage, in each cross-stage residual attention, features are gradually separated through channel separation and finally fused across stages, while different weights are assigned to different channel and spatial features.

[0044] It should be noted that, as Figure 2 , Figure 3 As shown, this lightweight network model comprises four steps: preprocessing, feature extraction, feature fusion, and reconstruction. The preprocessing stage includes a single ordinary convolutional layer; the feature extraction stage includes three local residual combinations, each consisting of six cross-stage residual attention modules and a convolutional layer, connected via residuals.

[0045] In one embodiment, the step of extracting features from the input feature image using the local residual combination module, outputting fused features through the adaptive feature fusion module, and finally reconstructing and outputting the dehazed sample network image through the deformable convolution module includes:

[0046] The adaptive feature fusion module is configured to assign different weights to the multi-level features output by different combinations of local residuals and perform adaptive fusion.

[0047] In one embodiment, the step of extracting features from the input feature image using the local residual combination module, outputting fused features through the adaptive feature fusion module, and finally reconstructing and outputting the dehazed sample network image through the deformable convolution module includes:

[0048] The deformable convolutional module includes two deformable convolutional layers and two ordinary convolutional layers, configured to perform feature enhancement during the reconstruction stage, and then perform positional summation with the input feature image in the final stage to connect them, ultimately outputting a dehazed sample network image.

[0049] It should be noted that the foggy image undergoes a preprocessing stage to obtain the input feature image; in the feature extraction stage, features are progressively separated through channel separation in each cross-stage residual attention and finally fused across stages, while different weights are assigned to different channels and spatial features; in the feature fusion stage, different weights are assigned to the multi-level features output by different local residual combinations and adaptive fusion is performed; in the reconstruction stage, deformable convolutions are used for feature enhancement and are finally connected to the input foggy image in the final stage to output the dehazed image of the network.

[0050] In one embodiment, based on the output dehazed image from the sample network and the corresponding haze-free image in the dataset, a reconstruction loss is calculated using the L1 distance loss function. On the dehazed dataset, for the dehazed image output by the network... The loss function for the corresponding sharp baseline image J is expressed as:

[0051]

[0052] In one embodiment, the contrast loss specifically includes: using the output dehazed image of the sample network as an anchor point, the corresponding hazy image in the dataset as a positive sample, and the hazy image as a negative sample, to calculate the contrast loss:

[0053]

[0054] Among them G i Let i = 1, 2, ..., M be the hidden features extracted from the VGG19 pre-trained model in the latent feature space, D(x,y) be the L1 distance between x and y, and ω be the hidden features at the i-th layer. i These are the weight coefficients of different layers in the feature space. This loss function makes the dehazed image closer to the positive sample and farther away from the negative sample in the feature space.

[0055] It should be noted that, as an optional implementation, the image dehazing network was trained and tested on an NVIDIA GeForce RTX 3090 graphics card during training. The network was built and trained on the PyTorch deep learning framework, using the ADAM optimizer with exponential decay rates set to 0.9 and 0.999. The batch size and total number of epochs were set to 4 and 200, respectively. The initial learning rate was set to 0.0001, and a cosine annealing strategy was used to adjust the learning rate during training. As an example, features from layers 1, 3, 5, 9, and 13 of the pre-trained VGG19 model were extracted, with coefficients corresponding to 1 / 32, 1 / 16, 1 / 8, 1 / 4, and 1, respectively. This loss function makes the dehazed image more closely approximate positive samples and further away from negative samples in the feature space.

[0056] In one embodiment, training the initial image dehazing network model involves calculating reconstruction loss and contrast loss, and then backpropagating to update the parameters of the initial image dehazing network model. This includes combining the reconstruction loss and contrast loss, with the total loss function being:

[0057] L = L rec +λL c ,

[0058] Where L rec L represents the reconstruction loss function. c Let λ represent the comparison loss function, and let λ represent the hyperparameter that balances the two loss functions. During the training iteration cycle, the network model parameters are updated by backpropagation by minimizing this total loss function.

[0059] It should be noted that the aforementioned trained network model is used for dehazing; the input is a hazy image, and the output is a corresponding dehazed image, such as... Figure 5 As shown, a cross-stage residual attention module is used as the basic unit of the dehazing network to improve network efficiency. Subsequently, multi-level features are weighted and fused using an adaptive feature fusion module to address the issues of increased parameters and loss of influence of multi-level features. Next, deformable convolution is used for dynamic feature enhancement to solve the problem of limited receptive field caused by fixed grids. Finally, the concept of contrastive learning is introduced, and a novel loss function is proposed by combining reconstruction loss and contrastive loss to address the problem of insufficient supervision information. The method provided in this invention further improves dehazing performance while maintaining a relatively small number of model parameters.

[0060] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications based on these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of this application.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A lightweight image dehazing method based on cross-stage local connectivity, characterized in that, The method includes the following steps: Acquire clear sample image data and corresponding synthetic foggy images as the target training set; The data in the target training set is preprocessed to obtain the input feature image; A cross-stage local connectivity-based image dehazing network model is constructed, including a local residual combination module, an adaptive feature fusion module, and a deformable convolution module. The local residual combination module is used to extract features from the input feature image, the adaptive feature fusion module outputs fused features, and finally the deformable convolution module reconstructs and outputs the sample network dehazing image. Train an initial image dehazing network model, calculate reconstruction loss and contrast loss, and update the parameters of the initial image dehazing network model through backpropagation; Obtain a trained image dehazing network model and perform dehazing operation on the target image; in: The local residual combination module includes three local residual combinations, configured such that each local residual combination consists of six cross-stage residual attention modules and one convolutional layer, connected in a residual manner. The cross-stage residual attention consists of a cross-stage local connection structure and a feature attention module. The cross-stage local connection structure includes three residual blocks and two dilated convolutions, while the feature attention module includes channel attention and spatial attention. During the feature extraction stage, in each cross-stage residual attention, features are gradually separated through channel separation and finally fused across stages, while different weights are assigned to different channel and spatial features. The adaptive feature fusion module is configured to assign different weights to the multi-level features output by different combinations of local residuals and perform adaptive fusion. The deformable convolutional module includes two deformable convolutional layers and two ordinary convolutional layers, configured to perform feature enhancement during the reconstruction stage, and then perform positional summation with the input feature image in the final stage to connect them, ultimately outputting a dehazed sample network image.

2. The lightweight image dehazing method based on cross-stage local connectivity according to claim 1, characterized in that, The preprocessing of the target training set data includes image pairing, image cropping, and data augmentation. The image pairing involves matching clear sample image data with corresponding synthetic hazy images to form data pairs. The image cropping cropped the training samples of the target training set to a size of 240×240; The data augmentation involves normalizing the pixel values ​​of the training samples in the target training set to the range [-1, 1], while simultaneously augmenting the cropped images by randomly horizontally flipping and rotating them by 90°.

3. The lightweight image dehazing method based on cross-stage local connectivity according to claim 2, characterized in that, The reconstruction loss specifically includes: calculating the reconstruction loss using the L1 distance loss function based on the output dehazed image from the sample network and the corresponding haze-free image in the dataset. On the dehazed dataset, for the dehazed image output by the network... The loss function for the corresponding sharp baseline image J is expressed as:

4. The lightweight image dehazing method based on cross-stage local connectivity according to claim 3, characterized in that, The contrast loss specifically includes: using the output dehazed image of the sample network as the anchor point, the corresponding haze-free image in the dataset as the positive sample, and the haze image as the negative sample, to calculate the contrast loss: Among them G i Let i = 1, 2, ..., M be the hidden features extracted from the VGG19 pre-trained model in the latent feature space, D(x,y) be the L1 distance between x and y, and w be the hidden features at the i-th layer. i These are the weight coefficients of different layers in the feature space. This loss function makes the dehazed image closer to the positive sample and farther away from the negative sample in the feature space.

5. The lightweight image dehazing method based on cross-stage local connectivity according to claim 4, characterized in that, The initial image dehazing network model is trained by calculating reconstruction loss and contrast loss, and then updating the parameters of the initial image dehazing network model via backpropagation. This includes combining the reconstruction loss and contrast loss, with the total loss function being: L=L rec +λL c , Where L rec L represents the reconstruction loss function. c Let λ represent the comparison loss function, and let λ represent the hyperparameter that balances the two loss functions. During the training iteration cycle, the network model parameters are updated by backpropagation by minimizing this total loss function.

Images