Semi-supervised learning image defogging method based on physical imaging model

Abstract

The present application relates to a kind of semi-supervised learning image defogging method based on physical imaging model, belong to computer vision and image processing technical field.The method includes: S1: design the light weight dense feature fusion network with attention mechanism, using feature extraction network sharing mechanism, using adaptive weight distribution module to estimate physical model parameters from degraded image;S2: atmospheric scattering model describes the imaging process of haze degraded image, proposes an improved physical model with adaptive information compensation, calculates and restores clear image;S3: design semi-supervised learning strategy, enhance the defogging performance of network in actual haze scene, weaken the domain difference between synthetic domain and real domain.The present application can solve the problem of single image defogging recovery with light algorithm complexity, and the superiority of the present application is proved by experiment verification in real data set.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image processing, and relates to a semi-supervised learning image defogging method based on a physical imaging model. BACKGROUND

[0002] As an important medium for information transmission in modern society, images are important carriers of modern information. However, actual imaging systems are easily affected by a large number of suspended particles in the atmosphere, resulting in degradation of the imaging image, which seriously affects the transmission of information by images. At the same time, the degraded image also has a bad influence on the normal operation of the computer vision system, such as the automatic driving system, the traffic monitoring system and the like. Therefore, it is of great value to the normal operation of various computer vision systems to recover a clear image from a degraded image, and at the same time, the clearification of the degraded image also restores the information amount of the image itself, which has practical significance for restoring image information.

[0003] The existing single image defogging methods can be divided into three categories: image enhancement defogging technology, image restoration defogging technology and deep learning image defogging technology. The defogging technology based on image enhancement improves the contrast of the image by using some traditional image enhancement methods, so that the image looks clearer in vision. The defogging technology based on image restoration considers the physical process of the degraded image, and restores based on the atmospheric scattering model. The defogging technology based on deep learning learns the mapping relationship from a large number of clear-foggy image data by constructing a neural network, which is the main technical method for processing single image defogging at present.

[0004] However, a large number of defogging networks based on deep learning use synthetic clear-foggy image data for training, which can achieve excellent defogging in synthetic fog images, but the defogging effect in real environment needs to be further improved. At the same time, the existing algorithms sharply increase the parameter amount of the network while improving the performance, which seriously affects the deployment and application of the defogging algorithm. SUMMARY

[0005] In view of the above technical deficiencies, the purpose of the present application is to provide a semi-supervised learning image defogging method based on a physical imaging model. The present application can complete the defogging task of a single image with a light-weight network structure. Through the proposed self-enhanced semi-supervised learning strategy, the problem of insufficient real scene defogging effect caused by training the defogging network on synthetic data is solved. A large number of experiments on commonly used image defogging test data sets prove the superiority of the technology.

[0006] The technical scheme adopted by the present application to solve its technical problems is:

[0007] The application discloses a semi-supervised learning image defogging method based on a physical imaging model, establishes a defogging system network structure including a feature extraction network and a defogging network, and iteratively performs supervised training and gradient optimization training of self-enhanced semi-supervised learning to generate an optimal image defogging system for actual image processing to achieve defogging, and comprises the following steps:

[0008] S1: synthesizing a clear-foggy image data set as a training set of the defogging system, and pre-processing the data set;

[0009] S2: constructing an image feature extraction network, that is, a feature fusion lightweight network with dense connection;

[0010] S3: constructing an image defogging network, that is, an improved physical imaging model with adaptive compensation information to restore a clear image;

[0011] S4: performing end-to-end supervised training on the network using the synthesized clear-foggy image data set to generate a defogging system with a preliminary image defogging effect, and further optimizing the image defogging system according to a self-enhanced semi-supervised learning strategy to enhance the defogging effect.

[0012] Step S1 specifically comprises the following steps:

[0013] S11: establishing a training data set, which includes a large number of clear-foggy image data pairs, denoted as (I, J), wherein I represents a foggy degraded image, and J represents a corresponding clear image;

[0014] S12: performing random cropping on the image pairs, performing data set enhancement, and fixing the size of the image to WxH;

[0015] S13: performing random flipping on the cropped images to further enhance the data set, and converting the images into a tensor form for inputting the network for training.

[0016] Step S2 of constructing the image feature extraction network specifically comprises the following steps:

[0017] S2a: the feature extraction network includes n convolutional structure groups, each convolutional structure group includes a dilated convolutional layer, an instance normalization layer, a spatial attention mechanism module and a Relu activation function, and only the first convolutional structure group does not include the spatial attention mechanism module;

[0018] S2b: the input of the first convolutional structure group is a degraded RGB image, the input channel number is 3, the channel number of the output feature map group is 6, and the input and output spatial sizes remain unchanged;

[0019] S2c: the input of the nth(n = 2, 3, 4) block convolution structure group is the spliced feature map group of the feature map group output by the previous (n-1) block convolution structure group, the number of input channels is (n-1) x 6, the number of channels of the output feature map group is 6, and the input and output spatial dimensions remain unchanged;

[0020] S2d: the output feature map groups of the n block convolution structure groups are spliced together to form a degraded image feature map group with a channel number of 6n.

[0021] In step S3, an image defogging network is constructed, specifically including the following steps:

[0022] S31: the 6n groups of feature map groups output by the shared feature extraction network are respectively assigned different weights through two adaptive weight distribution modules;

[0023] S32: the two groups of weighted 6n groups of feature map groups are respectively reduced to 3 channels through a convolution structure group, to obtain two parameter matrices K(x) and Δb(x) of the improved atmospheric scattering model containing adaptive compensation information;

[0024] S33: the clear image J is recovered according to the improved atmospheric scattering model;

[0025] The improved atmospheric scattering model is in the form of:

[0026]

[0027] Wherein, I(x) is the xth pixel value in the degraded image captured by the imaging device; J(x) is the xth pixel value of the clear image; A is the atmospheric light constant; t(x) is the transmittance matrix; K(x) and Δb(x) are parameters to be estimated; according to the improved atmospheric scattering model, the parameters K(x) and the direct information compensation Δb(x) are estimated from the image containing fog using a convolutional neural network, and then they are brought into the model to obtain the fog-free image.

[0028] The adaptive weight distribution module in step S31 is specifically:

[0029] The feature map group F c input into the module is subjected to one global average pooling and one global maximum pooling to obtain two groups of feature map groups G c with a size of C x 1 x 1;

[0030] The two groups of feature map groups are respectively subjected to one one-dimensional convolution operation with an input channel number of 1, and then the two groups of feature map groups are jointly subjected to one one-dimensional convolution operation with an input channel number of 2;

[0031] Finally, the weight α of the feature map group is determined by the Sigmoid function, and the feature map group is adaptively weighted according to the weight α;

[0032] The network is trained in step S4, specifically including the following steps:

[0033] S41: using the image data preprocessed in step S1 to perform end-to-end iterative supervised training on the input dehazing system network until the network converges, generating a pre-trained dehazing system with preliminary image dehazing effect.

[0034] The end-to-end supervised learning in step S41 uses a loss calculation network to calculate the perceptual loss and make it converge:

[0035] The loss calculation network uses a VGG16 feature extraction network pre-trained on the Image Net dataset and calculates the perceptual loss between the restored image and the clear reference image:

[0036]

[0037] wherein, φ j (C),j=1,2,3 respectively represent the use of VGG16 as a perceptual loss network for perceptual loss calculation in three channels, D j ,H j ,W j represent the size of the input image.

[0038] Step S42 enhances the dehazing effect of the dehazing system using a self-enhancing semi-supervised learning strategy, specifically including the following steps:

[0039] (1) input a foggy image into the pre-trained dehazing system to obtain Output 1; then input Output 1 into the system again to obtain Output 2;

[0040] (2) if the clear image Output 2 is better than Output 1, use Output 2 as a pseudo label to form a pair of data with the foggy image; determine whether the re-cleared image is better;

[0041] (3) use the paired data obtained in (2) to perform supervised learning on the pre-trained dehazing system network structure until no better dehazing effect can be obtained, stop the self-enhancing semi-supervised learning, and obtain an optimal image dehazing system; the optimal image dehazing system is used for dehazing processing on an input single foggy degraded image, and outputs a clear dehazed image.

[0042] The better image is determined by a two-class classifier based on the no-reference image evaluation index IL-NIQE, which calculates the IL-NIQE of the two images to be compared, and the one with lower value is determined as the better image.

[0043] The present application has the following advantages and benefits:

[0044] 1. The image defogging method proposed in the present application is based on a physical imaging model. The recovered image obtained by the recovery calculation based on the physical imaging model is more natural and realistic; the imaging model with adaptive compensation information has better adaptability to various unevenly illuminated environments; and the physical model also increases the explainability of the deep learning defogging network.

[0045] 2. The network structure proposed in the present application is light and efficient. The defogging network shares a feature extraction network, and the feature map groups are adaptively weighted based on an adaptive weight distribution module to realize efficient utilization of the feature map groups; the feature extraction network adopts a dense feature fusion lightweight network structure to realize the co-fusion of the overall information and the detail information of the input foggy degraded image.

[0046] 3. The semi-supervised defogging learning strategy proposed in the present application can enhance the defogging performance in the actual environment. The semi-supervised learning first uses the fog-clear image pair data for supervised learning, then generates pseudo labels by repeatedly defogging real fog images, and the pseudo labels and the fog images constitute a pair of data to optimize the network again. The self-enhanced semi-supervised learning strategy reduces the problem of essential difference between synthetic data and real environment, and further improves the defogging performance of the defogging network in the real scene.

[0047] 4. Through experimental verification on an open source dataset and defogging test in an actual environment, the defogging performance of the present application in the real environment has obvious advantages compared with other advanced methods; in addition, the defogging system structure of the present application is light and easy to deploy in small and medium-sized embedded systems. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a method flowchart of the present application;

[0049] Figure 2 is a whole structure diagram of the present application;

[0050] Figure 3 is a structure diagram of the adaptive weight distribution module;

[0051] Figure 4 is a structure diagram of the spatial attention mechanism module;

[0052] Figure 5 is a structure diagram of the self-enhanced semi-supervised learning strategy;

[0053] Figure 6 is a test result schematic diagram on the open source dataset O-HAZE;

[0054] Figure 7 is a test result schematic diagram in an actual daytime environment;

[0055] Figure 8 is a schematic diagram of test results in an actual nighttime environment. DETAILED DESCRIPTION

[0056] The application will be further described in conjunction with the embodiments. The method steps are described in conjunction with the drawings.

[0057] Referring to Figures 1-8 The application provides a semi-supervised learning image defogging method based on a physical imaging model. The method comprises an image feature extraction network, an image defogging network and a self-enhanced semi-supervised learning module, and specifically comprises the following steps:

[0058] S1: obtaining a synthetic clear-foggy image dataset RESIDE as a training set of the defogging system, and pre-processing the dataset, comprising the following steps:

[0059] S11: obtaining a group of clear-foggy image data pairs from the training dataset, denoted as (I, J), wherein I represents a foggy degraded image, and J represents a corresponding clear image.

[0060] S12: randomly cropping the image pair, performing dataset enhancement, and fixing the size of the image to 320x240.

[0061] S13: randomly flipping the cropped image, further enhancing the dataset, and converting it into a tensor form to prepare for input into the network for training.

[0062] S2: constructing an image feature extraction network: a feature fusion lightweight network with dense connection, comprising the following steps:

[0063] S21: the feature extraction network comprises four convolution structure groups, each of which is composed of a dilated convolution layer, an instance normalization layer, a spatial attention mechanism module and a Relu activation function, wherein only the first convolution structure group does not contain the spatial attention mechanism module.

[0064] Further, the detailed modules in the four convolution structure groups are:

[0065] (1) The design size of the dilated convolution is 3x3, the dilated rate is 2, and the convolution stride is 1. In addition, in order to ensure that the size of the output feature map is the same as the input size, the convolution zero padding design is 2.

[0066] (2) The instance normalization layer only normalizes a single channel of a single sample picture, which can accelerate convergence while ensuring that each image sample is independent, and is more conducive to bottom-level visual tasks than batch normalization and layer normalization.

[0067] (3) The spatial attention module enables the network to assign respective weights to each pixel in the feature map, so that the network pays attention to more important spatial detail information and enhances the network's ability to perceive local details. The implementation process of the spatial attention module is as follows:

[0068] Let the input feature map be F c , then F c ∈R C×H×W , where C represents the number of feature map groups, H and W represent the height and width of the feature map respectively. The weight matrix of the feature map is denoted as γ∈R 1×H×W . To obtain the weight γ, the input feature layer F c is pooled by taking the maximum and average of all feature points by channel, and the two results are stacked to obtain a feature map group G a with a size of 2xWxH. The channel number is adjusted from 2 to 1 using one convolution, and then the feature map is passed through a Sigmoid function to obtain the weight matrix γ.

[0069] (4) The Relu activation function adds non-linear mapping capability to the convolutional neural network.

[0070] S22: The input of the first block of convolutional structure group is the degraded RGB image, the input channel number is 3, the output feature map group has 6 channels, and the input and output spatial dimensions are kept as 320x240.

[0071] S23: The input of the second block of convolutional structure group is the feature map group output by the first block of convolutional structure group, the input channel number is 6, the output feature map group has 6 channels, and the input and output spatial dimensions are kept as 320x240.

[0072] S24: The input of the third block of convolutional structure group is the feature map group spliced by the feature map groups output by the first block of convolutional structure group and the second block of convolutional structure group, the input channel number is 12, the output feature map group has 6 channels, and the input and output spatial dimensions are kept as 320x240.

[0073] S25: The input of the fourth block of convolutional structure group is the feature map group spliced by the feature map groups output by the first block of convolutional structure group, the second block of convolutional structure group and the third block of convolutional structure group, the input channel number is 18, the output feature map group has 6 channels, and the input and output spatial dimensions are kept as 320x240.

[0074] S26: The output feature map groups of the four blocks of convolutional structure group are spliced together to form a degraded image feature map group with 24 channels.

[0075] S3: Constructing an image defogging network: according to the improved physical imaging model with adaptive compensation information, a clear image is recovered, which specifically includes the following steps:

[0076] S31: 24 groups of feature maps output by the shared feature extraction network are respectively assigned different weights by two adaptive weight distribution modules.

[0077] Further, the specific details of the adaptive weight distribution module are as follows:

[0078] The module assigns weights to the input feature map group F c with 24 groups in an adaptive manner, which is conducive to subsequent parameter matrix calculation through convolution layers. First, the input feature map group F c is subjected to global average pooling and global maximum pooling to obtain two groups of feature map groups G c with a size of Cx1x1. The two groups of feature map groups are respectively subjected to one-dimensional convolution with an input channel number of 1, and then the two groups of feature map groups are collectively subjected to one-dimensional convolution with an input channel number of 2. Finally, the weight a of the feature map group is determined by the Sigmoid function. The adaptive feature map group weight distribution is realized by the weight a.

[0079] S32: The two groups of weighted 24 groups of feature map groups are respectively reduced to 3 channels by one convolution structure group to obtain two groups of parameter matrices K(x) and Ab(x) of the improved atmospheric scattering model containing adaptive compensation information.

[0080] Further, the specific details of the convolution structure group are as follows:

[0081] (1) The design size of the ordinary convolution is 1x1, and the convolution stride is 1. In addition, in order to ensure that the size of the output feature map is the same as the input size, the convolution zero padding design is 0.

[0082] (2) The instance normalization layer, spatial attention module and step S21 are designed in the same way.

[0083] (3) The BRelu activation function limits the output range to ensure that the range of the output parameter matrix value is [0, 1].

[0084] S33: According to the improved atmospheric scattering model, the clear image is recovered, and the form of the improved atmospheric scattering model is as follows:

[0085]

[0086] where I(x) is the foggy degraded image, and J(x) is the clear image to be recovered. S4: The network is supervised and trained end-to-end using the synthetic image dataset RESIDE to form a dehazing system with preliminary image dehazing effect. In addition, in actual application, the dehazing effect is enhanced according to the self-enhanced semi-supervised learning strategy, which includes the following steps:

[0087] S41: using the image data preprocessed in step S1 to perform end-to-end supervised training on the network, forming a dehazing system with a preliminary image dehazing effect.

[0088] The overall structure of the network during training can be divided into two parts: a generation network and a loss network.

[0089] The generation network estimates the two parameters K(x) and Δb from the degraded image, and restores the clear image by bringing them into the physical imaging model.

[0090] The loss network uses a VGG16 feature extraction network pre-trained on the Image Net dataset, and calculates the perceptual loss between the restored image and the clear reference image: where the loss function of supervised learning adopts perceptual loss:

[0091]

[0092] where, φ j (C),j=1,2,3 respectively represent the use of VGG16 as a perceptual loss network in three channels for perceptual loss calculation, D j ,H j ,W j represent the size of the input image.

[0093] S42: perform self-enhanced semi-supervised learning on the O-HAZE dataset, and test the dehazing effect, with the specific details being:

[0094] Since the formation of foggy images in real environments is more complex, artificial synthetic datasets cannot fully represent all situations in real environments. In actual situations, a single dehazing by the network cannot achieve the best dehazing effect, and a self-enhanced semi-supervised learning strategy is adopted to further enhance the dehazing effect.

[0095] (1) dehaze each foggy image in the O-HAZE dataset to obtain Output 1; then use Output 1 as the input of the dehazing system and dehaze again to obtain Output 2.

[0096] (2) calculate the IL-NIQE of the two images, and if the value of Output 2 is low, then use Output 2 as a pseudo-label to form a set of paired image data with the original foggy image.

[0097] (3) use the set of paired image data to perform supervised learning on the network. If there is no pseudo-label, do not adjust the network.

[0098] (4) test the dehazing performance on the O-HAZE dataset, with the specific details including:

[0099] 1) Experimental environment: based on Pytorch1.8.1 + CUDA11.1 software environment, hardware configuration: CPU AMD Ryzen7 5800H, GPU NVIDIA GeForce RTX 3060Laptop.

[0100] 2) Experimental index: four quantitative indexes are used to evaluate the image defogging algorithm, including the structural similarity index (SSIM) of the image, the peak signal-to-noise ratio (PSNR), the average operation time of the algorithm and the space occupation of the algorithm. It is known that the larger the values of SSIM and PSNR are, the smaller the values of running time and space occupation are, and the better the performance is.

[0101] 3) Control experiment: In order to verify the effectiveness of the algorithm, the proposed image defogging system is compared with several most advanced image defogging methods: AOD-Net method, GCA-Net method, FFA-Net method, MSBDN-Net method, UHDUD method and D4 Framework method.

[0102] 4) Experimental results:

[0103] The quantitative results are shown in Table 1, and the recovery effect of image defogging is shown in Figure 6 .

[0104] Table 1 Quantitative evaluation of O-HAZE dataset defogging test

[0105]

[0106] S43: Self-enhanced semi-supervised learning is performed on real daytime haze environment, and its defogging effect is tested. The recovery effect of image defogging is shown in Figure 7 .

[0107] S44: Self-enhanced semi-supervised learning is performed on real nighttime haze environment, and its defogging effect is tested. The recovery effect of image defogging is shown in Figure 8 .

[0108] Finally, it should be pointed out that the above described is the preferred embodiment of the present application, and for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A semi-supervised learning image defogging method based on a physical imaging model, characterized in that, Establishing a network structure of a defogging system including a feature extraction network and a defogging network, and iteratively performing gradient optimization training of supervised training and self-enhanced semi-supervised learning to generate an optimal image defogging system for actual processing of images to make the images defogged, comprising the following steps: S1: Synthesizing clear-foggy image data sets as training sets of the defogging system, and pre-processing the data sets; S2: Building an image feature extraction network: a feature fusion lightweight network with dense connection; specifically comprising the following steps: S2a: The feature extraction network includes n convolutional structure groups, each of which includes a dilated convolution layer, an instance normalization layer, a spatial attention mechanism module and a Relu activation function, wherein only the first convolutional structure group does not contain a spatial attention mechanism module; S2b: The input of the first convolutional structure group is a degraded RGB image, the input channel number is 3, the channel number of the output feature map group is 6, and the input and output spatial sizes are kept unchanged; S2c: The input of the nth (n=2, 3, 4) convolutional structure group is the spliced feature map group output by the previous (n-1) convolutional structure groups, the input channel number is (n-1)×6, and the channel number of the output feature map group is 6, and the input and output spatial sizes are kept unchanged; S2d: The output feature map groups of the n convolutional structure groups are spliced together to form a degraded image feature map group with a channel number of 6n; S3: Building an image defogging network: using an improved physical imaging model with adaptive compensation information to restore clear images; specifically comprising the following steps: S31: The 6n groups of feature maps output by the shared feature extraction network are respectively assigned different weights through two adaptive weight distribution modules; S32: The two groups of weighted 6n groups of feature maps are respectively reduced to 3 channels through a convolutional structure group to obtain two parameter matrices K(x) and Δb(x) of the improved atmospheric scattering model containing adaptive compensation information; S33: According to the improved atmospheric scattering model, the clear image J is restored; The form of the improved atmospheric scattering model is: Wherein, I(x) is the xth pixel value in the degraded image captured by the imaging device; j(x) is the xth pixel value of the clear image; A is the atmospheric light constant; t(x) is the transmittance matrix; K(x) and Δb(x) are parameters to be estimated; according to the improved atmospheric scattering model, the parameters K(x) and the direct information compensation Δb(x) are estimated from the foggy image using a convolutional neural network, and then they are brought into the model to obtain the fog-free image; S4: Using the synthesized clear-foggy image data set to perform end-to-end supervised training of the network to generate a defogging system with preliminary image defogging effect; further optimizing the image defogging system according to the self-enhanced semi-supervised learning strategy to enhance the defogging effect.

2. The semi-supervised learning image defogging method according to claim 1, characterized in that, Step S1 specifically comprises the following steps: S11: Establishing a training data set, which includes a large number of clear-foggy image data pairs, denoted as (I, J), wherein I represents a foggy degraded image, and J represents a clear image corresponding thereto; S12: Randomly cropping the image pairs to perform data set enhancement, and fixing the size of the image to WxH; S13: The cropped image is randomly flipped to further enhance the data set, and converted into a tensor form for input into the network for training.

3. The semi-supervised learning image defogging method of claim 1, wherein, The adaptive weight distribution module in step S31 is specifically: The feature map group F input into the module c Global average pooling and global maximum pooling are performed to obtain two groups of feature map groups G with a size of Cx1x1 c ; The two feature map groups are respectively subjected to one-dimensional convolution operation with one input channel, and then the two feature map groups are jointly subjected to one-dimensional convolution operation with two input channels. Finally, the weight a of the feature map group is determined by the Sigmoid function, and the feature map group is adaptively weighted according to the weight a.

4. The semi-supervised learning image defogging method of claim 1, wherein, In step S4, the network is trained, specifically including the following steps: S41: The image data preprocessed in step S1 is used to input into the haze removal system network for end-to-end iterative supervised training until the network converges, and a pre-trained haze removal system with preliminary image haze removal effect is generated.

5. The semi-supervised learning image defogging method according to claim 4, characterized in that, In step S41, the end-to-end supervised learning is performed by using a loss calculation network to calculate the perceptual loss and make it converge: The loss calculation network uses a VGG16 feature extraction network pre-trained on an Image Net data set, and calculates the perceptual loss between the restored image and the clear reference image: wherein, φ j (C),j = 1,2,3 represent perceptual loss calculation using VGG16 as perceptual loss network in three channels respectively, D j ,H j ,W j denote the size of the input image.

6. The method of claim 1, wherein the method further comprises: Step S42 enhances the haze removal effect of the haze removal system by using a self-enhanced semi-supervised learning strategy, specifically including the following steps: (1) input a haze-containing image into the pre-trained haze removal system to obtain Output 1; then input Output 1 into the system again to obtain Output 2; (2) if the clear image Output 2 is better than Output 1, use Output 2 as a pseudo label to form a pair of data with the haze-containing image; determine whether the re-clearing image is better; (3) use the paired data obtained in (2) to perform supervised learning on the pre-trained haze removal system network structure until no better haze removal effect can be obtained, stop the self-enhanced semi-supervised learning, and obtain an optimal image haze removal system; the optimal image haze removal system is used for haze removal processing on an input single haze-containing degraded image, and outputs a clear haze-free image.

7. The semi-supervised learning image defogging method according to claim 6, characterized in that, The better image is determined by a two-class classifier based on the no-reference image evaluation index IL-NIQE, which calculates the IL-NIQE of the two images to be compared, and the one with lower value is determined as the better image.

Images