A two-stage dehazing method for dense fog images based on inference

Abstract

The present invention discloses a two-stage defogging method for dense fog images based on inference, which belongs to the field of image processing and computer vision technology. The present invention completes the defogging of images in two stages. First, the fog concentration category of the image block of the image to be defogged is obtained based on the set image fog concentration classification network. Then, in the first stage, the light fog and moderate fog areas are restored to fog-free areas through the dark channel prior method, which provides more information for the restoration of dense fog areas in the second stage. Compared with other image defogging algorithms, the final output result can restore more details in the original fog-free image. The second stage is based on the processing structure of the first stage, combined with the defogging network based on inference to process the defogging of the fog image marked as heavy fog, and complete the defogging of the entire foggy image. In the data set construction stage of the present invention, the fog-free images selected are mostly cloudy images that conform to the real scene, which reduces the probability of color deviation of the image after defogging.

Description

Technical Field

[0001] The present invention belongs to the fields of image processing and computer vision, and in particular relates to a two-stage defogging method for dense fog images based on reasoning. Background Art

[0002] Images captured from outdoor scenes often suffer from varying degrees of degradation due to particles and water droplets floating in the atmosphere. This degradation is exacerbated in foggy scenes. Image dehazing algorithms aim to recover clear, fog-free images from camera input captured in foggy scenes. This is crucial for high-level vision tasks such as object detection and semantic segmentation.

[0003] In recent years, with the rapid advancement of deep learning technology, convolutional neural networks have begun to be used for image dehazing. However, some of these methods still rely on traditional methods to estimate the global atmospheric light value A. At the same time, while CNN (convolutional neural network)-based algorithms have significantly improved performance compared to traditional prior-based algorithms, they still face the problem of inaccurate transmittance estimation. To overcome this problem, some CNN-based end-to-end algorithms attempt to directly recover a fog-free output image from the input foggy image. However, CNN-based algorithms typically require large quantities of images for neural network training, which is unrealistic in the practice of image dehazing. Although some datasets provide image pairs of real scenes, the corresponding foggy images are either generated algorithmically from fog-free images or obtained by generating fog using fog machines, and there are certain differences between them and the actual foggy images. The paper "Zongyu Guo, Zhibo Chen, Tao Yu, Jiale Chen, and Sen Liu. Progressive image inpainting with full-resolution residual network. In Proc. ACM MM, pages 85–100, 2019" discloses a partial convolution-based image restoration network that can gradually restore damaged image patches starting from the edges. The paper "S. Santra, R. Mondal, and B. Chanda, "Learning a patch quality comparator for single image dehazing," IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4598–4607, 2018" discloses a patch-comparison image dehazing network. Based on the assumption that the quality of foggy image patches is lower than that of non-hazy image patches, but higher than that of over-dehazed (failed) patches, it uses a binary search approach to estimate the transmittance of the image on a block-by-block basis. However, this algorithm is too time-consuming and has not been used in practical applications. Chinese patent application number CN202110676400.4, entitled “A Method and System for Image Defogging Based on Multi-Scale Multi-Stage Neural Network,” discloses an image defogging method based on a multi-scale multi-stage neural network. This method achieves better defogging results by training with global features obtained from multiple resolutions, but the effect is poor when processing dense fog images. Chinese patent application number CN202011069293.0, entitled “A Method for Image Defogging Using Multi-Scale Fusion Convolutional Neural Network,” discloses a multi-scale fusion convolutional neural network (CNN) image defogging algorithm.This method achieves image dehazing by learning the mapping relationship between foggy images and atmospheric transmittance, but does not directly consider the mapping between foggy images and fog-free images. The use of intermediate variables (atmospheric transmittance) will cause a certain degree of deviation in the final output. Summary of the Invention

[0004] Aiming at the technical problems that existing image defogging algorithms are not good at processing dense fog images and are prone to color deviation and local blocks, the present invention provides a two-stage dense fog image defogging method based on reasoning.

[0005] The technical solution adopted in the present invention is:

[0006] A two-stage defogging method for dense fog images based on inference, the method comprises the following steps:

[0007] Step S1: setting and training an image fog density classification network (also called an image fog density perception network) based on a fog-free image and a corresponding artificial fog image;

[0008] The input data of the image fog density perception network is the segmented image block, and the output of the image fog density perception network includes three types of fog density: light fog, moderate fog and heavy fog, so as to obtain the fog density categories of different image blocks;

[0009] Step S2: Divide the image I to be defogged into blocks, and match the input size of each defogged image block to the image fog density perception network.

[0010] The image blocks to be defogged are input into the trained image fog density classification network to obtain the fog density category of each image block to be defogged;

[0011] Based on the pixel position index, the mask matrix M of the image to be defogged is generated according to the fog density categories of all the image blocks to be defogged, and the defogged image I is updated:

[0012] If the fog density category of the current image block to be defogged is light fog or moderate fog, the pixel value of the pixel position corresponding to the current image block to be defogged in the mask matrix M is set to 1, and the pixel value of the pixel position corresponding to the current image block to be defogged in the defogged image I remains unchanged;

[0013] If the fog density category of the current image block to be defogged is heavy fog, the pixel value of the pixel position corresponding to the current image block to be defogged in the mask matrix M is set to 0, and the pixel value of the pixel position corresponding to the current image block to be defogged in the defogged image I is set to a specified invalid value;

[0014] Step S3, performing a first-stage image defogging: using a dark channel prior method, defogging is performed on the non-invalid pixel points of the defogging image to obtain a first defogging image I′;

[0015] In the updated defogging image I, the pixel values ​​of the pixels with heavy fog concentration have been set to invalid values. Therefore, the first-stage image defogging is actually to defog the areas marked as light fog and moderate fog in the foggy image, thereby restoring them to clear and fog-free areas.

[0016] Step S4: perform two-stage image defogging:

[0017] The inference-based dehazing network performs dehazing on the mask matrix M and the first dehazed image I′ to obtain the final dehazed image.

[0018] Since the areas marked as light fog and moderate fog have been defogged in step S3, the defogging network is mainly used to defog the areas marked as heavy fog in step S4.

[0019] Preferably, in step S1, the image fog concentration classification network includes three stacked first convolution blocks and at least one fully connected layer in sequence, wherein each first convolution block includes a convolution layer (Conv), a batch normalization layer (BatchNorm layer) and an activation function layer in sequence (the activation function used can be selected based on the actual application scenario, such as the Tanh activation function), and the last fully connected layer is a fully connected layer with a Softmax function (i.e., a Softmax classifier), which is used to output the predicted probabilities of three types of fog concentrations and determine the current fog concentration category recognition result based on the maximum predicted probability.

[0020] Preferably, the image fog concentration classification network includes three fully connected layers, wherein the first fully connected layer is used to splice the feature map output by the third convolution block according to the channel dimension, that is, to convert the feature map output by the third convolution block into a one-dimensional vector, the second fully connected layer is followed by an activation function layer, and the third fully connected layer is a fully connected layer with a Softmax function.

[0021] Preferably, in step S1, the training data set of the image fog concentration classification network is obtained by:

[0022] Select fog-free and cloudy images from the public dataset to construct a dataset D for the fog concentration classification network. init ;

[0023] For the dataset D init For each image in , the image is divided into multiple non-overlapping image blocks p of size s×s, where the size of the image block p matches the input of the fog density classification network. init All image patches p of all images in the dataset D1 are obtained.

[0024] The image can be divided into multiple image blocks in one of the following two ways:

[0025] Method 1: First, the image is centered and cropped so that the width and height of the cropped image are both integer multiples of s. Then, the cropped image is evenly and non-overlappedly divided into blocks to obtain multiple s×s image blocks p.

[0026] Method 2: Directly divide the image into blocks of size s×s, evenly and non-overlappingly. The division is usually performed from left to right and from top to bottom, or vice versa. After the division is completed, the blocks that are smaller than s×s are discarded.

[0027] For each image block p in the dataset D1, artificial fog is added to obtain multiple artificial fog image blocks with the same global atmospheric light value but different transmittance corresponding to each image block p; and the artificial fog image blocks are classified into light fog, moderate fog, and heavy fog according to their different transmittances, and the corresponding category labels are set for each category. Based on all the artificial fog image blocks and their category labels, the dataset D2 is obtained as the training dataset for the image fog concentration classification network.

[0028] Preferably, in step S1, the training of the image fog concentration classification network is specifically as follows:

[0029] The contrast of each artificial fog image block in the training data set D2 is normalized and then input into the image fog density classification network for feedforward calculation. During training, the network parameters of the image fog density classification network are trained using random gradient descent, and the loss function L is used. c for:

[0030]

[0031] Among them, z represents the category label value output by the image fog concentration classification network, Represents the true category label value, that is, in the present invention, the cross entropy loss function is preferably used as the objective function for training the network.

[0032] Preferably, the step S3 is specifically as follows:

[0033] Select the pixel value of the point with the highest brightness in image I as the global atmospheric light value A;

[0034] Estimate the transmittance of the pixel x with non-invalid value:

[0035]

[0036] According to the atmospheric scattering model, the transmittance estimate of pixel x is substituted into the atmospheric scattering model:

[0037]

[0038] Calculate the pixel value J(x) after defogging;

[0039] Where I(x) represents the pixel value of image I, c represents the channel identifier of the image, and R, G, and B represent the three channels of the RGB image. Usually, the selected image I is an image in the RGB color space. If the selected image is not in the RGB color space, it is first converted to the RGB color space, and then the dark channel prior method is used to perform defogging on the non-invalid pixel points of the defogging image. Ω(x) represents the image block to be defogged centered on x, and A c Indicates the global atmospheric light value of the corresponding channel, I c (y) represents the value of the corresponding channel of the pixel at position y in the corresponding image block, Represents the estimated transmittance value of pixel x.

[0040] Preferably, in step S4, the defogging network used is a stacked structure of multiple defogging modules (Blocks);

[0041] The defogging module includes three branches: a first branch, a second branch, and a third branch, and a feature map fuser (abbreviated as F fuser or processor F);

[0042] The first branch sequentially includes a plurality of stacked second convolution blocks and a plurality of stacked first upsampling blocks, wherein the number of second convolution blocks and first upsampling blocks in the first branch is the same, the first convolution block sequentially includes a partial convolution layer (Pconv), a batch normalization layer, and an activation function layer; the first upsampling block sequentially includes an upsampling layer, a partial convolution layer, a batch normalization layer, and an activation function layer;

[0043] The second branch includes a plurality of stacked second convolutional blocks;

[0044] The third branch sequentially includes a plurality of stacked third convolution blocks and a plurality of stacked second upsampling blocks, wherein the number of third convolution blocks in the second branch is the same as that of the second upsampling blocks, the second convolution block sequentially includes a convolution layer, a batch normalization layer, and an activation function layer; the second upsampling block sequentially includes an upsampling layer, a convolution layer, a batch normalization layer, and an activation function layer;

[0045] The F fusion device is used to fuse the feature maps output by the three branches and output the mask matrix M i and dehazed image I″ i , the subscript i represents the number of the defogging module;

[0046] For any i-th defogging module, its input is the M output of the previous defogging module i-1 and dehazed image I″ i-1 , where M0 = M, I″ o= I′, that is, the input of the first defogging module is the mask matrix M and the first defogging image I′; and for any i-th defogging module, the input of the first branch and the second branch are the same, which is M i-1 and I″ i-1 , the input of the third branch is I″ i-1 The final defogging image is the defogging image output by the last defogging module. N BLOCK Indicates the number of dehazing modules.

[0047] Preferably, the mask matrix M output by the F fuser i and dehazed image I″ i for:

[0048] M i =M U ⊙M M (4)

[0049] I″ i =ω1(I U ⊙M U )+ω2(I M ⊙M M )+ω3I D ⊙M i (5)

[0050] Among them, M U Represents the mask matrix M of the first branch of the i-th defogging module to the input i-1 The processed output, M M The mask matrix M represents the input of the second branch of the defogging module i-1 The processed output, I U Denotes the first branch of the i-th defogging module to the input defogging image I″ i-1 The processed output, I M Denotes the second branch of the i-th defogging module to the input defogging image I″ i-1 The processed output, I D Represents the third branch of the i-th defogging module for the input defogging image I″ i-1 The processing output of , ⊙ represents the element-by-element product, the three weight coefficients ω1, ω2 and ω3 are all non-zero, and ω1+ω2+ω3=1. Preferably, ω1=ω2.

[0051] The technical solution provided by the present invention brings at least the following beneficial effects:

[0052] In the first stage, the dark channel prior method is used to restore light fog and moderate fog areas to fog-free areas, providing more information for the restoration of dense fog areas in the second stage. Compared with other image dehazing algorithms, the final output result can restore more details in the original fog-free image.

[0053] At the same time, during the dataset construction stage, considering the low saturation of fog images in actual scenes, the fog-free images selected are mostly cloudy images that conform to real scenes, which reduces the probability of color cast in the images after defogging. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 3 is a schematic diagram of the processing process of the two-stage defogging method for dense fog images provided by an embodiment of the present invention.

[0056] Figure 2 1 is a schematic diagram of the structure of the fog concentration classification network used in an embodiment of the present invention;

[0057] Figure 3 Schematic diagram of the overall structure of the defogging network used in the two-stage image defogging in an embodiment of the present invention;

[0058] Figure 4 1 is a schematic structural diagram of a defogging module (Block) used in an embodiment of the present invention;

[0059] Figure 5 Schematic diagram of the structure of the F fuser of the defogging module in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0061] Existing algorithms generally use image block or pixel-level neural networks to perform image defogging, processing both light and heavy fog areas. However, they are ineffective in processing heavy fog areas and may cause color shift and other problems. The present invention first pre-trains a fog concentration classification network that can perceive the fog concentration level information of different blocks in the foggy image and classify the fog concentration of the image blocks into three categories: light fog, moderate fog, and heavy fog. Before defogging the foggy image, the foggy image is first input into the trained fog concentration classification network to obtain the fog concentration level of each image block. The dark channel prior algorithm is then used to process the image blocks classified as light fog and moderate fog. Finally, the processed image is input into the image defogging network, which independently processes the heavy fog image blocks to obtain the final image defogging result.

[0062] like Figure 1 As shown, the embodiment of the present invention provides a two-stage defogging method for dense fog images based on reasoning, which includes the following steps:

[0063] S100: Setting an image fog density perception network based on a fog-free image and a corresponding artificial fog image to obtain fog density information of different image blocks.

[0064] (1) Constructing a dataset D for fog concentration classification network init In this embodiment, we select clear and fog-free images with cloudy weather conditions from the RESIDE set (real single image defogging set) and add D init .

[0065] (2) For dataset D init For each image in , starting from its center point, the image is divided into multiple small image blocks p of size 10×10 in the horizontal and vertical directions. The p with a size less than 10×10 is discarded. The multiple p obtained in sequence constitute the preprocessing dataset D. The number of p in the dataset D is recorded as Num.

[0066] (3) Taking the atmospheric scattering model shown in formula (3) as the mathematical basis for defogging, for each image block p in the data set D obtained in step (2), artificial fog is added to the image block p according to the global atmospheric light value A being (1, 1, 1) and t(x) being 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9, respectively, to obtain 9 artificial fog image blocks with the same A and different t(x) corresponding to each image block p.

[0067] (4) For the 9×Num artificial fog image blocks obtained in step (3), t(x) equal to 0.1, 0.2, and 0.3 are classified as light fog, t(x) equal to 0.4, 0.5, and 0.6 are classified as moderate fog, and t(x) equal to 0.7, 0.8, and 0.9 are classified as heavy fog. Finally, three image categories are obtained: light fog, moderate fog, and heavy fog, and the corresponding labels are set to 0, 1, and 2 respectively; each category has 3×Num artificial fog image blocks, and the classified labeled artificial fog image blocks are updated to the dataset D.

[0068] (5) Set the image fog concentration classification network N cla ,like Figure 2As shown in this example, the main body of the image fog density classification network consists of three convolutional layers and three fully connected layers. The input is a 10×10 image block of size to be classified, and the output is the probability of the image block being classified as 0, 1, or 2 (0 represents light fog, 1 represents moderate fog, and 2 represents heavy fog). The number of channels of the three convolutional layers is 64, 32, and 16, respectively. The convolution kernel size is 3×3, the stride size is 1, and no padding is performed on the three convolutional layers. Therefore, the output image sizes of the three convolutional layers are 8×8, 6×6, and 4×4, respectively. Each convolutional layer is followed by a BatchNorm layer and a Tanh fog density classification network. Flatten is used from the convolutional layer to the fully connected layer, resulting in a one-dimensional vector of length 256. The first and second fully connected layers have lengths of 16 and 8, respectively. The fully connected layers are followed by a Tanh activation layer, and the third fully connected layer has a length of 3, followed by a Softmax activation function, which outputs the probability of each fog density category. The predicted fog density category is obtained based on the maximum value.

[0069] (6) Use the dataset D to train the image fog density classification network. In this embodiment, the input contrast is normalized, the network learning rate is α1, and the value range of α1 is [10 -6 ,10 -2 ], using stochastic gradient descent to train the network. Let z represent the label value, z represent the actual classification value, and use the cross entropy loss function L c As the objective function of training the network, it is shown in formula (1).

[0070] It should be noted that the training method for the image fog concentration classification network can also adopt any conventional method in the art.

[0071] S200 obtains fog density information of different areas of the image to be defogged based on the trained image fog density classification network, and generates a mask matrix corresponding to areas with different fog density.

[0072] (7) For the fog image dataset D to be processed hazy First, each image is cropped to the center to obtain a new image of size w×h, where w represents the width of the image and h represents the height of the image. Both w and h are multiples of 10. The new cropped image replaces the corresponding uncropped image to update the dataset D. hazy ; Similarly, the fog image dataset D hazy The corresponding ground truth haze-free image dataset D clear Do and D hazy The same cropping results in an updated dataset D clear Then construct a full-1 mask matrix M of size w×h, where the elements at corresponding positions in M ​​correspond one to one with the pixels at corresponding positions in I.

[0073] (8) For dataset D hazy The image I in the image is divided into multiple image blocks p of size 10×10 in the horizontal and vertical directions starting from the upper left corner, and each image block p is fed into the trained fog density classification network N cla , get the fog density estimate for the image block p. If the image block p is estimated to be light fog or moderate fog, the element at the corresponding position in M ​​corresponding to the pixel point in the image block p remains 1, and the element at the corresponding position in image I remains unchanged; if the image block p is estimated to be heavy fog, the element at the corresponding position in M ​​corresponding to the pixel point in the image block p is set to 0, and the element at the corresponding position in image I is set to null:

[0074]

[0075]

[0076] Among them, the subscript i of p represents the number of the image block p, (x, y) represents the position index of M and the position index of the pixel point in p in the image I, Ω(p i ) represents the set of all pixels in the i-th image block. hazy For each image p of each image in , update each image I and the corresponding mask matrix M.

[0077] S300 performs a one-stage image defogging, processing the areas marked as light fog and moderate fog in the foggy image based on the dark channel prior, and restoring them to clear and fog-free areas.

[0078] (9) According to the atmospheric scattering model shown in formula (3), the pixel value of the point with the highest brightness in image I is selected as the global atmospheric light value A, and the transmittance estimation values ​​of the light fog area and the moderate fog area are obtained. (As shown in formula (2)). Each image block has only one estimated The coordinates of the image blocks here are consistent with the image blocks marked as light fog and moderate fog in step (8).

[0079] (10) After obtaining the transmittance estimation images corresponding to the light fog and moderate fog areas through step (9), the transmittance estimation images are smoothed using guided filtering. Then, combined with the atmospheric scattering model of formula (3), the defogging results for the light fog and moderate fog areas are obtained. At this time, the defogging of heavy fog remains unchanged. The image obtained at this stage is denoted as I stage1 , then input the two-stage defogging network N in step (11) dehaze middle.

[0080] S400 performs two-stage image defogging. Based on the fog image processed by S300, it combines the defogging network N based on inference.dehaze , completing the defogging of the entire foggy image.

[0081] (11) Set up a two-stage image defogging network N dehaze ,like Figure 3 As shown, in this embodiment, the image defogging network N dehaze It consists of 4 blocks with the same structure, namely the image defogging network N dehaze It is a stacked structure of 4 defogging blocks. The input of each block is the image I and the mask matrix M output by the previous block. In particular, the input of the first block is the image I after one stage of defogging. stage1 And the original mask matrix M. The I, M of each Block input and the I, M of the output are the same size. The I of each Block output is compared with the input I in the dense fog defogging. The update of M is the same as I. In order to distinguish, Figure 3 In the example, the image I and mask matrix M output by the jth (j=1…,4) Block are marked as I j and M j In particular, if the dense fog area in the image is large, 4 blocks are not enough to complete the pixel inference of the entire dense fog area. It is only necessary to cascade an appropriate number of blocks with the same structure after Block4.

[0082] Image Dehazing Network N dehaze The main function is to process fog image I stage1 The area marked as heavy fog in the image is restored to a clear and fog-free area by reasoning and integrating the original foggy pixels. In particular, the input of the first block is the mask matrix M obtained in step (8) and the image I obtained in step (10). stage1 .

[0083] Among them, Pconv represents partial convolution, which allows the convolution layer to perform convolution and normalization operations only on valid pixels that meet the conditions, as shown in formula (8):

[0084]

[0085] Where w represents the weight of the convolution kernel, X represents the pixel (feature) value corresponding to the current convolution (sliding) window, M represents the mask matrix corresponding to X, b represents the corresponding bias, ⊙ represents the element-by-element product, sum(M) represents the sum of M corresponding to the current convolution (sliding) window, and l has the same shape as M, but all values ​​are 1. After performing partial convolution, the mask matrix corresponding to the image is updated. The update rule is that if at least one value in the mask matrix M corresponding to the convolution window is 1, then the value in the mask matrix M at the corresponding position after convolution is updated to 1, as shown in formula (9).

[0086]

[0087] In each Block, there are three branches, such as Figure 3 As shown in Figure 1, it consists of three branches: upper, middle, and lower. The upper and middle branches input the image I and mask matrix output by the previous block (the input of the first block is the image I after one stage of dehazing and the original mask matrix M). The lower branch only inputs the image I output by the previous block.

[0088] The main body of the upper branch consists of six partial convolutional layers (Pconv), each with a 3×3 kernel size. The stride of the first three convolutional layers is 2, and the stride of the last three is 1. The number of channels in the first to sixth convolutional layers is 64, 96, 128, 96, 64, and 3, respectively. Each convolutional layer is followed by a BatchNorm layer and a Tanh activation layer. The third, fourth, and fifth Tanh layers are followed by an upsampling layer, respectively. Through multiple downsampling and upsampling, the upper branch can better capture and fuse multi-scale image features.

[0089] The middle branch consists of three partial convolutional layers, all with a convolution kernel size of 5×5 and a stride of 1. The number of channels in the three convolutional layers is 32, 32, and 3, respectively. The overall network structure of the lower branch is the same as that of the upper branch, except that some of the convolutional layers in the upper branch are replaced with ordinary convolutional layers to obtain the original features of the foggy area of ​​the image. After passing through the corresponding branches, I and M are processed by the processor F (such as Figure 5 As shown), the updated I and M are finally output.

[0090] Upper branch B U After each partial convolution Pconv, one pixel is inferred from the edge of the light fog area and the moderate fog area to the dense fog area, and M is updated once. A total of 6 pixels are inferred 6×1. The obtained image and mask matrix are recorded as I U and M U ; Middle branch B M After each partial convolution Pconv, 2 pixels are inferred from the edge of the light fog area and the moderate fog area to the dense fog area, and M is updated once. A total of 3×2 pixels are inferred, totaling 6 pixels. The obtained image and mask matrix are recorded as I M and M M ; Lower branch B D To extract the features of the dense fog area itself, ordinary convolution is used, and the obtained image and mask matrix are recorded as I D and M DThe image and mask matrix output by each Block are calculated as shown in formula (4) and formula (5) respectively. Preferably, in this embodiment, the values ​​of ω1, ω2 and ω3 are set to: Where I i and M i Represent the image and mask matrix output by the i-th Block respectively.

[0091] As a possible implementation, training the network N dehaze The objective loss function L when d By reconstruction loss L rec and step-by-step loss L step composition:

[0092] L rec =E[||I gt -I rec ||2] (10)

[0093]

[0094] Among them, E[·] represents the mathematical expectation, N BLOCK Represents the image dehazing network N dehaze The number of defogging blocks included, in this embodiment, N BLOCK =4,I gt represents a clear and fog-free image (true value), I rec Represents the final output image of the image dehazing network, that is In this embodiment, it is the image I″4,I″ output by the fourth defogging block i represents the image output by the i-th defogging block, M i represents the mask matrix output by the i-th dehazing block, ⊙ represents the element-by-element product, and the symbol || ||2 represents the 2-norm.

[0095] About to L rec With L step Add the set weights to get the target loss function L d :L d =β1L rec +β2L step , where β1 and β2 represent the reconstruction loss L rec , distribution loss L step The weight of .

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0097] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A two-stage defogging method for dense fog images based on inference, the method comprising the following steps: Step S1, setting and training an image fog concentration classification network based on a fog-free image and a corresponding artificial fog image; The input data of the image fog density classification network is the segmented image block, and the output of the image fog density classification network includes three types of fog density: light fog, moderate fog and heavy fog, so as to obtain the fog density categories of different image blocks; Step S2, dividing the image I to be defogged into blocks, and matching the input size of each defogged image block with the image fog density classification network; The image blocks to be defogged are input into the trained image fog density classification network to obtain the fog density category of each image block to be defogged; Based on the pixel position index, the mask matrix M of the image to be defogged is generated according to the fog density categories of all the image blocks to be defogged, and the defogged image I is updated: If the fog density category of the current image block to be defogged is light fog or moderate fog, the pixel value of the pixel position corresponding to the current image block to be defogged in the mask matrix M is set to 1, and the pixel value of the pixel position corresponding to the current image block to be defogged in the defogged image I remains unchanged; If the fog density category of the current image block to be defogged is heavy fog, the pixel value of the pixel position corresponding to the current image block to be defogged in the mask matrix M is set to 0, and the pixel value of the pixel position corresponding to the current image block to be defogged in the defogged image I is set to a specified invalid value; Step S3, using a dark channel prior method, performing defogging processing on the non-invalid pixel points of the defogging image to obtain a first defogging image I′; In step S4, the mask matrix M and the first defogged image I′ are defogged based on the inference defogging network to obtain a final defogged image.

2. The method according to claim 1, wherein In step S1, the image fog concentration classification network includes three stacked first convolution blocks and at least one fully connected layer in sequence, wherein each first convolution block includes a convolution layer, a batch normalization layer and an activation function layer in sequence, and the last fully connected layer is a fully connected layer with a Softmax function.

3. The method according to claim 2, wherein In step S1, the image fog concentration classification network includes three fully connected layers, where the first fully connected layer is used to splice the feature map output by the third convolution block according to the channel dimension, the second fully connected layer is followed by an activation function layer, and the third fully connected layer is a fully connected layer with a Softmax function.

4. The method according to claim 1, wherein In step S1, the training data set of the image fog concentration classification network is obtained as follows: Select fog-free and cloudy images from the public dataset to construct a dataset D for the fog concentration classification network. init ; For the dataset D init For each image in the dataset D, the image is divided into multiple non-overlapping image blocks p of the same size, where the size of the image block p matches the input of the fog density classification network. init All image blocks p of all images in get the dataset D1; For each image block p in the dataset D1, add artificial fog to obtain multiple artificial fog image blocks with the same global atmospheric light value but different transmittance corresponding to each image block p; According to the different transmittances of artificial fog image blocks, they are classified into light fog, moderate fog, and heavy fog, and corresponding category labels are set for each category. Based on all artificial fog image blocks and their category labels, dataset D2 is obtained as the training dataset for the image fog concentration classification network.

5. The method according to claim 4, wherein In step S1, the training of the image fog concentration classification network is specifically as follows: Normalize the contrast of each artificial fog image block in the training data set D2 and then input it into the image fog density classification network for feedforward calculation; During training, random gradient descent is used to train the network parameters of the image fog concentration classification network, and the loss function L is used. c for: Among them, z represents the category label value output by the image fog concentration classification network, Represents the true category label value.

6. The method according to claim 1, wherein Step S3 is specifically as follows: If the color space of the image I to be dehazed is not RGB color space, convert it to RGB color space; The image to be defogged I is divided into multiple non-overlapping image blocks of the same size as the image blocks to be defogged, and the size of the image blocks to be defogged matches the input of the fog concentration classification network; Select the pixel value of the point with the highest brightness in image I as the global atmospheric light value A; Estimate the transmittance of the pixel x with non-invalid value: The transmittance estimate of pixel x Substitute into the atmospheric scattering model Calculate the pixel value J(x) after defogging; where I(x) represents the pixel value of image I, c represents the channel identifier of the image, R, G, B represent the three channels of the RGB image, Ω(x) represents the image block to be defogged centered on x, and A c Indicates the global atmospheric light value of the corresponding channel, I c (y) represents the value of the corresponding channel of the pixel at position y in the image block to be defogged.

7. The method according to claim 1, wherein In step S4, the defogging network used is a stacked structure of multiple defogging modules; The defogging module includes three branches: a first branch, a second branch, and a third branch, and a feature map fusion device; The first branch sequentially includes a plurality of stacked second convolution blocks and a plurality of stacked first upsampling blocks, wherein the number of second convolution blocks and first upsampling blocks in the first branch is the same, the first convolution block sequentially includes a partial convolution layer, a batch normalization layer, and an activation function layer; the first upsampling block sequentially includes an upsampling layer, a partial convolution layer, a batch normalization layer, and an activation function layer; The second branch includes a plurality of stacked second convolutional blocks; The third branch sequentially includes a plurality of stacked third convolution blocks and a plurality of stacked second upsampling blocks, wherein the number of third convolution blocks in the second branch is the same as that of the second upsampling blocks, the second convolution block sequentially includes a convolution layer, a batch normalization layer, and an activation function layer; the second upsampling block sequentially includes an upsampling layer, a convolution layer, a batch normalization layer, and an activation function layer; The F fusion device is used to fuse the feature maps output by the three branches and output the mask matrix M i and dehazed image I″ i , the subscript i represents the number of the defogging module; For any i-th defogging module, its input is the M output of the previous defogging module i-1 and dehazed image I″ i-1 , where M0 = M, I″ o =I′; and for any i-th defogging module, the input of the first branch and the second branch are the same, which is M i-1 and I″ i-1 , the input of the third branch is I″ i-1 .

8. The method according to claim 7, wherein In step S4, the mask matrix M output by the feature map fuser i and dehazed image I″ i for: M i =M U ⊙M M I″ i =ω1(I U ⊙M U )+ω2(I M ⊙M M )+ω3I D ⊙M i Among them, M U Represents the mask matrix M of the first branch of the i-th defogging module to the input i-1 The processed output, M M The mask matrix M represents the input of the second branch of the defogging module i-1 The processed output, I U Denotes the first branch of the i-th defogging module to the input defogging image I″ i-1 The processed output, I M Denotes the second branch of the i-th defogging module to the input defogging image I″ i-1 The processed output, I D Represents the third branch of the i-th defogging module for the input defogging image I″ i-1 The processed output of , ⊙ represents the element-by-element product, the three weight coefficients ω1, ω2 and ω3 are all non-zero, and ω1+ω2+ω3=1.

9. The method according to claim 8, wherein In step S4, the weight coefficients ω1 and ω2 have the same value.

10. The method according to any one of claims 7 to 9, characterized in that The objective loss function L when training the dehazing network d For: L d =β1L rec +β2L step , β1 and β2 represent the reconstruction loss L rec and distribution loss L step The weight of Reconstruction loss L rec For: L rec =E[||I gt -I rec ||2]; Step loss L step for: Among them, E[·] represents the mathematical expectation, I gt represents the clear and fog-free images in the training set of the dehazing network, that is, the true label of the image to be dehazed during training, I rec Represents the dehazed image output by the last dehazing module of the dehazing network N BLOCK represents the number of dehazing modules, ⊙ represents the element-wise product, and the symbol || ||2 represents the 2-norm.

Images