Image defogging method and system based on near and far scene segmentation and fusion

Abstract

The present invention discloses an image defogging method and system based on long-range and short-range segmentation and fusion, which is applied to the field of image processing technology, including: performing long-range and short-range segmentation on the defogging image based on the graph cut method, and predicting the segmented long-range image using a fog concentration level classification model based on a convolutional neural network, and calculating the classification model training loss by means of cross entropy loss and cosine similarity loss; constructing a residual network defogging model based on a fusion attention mechanism, and calculating the defogging model training loss by means of weighted fusion of L1 loss, color loss and perceptual loss, respectively performing defogging operation on the long-range image under different classification prediction results by means of the defogging models trained under different loss weights; splicing the long-range image after the defogging operation with the short-range image, and performing fusion transition on the segmentation edge pixels using Gaussian filtering, and completing image defogging by adaptive contrast adjustment. The present invention effectively improves the image defogging accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image defogging method and system based on near and far scene segmentation and fusion. Background Art

[0002] Fog is a very common phenomenon in nature due to atmospheric absorption and scattering, which often leads to image degradation of outdoor scenes, loss of contrast and color fidelity. Defogging methods can bring better visual effects to foggy images, making them clearer and richer in details, and can play a significant role in both computer vision and consumer photography. First, defogging can improve the visibility of the scene and correct the color deviation caused by air scattering, making the image more pleasing to the eye. Secondly, the performance of most computer vision algorithms (such as feature extraction, segmentation, and classification detection) will inevitably be affected by the low contrast of the image. Effective defogging methods can provide more sufficient information for subsequent high-level visual algorithms and image editing, laying a good foundation. In addition, after defogging, the degraded and blurred foggy images can be reused, reducing data waste.

[0003] Existing defogging methods are mainly divided into two categories. One is the end-to-end method. Thanks to the development of deep learning, the end-to-end method can directly realize the output from foggy image input to fog-free image by constructing a convolutional neural network without the need for additional prior knowledge. The other is the method based on physical priors. Most of these methods are based on the atmospheric scattering model. They first estimate the atmospheric light value and transmittance, and then restore the fog-free image according to the model. Although the end-to-end learning method can directly learn the complex mapping transformation from foggy to fog-free through the network and restore the fog-free image, due to the sharing of convolutional network weight parameters, the same parameters are used for areas with different fog concentrations, which easily leads to the defogging image being too bright or too dark, and the defogging is not comprehensive enough, which will also cause information loss and image distortion. The methods based on the atmospheric scattering model believe that the atmospheric light value is the same in the entire image, so they are more suitable for processing images with a single depth. The effect is not ideal for images with multiple distance depths and large amplitude differences. Taking the classic dark channel dehazing method as an example, this method is not suitable for natural images with large depth of field differences, because the scene depth in natural images usually changes suddenly at the edge of the scene, and the image processed by this method will produce obvious halo effect at the edge. It can be seen that for images with both distant and near scenes, if the whole image is dehazed in a unified way, it is likely to over-enhance the near image in order to highlight the distant scene objects, or make the distant scene unclear in order to balance the near image.

[0004] Therefore, how to provide an image defogging method and system based on near and far scene segmentation fusion that can take into account the processing of near and far scenes and effectively improve the defogging accuracy of images with near and far scenes is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0005] In view of this, the present invention proposes an image defogging method and system based on near and far scene segmentation and fusion.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An image defogging method based on near-far scene segmentation and fusion, comprising:

[0008] Step 1: Segment the dehazed image into near and far scenes based on the graph cut method, and classify and predict the segmented far scene image using the fog density classification model based on the convolutional neural network. Calculate the classification model training loss based on the cross entropy loss and cosine similarity loss.

[0009] Step 2: Construct a residual network defogging model based on the fusion attention mechanism, and calculate the defogging model training loss by weighted fusion of L1 loss, color loss and perceptual loss. Defogging operations are performed on distant images under different fog concentration level classification prediction results using the defogging models trained under different loss weights.

[0010] Step 3: Splice the distant image and the near image after the defogging operation, and use Gaussian filtering to fuse the segmented edge pixels, and complete the image defogging after adaptive contrast adjustment.

[0011] Optionally, in step 1, the dehazed image is segmented into near and far scenes based on a graph cut method, specifically:

[0012] Create an undirected graph , represents the image to be segmented; and are the sets of vertices and edges respectively; each pixel in the image is The vertices in the image are set as ordinary vertices, and the adjacent ordinary vertices are connected by edges; in addition to the ordinary vertices, there are two vertices in the image, respectively and Indicates that, let it be a terminal vertex, all ordinary vertices are connected to two terminal vertices respectively, forming an edge set Part of; Undirected graphs have two types of vertices and two types of edges;

[0013] Will Represents a vision, Represented as a close-up; each edge in the image has a non-negative weight , set as cost; according to the edge set The edges in , divide the common vertices into two disjoint subsets and , For Connected vertices, For connected vertices, and and The subsets in are not connected;

[0014] Minimize the sum of the weights of the edge subsets to achieve the optimal near-far scene segmentation, specifically:

[0015] When the segmentation is C, the cost function is expressed as follows:

[0016]

[0017] in, is a hyperparameter; is a regional item; is the boundary item;

[0018] Area Item ,as follows:

[0019]

[0020]

[0021] in, To give pixels Assigning Tags of loss, ; Pixel Gray value of

[0022] Boundary Item ,as follows:

[0023]

[0024]

[0025] in, and are adjacent pixels; is the set of all pairs of adjacent pixels; is an indicative function. When the labels of adjacent pixels are different, the function value is , other times ; To give pixels Assigned labels, ; Pixel Gray value of is a hyperparameter; is the distance function.

[0026] Optionally, in step 1, the fog concentration level classification model based on the convolutional neural network is specifically:

[0027] The feature map finally output by the convolutional neural network is expanded into a vector with a length of 1000, and input into the fully connected layer to obtain an output vector with a length equal to the number of fog concentration level labels. The output vector is subjected to a softmax operation to obtain the classification probability of each fog concentration level, and the fog concentration level with the highest probability is taken as the final classification result. The softmax formula is as follows:

[0028]

[0029] in, For the The classification probability corresponding to each element; is the number of labels in the output vector whose length is the number of fog concentration level labels. elements; is the number of labels in the output vector whose length is the number of fog concentration level labels. elements.

[0030] Optionally, in step 1, the cross entropy loss is as follows:

[0031]

[0032] in, is the number of training samples; For the Is the training image of A category, 1 for yes and 2 for no; The output of the model is The training image belongs to The probability of a classification.

[0033] Optionally, in step 1, the cosine similarity loss is as follows:

[0034]

[0035] in, is the number of training samples; For the The label vector corresponding to the training image; The output of the model is The probability vector of the training images; It is the vector dot product operation; Modulo operation for vectors.

[0036] Optionally, in step 2, the residual network dehazing model based on the fusion attention mechanism is as follows:

[0037] The model input undergoes three convolution + downsampling operations in sequence, followed by three convolution + upsampling operations, and the output after the third upsampling operation is added to the model input as the residual to obtain the final output result; among them, the output after the first convolution operation is spliced ​​with the output after the third upsampling operation, the output after the second convolution operation is spliced ​​with the output after the second upsampling operation, and the output after the third convolution operation is spliced ​​with the output after the first upsampling operation; the output after the second convolution operation is input into the spatial attention module and the channel attention module respectively, and the output results are multiplied with the output of the convolution operation after the second upsampling operation in sequence.

[0038] Optionally, in step 2, the dehazing model training loss is calculated by weighted fusion of L1 loss, color loss, and perceptual loss, specifically:

[0039] L1 loss, as follows:

[0040]

[0041] in, is the number of training samples; is the number of pixels in a single image; For the training image The pixel value of pixels; The output of the model is training image The pixel value of pixels;

[0042] Color loss, as follows:

[0043]

[0044]

[0045] in, is the arccosine function; Modulo operation for vectors; is the vector after the blurred image is expanded after convolution with Gaussian kernel; For the training images; The output of the model is The dehazed image of the training image;

[0046] Perceptual loss is as follows:

[0047]

[0048] in, To input the image into the features extracted from the pre-trained network; To be the first The training image is input into the first The output of the layer; To output the model corresponding to the The dehazed image of the training image is input into the first The output of the layer; is the total number of layers of the pre-trained network;

[0049] The dehazing model training loss is as follows:

[0050]

[0051] in, , is a hyperparameter that controls the weights of different losses.

[0052] Optionally, in step 3, Gaussian filtering is used to perform fusion transition on the segmented edge pixels, specifically:

[0053] Gaussian filter kernel, as follows:

[0054]

[0055] in, , is the distance from the origin; is the variance;

[0056] Normalization is performed as follows:

[0057]

[0058] in, is the range covered by the filter kernel;

[0059] The edge pixels are segmented after fusion transition as follows:

[0060]

[0061] in, for The pixel value at .

[0062] Optionally, in step 3, the adaptive contrast adjustment is specifically as follows:

[0063] Calculate the upper and lower thousandths of the entire pixel value as the cutoff value to eliminate the influence of extreme pixel values, and linearly stretch the truncated pixel value as follows:

[0064]

[0065] in, is the pixel value of the input image; is the lower thousandth quantile; It is a quantile of one thousandth;

[0066] Perform gamma transformation on the pixel values ​​after linear stretching, as follows:

[0067]

[0068]

[0069] in, is the median pixel value of the input image and then normalized; As the coefficient of gamma transformation, when When , the dark contrast is stretched to increase the overall brightness. When , the contrast of bright areas is stretched to reduce the overall brightness;

[0070] Linear stretching is performed according to the output dynamic range, as follows:

[0071]

[0072] in, is the lower bound of the output dynamic range; The upper bound of the output dynamic range.

[0073] An image defogging system based on near and far scene segmentation and fusion using an image defogging method based on near and far scene segmentation and fusion, comprising:

[0074] Image segmentation and classification module: The dehazed image is segmented into near and far scenes based on the graph cut method, and the segmented far scene image is classified and predicted using the fog density level classification model based on the convolutional neural network. The classification model training loss is calculated based on the cross entropy loss and cosine similarity loss.

[0075] Image defogging module: Construct a residual network defogging model based on the fusion attention mechanism, and use the weighted fusion of L1 loss, color loss and perception loss to calculate the defogging model training loss. The defogging models trained under different loss weights are used to perform defogging operations on distant images under different fog concentration level classification prediction results.

[0076] Image stitching module: stitch the distant image and the near image after the defogging operation, use Gaussian filtering to fuse the segmented edge pixels, and complete the image defogging after adaptive contrast adjustment.

[0077] It can be seen from the above technical solutions that, compared with the prior art, the present invention proposes an image defogging method and system based on the fusion of distant and near scenes. First, the present invention performs distant and near scene segmentation on the defogged image based on the graph cut method, and uses the fog concentration level classification model based on the convolutional neural network to classify and predict the segmented distant view image, adopts a divide-and-conquer strategy for distant and near scenes, and further classifies and predicts the fog concentration level of the distant view, so that the image defogging processing accuracy is further improved. Secondly, a residual network defogging model based on the fusion attention mechanism is constructed, and the defogging model trained under different loss weights (including: L1 loss, color loss and perceptual loss) is used to perform defogging operations on distant view images under different fog concentration level classification prediction results, realizing the defogging of images under different fog concentrations, so that ideal defogging effects can be obtained from light fog to dense fog. Finally, the distant image and the near-field image after the defogging operation are spliced, and the segmented edge pixels are fused and transitioned using Gaussian filtering. The image defogging is completed after adaptive contrast adjustment, which not only avoids the difference in brightness or hue between the separately processed images, but also prevents mutations at the junction of the foreground and background, making the image transition more natural. At the same time, the concentration level discrimination output during the image defogging process of the present invention can also provide effective reference information for other subsequent application levels, and the classification model and defogging model loss calculation performed during the image defogging process of the present invention further improve the image defogging accuracy. In summary, the present invention effectively improves the image defogging accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0079] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0080] Figure 2 This is a schematic diagram of the structure of the residual network dehazing model based on the fusion attention mechanism of the present invention.

[0081] Figure 3 Schematic diagram of the channel attention module structure of the present invention. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Embodiment 1:

[0084] Embodiment 1 of the present invention discloses an image defogging method based on long-range and short-range segmentation and fusion. Figure 1 As shown, including:

[0085] Step 1: Segment the dehazed image into near and far scenes based on the graph cut method, and classify and predict the segmented far scene image using the fog concentration level classification model based on the convolutional neural network. Calculate the classification model training loss based on the cross entropy loss and cosine similarity loss.

[0086] Based on the graph cut method, the dehazed image is segmented into near and far scenes, specifically:

[0087] Create an undirected graph , represents the image to be segmented; and are the sets of vertices and edges respectively; each pixel in the image is The vertices in the image are set as ordinary vertices, and the adjacent ordinary vertices are connected by edges; in addition to the ordinary vertices, there are two vertices in the image, respectively and Indicates that, let it be a terminal vertex, all ordinary vertices are connected to two terminal vertices respectively, forming an edge set Part of; Undirected graphs have two types of vertices and two types of edges;

[0088] Will Represents a vision, Represented as a close-up; each edge in the image has a non-negative weight , set as cost; according to the edge set The edges in , divide the common vertices into two disjoint subsets and , For Connected vertices, For connected vertices, and and The subsets in are not connected, so the near and far images are separated;

[0089] Minimize the sum of the weights of the edge subsets, that is, minimize the cost, to achieve the optimal near-far scene segmentation, specifically:

[0090] When divided into When , the cost function is expressed as follows:

[0091]

[0092] in, It is a hyperparameter that controls the relative weight of the two. After minimizing the cost function, the corresponding near-far scene segmentation can be obtained. is the regional item; is the boundary item;

[0093] Area Item ,as follows:

[0094]

[0095]

[0096] in, To give pixels Assigning Tags of loss, ; First calculate the pixel Belongs to the tag The probability can be obtained by comparing pixels The grayscale of the given distant and near scenes is obtained by histogram; when the pixel Assign the label with the highest probability When , the loss is minimal, so take the negative logarithm of the above probability; Pixel Gray value of

[0097] Boundary Item , which is used to measure the difference between adjacent pixels. The more the adjacent pixels differ, the more they should be separated and the smaller the penalty, as follows:

[0098]

[0099]

[0100] in, and are adjacent pixels; is the set of all pairs of adjacent pixels; It is an indicative function. When the labels of adjacent pixels are different, the function value is 1, and otherwise it is 0. To give pixels Assigned labels, ; Pixel Gray value of is a hyperparameter; is a distance function that measures the distance between two pixels.

[0101] For the distant and near images obtained by segmentation, the near image is assumed to be clear and fog-free by default, and the distant image is sent to the classification model to obtain the fog concentration level classification result, which is convenient for subsequent further defogging processing.

[0102] The fog concentration level classification model based on convolutional neural network is as follows:

[0103] Expand the feature map output by the convolutional neural network into a vector with a length of 1000, input it into the fully connected layer to obtain an output vector with a length equal to the number of fog concentration level labels, and perform a softmax operation on the output vector to obtain the classification probability of each fog concentration level, and take the fog concentration level with the highest probability as the final classification result; the convolutional neural network can adopt the commonly used CNN network architecture as the main architecture, such as the lightweight mobilenetv3, etc.; the softmax formula is as follows:

[0104]

[0105] in, For the The classification probability corresponding to each element; is the number of labels in the output vector whose length is the number of fog concentration level labels. elements; is the number of labels in the output vector whose length is the number of fog concentration level labels. elements.

[0106] In the embodiment of the present invention, fog is divided into three levels according to its concentration: light fog, heavy fog and dense fog, and each level is labeled.

[0107] The classification model training loss adds cosine similarity loss to the cross entropy loss. Cross entropy is used to measure the similarity between two discrete distributions, and cross entropy loss measures the accuracy of model classification.

[0108] The cross entropy loss is as follows:

[0109]

[0110] in, is the number of training samples; For the Is the training image of A category, 1 for yes and 2 for no; The output of the model is The training image belongs to The probability of a classification.

[0111] Cosine similarity is a commonly used similarity measurement method, which is usually used to calculate the similarity between two vectors. The cosine similarity loss function can measure the similarity between the vector output by the model and the label vector. The cosine similarity loss is as follows:

[0112]

[0113] in, is the number of training samples; For the The label vector corresponding to the training image; The output of the model is The probability vector of the training images; It is the vector dot product operation; Modulo operation for vectors.

[0114] Step 2: Construct a residual network dehazing model based on the fused attention mechanism, and calculate the dehazing model training loss by weighted fusion of L1 loss, color loss and perceptual loss. The dehazing models trained under different loss weights are used to perform dehazing operations on distant images under different fog concentration level classification prediction results.

[0115] The residual network dehazing model based on the fusion attention mechanism, such as Figure 2 As shown, specifically:

[0116] The model input undergoes three convolution + downsampling operations in sequence to obtain high-level semantic information, followed by three convolution + upsampling operations. The output after the third upsampling operation is used as the residual and added to the model input to obtain the final output result. Compared with directly learning the dehazed image, learning the residual can reduce the difficulty of network fitting and improve the efficiency of network fitting. Among them, the output after the first convolution operation is spliced ​​with the output after the third upsampling operation, the output after the second convolution operation is spliced ​​with the output after the second upsampling operation, and the output after the third convolution operation is spliced ​​with the output after the first upsampling operation to ensure that the network can obtain rich local information and fully restore image details; the output after the second convolution operation is input into the spatial attention module and the channel attention module respectively to further explore the correlation between different pixels and the correlation between different channels, and the output results are multiplied with the output of the convolution operation after the second upsampling operation in sequence to better integrate the learned features.

[0117] The spatial attention module consists of two convolutional layers with 3*3 convolution kernels and a stride of 1, each of which is followed by an activation layer. The first convolutional layer doubles the number of feature channels, and the second convolutional layer restores the number of channels to 1, achieving the exchange and fusion of spatial information through the expansion and compression of the number of channels.

[0118] The channel attention module structure is as follows: Figure 3 As shown in the figure, each convolutional layer and fully connected layer is followed by an activation layer. First, it passes through a convolutional layer, and then the adaptive average pooling layer compresses the feature map to 1*1, and then expands the feature map into a vector with a length equal to the number of channels and inputs it into the following fully connected layer. The first fully connected layer expands the vector dimension to twice the original, and the second fully connected layer restores the vector dimension to the original length, and finally outputs a one-dimensional vector with a length equal to the number of input channels.

[0119] The dehazing model training loss is calculated by weighted fusion of L1 loss, color loss, and perceptual loss, specifically:

[0120] L1 loss calculates the mean absolute error of the pixel values ​​of the two images. Intuitively, the smaller the L1 loss, the closer the two images are as follows:

[0121]

[0122] in, is the number of training samples; is the number of pixels in a single image; For the training image The pixel value of pixels; The output of the model is training image The pixel value of pixels;

[0123] Color loss, by blurring the texture of the output image and the ground truth, only saving the color of the image, and then calculating the cosine similarity between the two, so as to achieve image color correction, as follows:

[0124]

[0125]

[0126] in, is the arccosine function. The closer the two vectors in the formula are, the greater the cosine similarity is, and the smaller the result after the arccosine calculation is, that is, the smaller the loss is; Modulo operation for vectors; The value range is , the closer the two vectors are, the greater the cosine similarity; is a vector obtained by expanding the blurred image after convolution with a Gaussian kernel. The embodiment of the present invention uses a Gaussian kernel with a kernel_size of 5 and a variance of 1; is the cth training image; The dehazed image of the cth training image output by the model;

[0127] Perceptual loss is a loss function based on neural network features. It measures the similarity of images by comparing the difference between the output image and the groundtruth in the high-level feature map. It can better capture the semantic information of the image and make the output image of the model more consistent with human visual perception, as follows:

[0128]

[0129] in, In order to more fully capture and compare the feature information of the two images, the outputs of multiple layers in the pre-trained network are selected for feature comparison. is the output of the dth layer of the pre-trained network inputting the cth training image; The dehazed image output by the model corresponding to the cth training image is input into the output of the dth layer in the pre-trained network; is the total number of pre-trained network layers. The embodiment of the present invention uses the classic network VGG19, and the feature output layers obtained are 2, 7, 12, 21, and 30 layers;

[0130] The dehazing model training loss is as follows:

[0131]

[0132] in, , is a hyperparameter that controls the weights of different losses.

[0133] For light fog images, the ideal defogging effect should be that the hazy image becomes clear, the color is restored, the saturation is increased, and the image looks natural. For dense fog images, the picture is often gray and the color information is lost. The purpose of defogging is mainly to restore the outline details of the picture and reduce information loss. For foggy images between light fog and dense fog, it is necessary to balance the pursuit of details and colors, and make the colors real and natural while restoring the image details as much as possible.

[0134] Since the convolutional network weight parameters are shared, the same parameters are used for different concentrations of fog. It is difficult for a single convolutional network to handle the above multiple situations at the same time. Therefore, for different situations, the weights of different loss functions are adjusted to train the defogging model. The light fog defogging model uses light fog and heavy fog images as training data. , and The weights are 1, 2 and 0.2 respectively; the heavy fog defogging model uses light fog, heavy fog and dense fog images as training data. , and The weights are 1, 1 and 0.2 respectively; the dense fog defogging model uses heavy fog and dense fog images as training data. , and The weights are 1, 0, and 0.2 respectively. In this method, the fog density classification result obtained by the previous classification network is used to input the distant fog image into the corresponding defogging network to obtain the defogged image.

[0135] Step 3: Splice the distant image and the near image after the defogging operation, and use Gaussian filtering to fuse the segmented edge pixels, and complete the image defogging after adaptive contrast adjustment.

[0136] The dehazed distant image and the near image are reassembled into a complete image. However, the stitched image often appears stiff and unnatural at the stitching point, and a transition is required at the stitching edge. Therefore, Gaussian filtering is used to blend the segmented edge pixels. Gaussian filtering is actually a process of weighted averaging of pixels, and the weights follow the Gaussian distribution. After Gaussian filtering, the value of the edge pixel is calculated by weighted averaging of itself and other pixel values ​​in the neighborhood. Use Gaussian filtering to blend the segmented edge pixels, specifically:

[0137] Gaussian filter kernel, as follows:

[0138]

[0139] Among them, the position of the pixel to be calculated is taken as the origin, , is the distance from the origin; is the variance; the closer the distance, the greater the weight, and vice versa.

[0140] Because it is a weighted average, the weights need to be added up to be equal to 1 and normalized, as follows:

[0141]

[0142] in, is the range covered by the filter kernel;

[0143] The edge pixels are segmented after fusion transition as follows:

[0144]

[0145] in, for The embodiment of the present invention uses a filter kernel with a variance of 1 and a size of 3*3 to achieve a fusion transition effect while minimizing image blur.

[0146] The stitched images may have different contrasts, which may lead to differences in perception and require further adjustment. Therefore, the embodiment of the present invention adopts adaptive contrast adjustment. The adaptive contrast adjustment is as follows:

[0147] Calculate the upper and lower thousandths of the entire pixel value as the cutoff value to eliminate the influence of extreme pixel values, and linearly stretch the truncated pixel value as follows:

[0148]

[0149] in, is the pixel value of the input image; is the lower thousandth quantile; It is a quantile of one thousandth;

[0150] Perform gamma transformation on the pixel values ​​after linear stretching to correct the overly bright or dark areas, as follows:

[0151]

[0152]

[0153] in, is the median pixel value of the input image and then normalized; As the coefficient of gamma transformation, when When , the dark contrast is stretched to increase the overall brightness. When , the contrast of bright areas is stretched to reduce the overall brightness;

[0154] Linear stretching is performed according to the output dynamic range, as follows:

[0155]

[0156] in, is the lower bound of the output dynamic range; The upper and lower bounds can be defined by the user. The upper and lower bounds are defined as follows in the embodiment of the present invention: .

[0157] Embodiment 2:

[0158] Embodiment 2 of the present invention discloses an image defogging system based on near and far scene segmentation and fusion using an image defogging method based on near and far scene segmentation and fusion, comprising:

[0159] Image segmentation and classification module: The dehazed image is segmented into near and far scenes based on the graph cut method, and the segmented far scene image is classified and predicted using the fog density level classification model based on the convolutional neural network. The classification model training loss is calculated based on the cross entropy loss and cosine similarity loss.

[0160] Image defogging module: Construct a residual network defogging model based on the fusion attention mechanism, and use the weighted fusion of L1 loss, color loss and perception loss to calculate the defogging model training loss. The defogging models trained under different loss weights are used to perform defogging operations on distant images under different fog concentration level classification prediction results.

[0161] Image stitching module: stitch the distant image and the near image after the defogging operation, use Gaussian filtering to fuse the segmented edge pixels, and complete the image defogging after adaptive contrast adjustment.

[0162] The embodiment of the present invention discloses an image defogging method and system based on the fusion of near and far scene segmentation. First, the present invention performs near and far scene segmentation on the image to be defogged based on the graph cut method, and uses the fog concentration level classification model based on the convolutional neural network to classify and predict the segmented far scene image, adopts a divide-and-conquer strategy for the far scene and the near scene, and further classifies and predicts the fog concentration level of the far scene, so that the image defogging processing accuracy is further improved. Secondly, a residual network defogging model based on the fusion attention mechanism is constructed, and the defogging model obtained by training under different loss weights (including: L1 loss, color loss and perceptual loss) is used to perform defogging operations on the far scene images under different fog concentration level classification prediction results, thereby realizing the defogging of images under different fog concentrations, so that ideal defogging effects can be obtained from light fog to dense fog. Finally, the distant image and the near-field image after the defogging operation are spliced, and the segmented edge pixels are fused and transitioned using Gaussian filtering. The image defogging is completed after adaptive contrast adjustment, which not only avoids the difference in brightness or hue between the separately processed images, but also prevents mutations at the junction of the foreground and background, making the image transition more natural. At the same time, the concentration level discrimination output during the image defogging process of the present invention can also provide effective reference information for other subsequent application levels, and the classification model and defogging model loss calculation performed during the image defogging process of the present invention further improve the image defogging accuracy. In summary, the present invention effectively improves the image defogging accuracy.

[0163] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0164] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image defogging method based on near-far scene segmentation and fusion, characterized in that: include: Step 1: Segment the dehazed image into near and far scenes based on the graph cut method, and classify and predict the segmented far scene image using the fog density classification model based on the convolutional neural network. Calculate the classification model training loss based on the cross entropy loss and cosine similarity loss. Step 2: Construct a residual network defogging model based on the fusion attention mechanism, and calculate the defogging model training loss by weighted fusion of L1 loss, color loss and perceptual loss. Defogging operations are performed on distant images under different fog concentration level classification prediction results using the defogging models trained under different loss weights. Step 3: Splice the distant image and the near image after the defogging operation, and use Gaussian filtering to fuse the segmented edge pixels, and complete the image defogging after adaptive contrast adjustment; In step 1, the dehazed image is segmented into near and far scenes based on the graph cut method, specifically: Establish an undirected graph G = (V; E), which represents the image to be segmented; V and E are sets of vertices and edges respectively; each pixel in the image is a vertex in V, set as a common vertex, and adjacent common vertices are connected by edges; in addition to the common vertices, there are two vertices in the image, represented by a and b, set as terminal vertices, and all common vertices are connected to the two terminal vertices respectively, forming a part of the edge set E; the undirected graph has two vertices and two edges; Let s be the distant view and t be the near view; each edge in the image has a non-negative weight W e , set as the cost; according to the edges in the edge set E, divide the common vertices into two disjoint subsets S and T, S is the vertex connected to s, T is the vertex connected to t, and the subsets in S and T are not connected; Minimize the sum of the weights of the edge subsets to achieve the optimal near-far scene segmentation, specifically: When the segmentation is C, the cost function is expressed as follows: L(C)=λR(C)+B(C); Among them, λ is a hyperparameter; R(C) is the region term; B(C) is the boundary term; The region term R(C) is as follows: R p (l p )=-lnPr(v p |l p ); Among them, R p (l p ) is to assign label l to pixel p p The loss, l p ∈{s,t};v p is the gray value of pixel p; The boundary term B(C) is as follows: Among them, p and q are adjacent pixels; N is the set of all pairs of adjacent pixels; δ(l p , l q ) is an indicative function, the function value is 1 when the labels of adjacent pixels are different, and 0 otherwise; l q is the label assigned to pixel q, l q ∈{s,t};v q is the gray value of pixel q; σ is a hyperparameter; dist(·) is the distance function; The step of minimizing the sum of weights of the edge subsets is to minimize the cost of the cost function.

2. The image defogging method based on near-far scene segmentation and fusion according to claim 1 is characterized in that: In step 1, the fog concentration level classification model based on the convolutional neural network is as follows: The feature map finally output by the convolutional neural network is expanded into a vector with a length of 1000, and input into the fully connected layer to obtain an output vector with a length of the number of fog concentration level labels. The output vector is subjected to a softmax operation to obtain the classification probability of each fog concentration level, and the fog concentration level with the highest probability is taken as the final classification result. The softmax formula is as follows: Among them, q s is the classification probability corresponding to the sth element; g s is the sth element in the output vector whose length is the number of fog concentration level labels; g k It is the kth element in the output vector whose length is the number of fog concentration level labels.

3. The image defogging method based on near-far scene segmentation and fusion according to claim 1 is characterized in that: In step 1, the cross entropy loss is as follows: Where N is the number of training samples; y cd Whether the c-th training image belongs to the d-th category, 1 if yes, 2 if no; p cd The probability that the cth training image output by the model belongs to the dth category.

4. The image defogging method based on near-far scene segmentation and fusion according to claim 1, characterized in that: In step 1, the cosine similarity loss is as follows: Where N is the number of training samples; h c is the label vector corresponding to the cth training image; p c is the probability vector of the cth training image output by the model; · is the vector dot product operation; ||·|| is the vector modulus operation.

5. The image defogging method based on near-far scene segmentation and fusion according to claim 1, characterized in that: In step 2, the residual network dehazing model based on the fusion attention mechanism is as follows: The model input undergoes three convolution + downsampling operations in sequence, followed by three convolution + upsampling operations, and the output after the third upsampling operation is added to the model input as the residual to obtain the final output result; wherein, the output after the first convolution operation is spliced ​​with the output after the third upsampling operation, the output after the second convolution operation is spliced ​​with the output after the second upsampling operation, and the output after the third convolution operation is spliced ​​with the output after the first upsampling operation; the output after the second convolution operation is respectively input into the spatial attention module and the channel attention module, and the output results are multiplied with the output of the convolution operation after the second upsampling operation in sequence.

6. The image defogging method based on near-far scene segmentation and fusion according to claim 1, characterized in that: In step 2, the dehazing model training loss is calculated by weighted fusion of L1 loss, color loss, and perceptual loss, specifically: L1 loss, as follows: Where N is the number of training samples; M is the number of pixels in a single image; z ci is the pixel value of the i-th pixel in the c-th training image; is the pixel value of the i-th pixel of the c-th training image output by the model; Color loss, as follows: Wherein, arccos is the inverse cosine function; ||·|| is the vector modulus operation; Φ(·) is the vector after the blurred image is expanded after convolution with the Gaussian kernel; z c is the cth training image; The dehazed image of the cth training image output by the model; Perceptual loss is as follows: Where Ω(·) is the feature extracted by inputting the image into the pre-trained network; (z c ) d is the output of the dth layer of the pre-trained network inputting the cth training image; The dehazed image output by the model corresponding to the cth training image is input into the output of the dth layer in the pre-trained network; C is the total number of layers in the pre-trained network; The dehazing model training loss is as follows: L dehaze =L1+λ1L color +λ2L percep ; Among them, λ1 and λ2 are hyperparameters that control the weights of different losses.

7. The image defogging method based on near-far scene segmentation and fusion according to claim 1 is characterized in that: In step 3, Gaussian filtering is used to fuse the segmented edge pixels, specifically: Gaussian filter kernel, as follows: Where x and y are the distances from the origin; m is the variance; Normalization is performed as follows: Among them, Θ is the range covered by the filter kernel; The edge pixels are segmented after fusion transition as follows: Among them, v x，y is the pixel value at (x, y).

8. The image defogging method based on near-far scene segmentation and fusion according to claim 1, characterized in that: In step 3, the adaptive contrast adjustment is as follows: Calculate the upper and lower thousandths of the entire pixel value as the cutoff value to eliminate the influence of extreme pixel values, and linearly stretch the truncated pixel value as follows: Among them, v in is the input image pixel value; L in is the lower thousandth quantile; H in It is a quantile of one thousandth; Perform gamma transformation on the pixel values ​​after linear stretching, as follows: <h2 style=";text-align:left;direction:ltr">v2 = 255*(v1 / 255)<h2 style=";text-align:left;direction:ltr"> 1 / O <h2 style=";text-align:left;direction:ltr"> ; O=128 / median(V in ); Wherein, O is the median of the input image pixel value and then normalized; O is used as the coefficient of gamma transformation. When O>1, the contrast of dark areas is stretched to increase the overall brightness; when O<1, the contrast of bright areas is stretched to reduce the overall brightness. Linear stretching is performed according to the output dynamic range, as follows: Among them, L out is the lower bound of the output dynamic range; H out The upper bound of the output dynamic range.

9. An image defogging system based on near and far scene segmentation and fusion using the image defogging method based on near and far scene segmentation and fusion according to any one of claims 1 to 8, characterized in that: include: Image segmentation and classification module: The dehazed image is segmented into near and far scenes based on the graph cut method, and the segmented far scene image is classified and predicted using the fog density level classification model based on the convolutional neural network. The classification model training loss is calculated based on the cross entropy loss and cosine similarity loss. Image defogging module: Construct a residual network defogging model based on the fusion attention mechanism, and use the weighted fusion of L1 loss, color loss and perception loss to calculate the defogging model training loss. The defogging models trained under different loss weights are used to perform defogging operations on distant images under different fog concentration level classification prediction results. Image stitching module: stitch the distant image and the near image after the defogging operation, use Gaussian filtering to fuse the segmented edge pixels, and complete the image defogging after adaptive contrast adjustment.

Images