An improved MSCNN underwater image dehazing method
The brightness equalization and contrast improvement of underwater images were performed through Retinex and CLAHE treatment methods, and combined with the MSCNN network to estimate the transmittance, the problems of uneven light and Rayleigh scattering in underwater images were solved, and the defog effect was significantly improved and Objective evaluation indicators.
Patent Information
- Application Number
- CN202111066603.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-13
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-09-13
AI Technical Summary
The prior art has problems of uneven light and obvious Rayleigh scattering in underwater image defog removal, resulting in unsatisfactory defog removal effect. The traditional method has poor image processing effect on large-area sky areas and is estimated to be low in transmission.
The brightness equalization and contrast improvement of the image was used based on Retinex and CLAHE, and the transmittance was estimated in combination with the MSCNN network. The dark channel prior method was used to obtain the background light intensity, and finally the defogging image was restored.
It significantly improves the fog removal effect of underwater images, solves the halo and overexposure problems, improves objective evaluation indicators, and is suitable for complex underwater environments.
Smart Images

Figure CN113920018B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to an improved MSCNN underwater image defogging method. Background Art
[0002] Underwater images are an important carrier and presentation method of underwater information, and they play a vital role in the exploration, development, and utilization of marine resources. However, due to the limitations of the objective imaging environment and equipment, the quality of underwater images is always unsatisfactory, with degradation phenomena such as low contrast, blurred details, and color deviation, which seriously restricts the development of related fields. Therefore, how to enhance and restore degraded underwater images through post-processing algorithms has attracted more and more attention from scholars. In recent years, with the rapid development of deep learning technology, underwater image enhancement and restoration technology based on deep learning has made great progress.
[0003] In recent years, in the upsurge of machine learning, many algorithms for image defogging using machine learning techniques have emerged. Tang et al. systematically studied different fog-related features in a learning framework to determine the best feature combination for image defogging, and used random forests to regress the atmospheric transmittance of foggy images from multiple prior features, thereby restoring clear images. The synthetic method proposed by Levin et al. to generate training data also made many subsequent machine learning-based methods possible. Zhu et al. proposed to use color attenuation prior to defog the input foggy image. By establishing a linear model to model the scene depth of the blurred image under this new prior, and using supervised learning methods to learn the parameters of the model, the depth information can be well restored. Image algorithms based on machine learning have greatly improved the effect on the basis of traditional algorithms, but compared with image reconstruction tasks such as image super-resolution, single image defogging has a lot of room for improvement in both subjective and objective evaluations, and there are still many problems that have not been optimized and solved.
[0004] It can be seen from the atmospheric scattering model that the key to image defogging under active light illumination is to correctly solve the atmospheric light and transmittance values, and the use of deep learning methods for image defogging can achieve relatively ideal results. However, since the foggy training data set of this type of method is artificially synthesized, there is a certain gap between the model fitted by the network and the real scene. At the same time, the loss function adopts the form of mean square error, which means that the network can only reduce the overall pixel value difference of the image, and cannot guarantee the similarity between the overall area of the defogged image and the foggy image. Therefore, the method based on deep learning has the disadvantages of being difficult to train and the defogging effect is seriously dependent on the quality of the training data set. At the same time, traditional methods such as the Retinex method cannot obtain effective processing effects on images with large areas of sky areas, and the estimated value of transmittance is low. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide an improved MSCNN underwater image defogging method, which can effectively improve the defogging effect.
[0006] An image defogging algorithm comprises the following steps:
[0007] Step 1: extract the R, G, and B channels of the image, perform normalization, and process them using the CLAHE algorithm respectively, and then synthesize the processed three-channel images into one image;
[0008] Step 2: extract the R, G, and B channels of the image, process the three-channel images using the MSRCR algorithm in the Retinex algorithm, and then synthesize the processed three-channel images into one image;
[0009] Step 3: Perform weighted fusion on the images processed in step 1 and step 2 to obtain a fused image, which is set as image D;
[0010] Step 4: preprocess the image D;
[0011] Step 5: Train the MSCNN network, and then import the processed image D into the trained MSCNN model to estimate its transmittance t(x);
[0012] Step 6: Use the dark channel prior method to estimate the maximum illumination value of image D to obtain the background light value C;
[0013] Step 7: Substitute into the formula I(x)=J(x)t(x)+C[1-t(x)] to restore the haze-free image I(x).
[0014] Preferably, when the MSCNN network is trained, a CNN network that can estimate the image depth of field d(x) is first used, and the transmittance of the input clear image e is estimated using the formula t(x)=e-βd(x) (where β is the atmospheric absorption coefficient). Then the model randomly generates a background light value A, and uses the formula I(x)=J(x)t(x)+A[1-t(x)] to synthesize a foggy image, and then the synthesized foggy image is sent to the MSCNN network for training.
[0015] Preferably, in step 1, the image is divided into a plurality of different sub-regions using 8×8 rectangular blocks.
[0016] Preferably, in step 1, Clip-Limit is 0.01.
[0017] Preferably, in step 2, when merging the R, G, B three-channel images, Gaussian functions with scales of σ=20, σ=100, and σ=200 are used respectively.
[0018] Preferably, in step 3, by adjusting the weighted fusion coefficient, the formula is as follows:
[0019] D=tA+(1-t)B
[0020] Where D represents the final fused image, A represents the image after CLAHE processing, B represents the image after MSRCR processing, and t is the fusion weight coefficient, which is 0.95.
[0021] Preferably, in step 4, the preprocessing includes white balance.
[0022] Preferably, in step 4, the preprocessing includes contrast enhancement.
[0023] Preferably, in step 4, the preprocessing includes gamma enhancement.
[0024] Preferably, in step 4, the gamma value used in gamma enhancement is 0.9.
[0025] The present invention has the following beneficial effects:
[0026] The present invention proposes an improved MSCNN underwater image defogging method, which first performs Retinex and CLAHE processing on the foggy image to make the illumination of the image uniform, then performs white balance correction, contrast enhancement and other preprocessing on the processed image, and then inputs the image into a trained MSCNN model to estimate the transmittance of the image, and uses a dark channel prior method to obtain the background light intensity, and finally obtains the defogged image; the present invention combines Retinex and CLAHE to balance the brightness and enhance the contrast of the underwater image, so that the method is more advantageous for complex situations such as low illumination, uneven illumination and obvious Rayleigh scattering in the underwater environment, and solves the problem that halo and overexposure are easily generated in the restoration process of underwater images with non-uniform illumination in the prior art; compared with the existing method, the method has greatly improved the defogging effect of underwater images, and various objective evaluation indicators have been improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is the overall operation flow chart of the present invention.
[0028] Figure 2 This is a flow chart of the MSCNN dehazing algorithm used in the present invention.
[0029] FIG. 3( a ) is an original underwater foggy image; FIG. 3( b ) is a defogged image obtained using the defogging method of the present invention. DETAILED DESCRIPTION
[0030] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0031] 1. Principle of underwater imaging
[0032] In an underwater environment, the image obtained by the imaging system is mainly composed of two parts: one is the target reflected light attenuated by absorption and scattering of water particles, and the other is the background light formed by scattering of water particles. The mathematical expression is:
[0033] I total (x,y)=D(x,y)+B(x,y) (1)
[0034]
[0035] B(x,y)=∫ Θ B(θ) approx dΘ=B ∞ [1-t(x,y)] (3)
[0036] Among them I totalis the original image obtained by the imaging system, that is, the underwater blurred image; D(x, y) is the light intensity of the light reflected from the surface of the scene reaching the imaging system after being attenuated by the water body and forward scattered; B(x, y) is the underwater background light intensity generated by backscattering of underwater particles.
[0037] The expression of D(x,y) is:
[0038]
[0039] in is the effective radiation information of the underwater target, t(x,y)=e -β(x,y)ρ(x,y) is the transmission rate of light underwater, ρ(x,y) is the relative distance between the light source and the imaging system, and β(x,y) is the total attenuation coefficient of the water body's absorption and scattering effects on the transmitted light.
[0040] The expression of B(x,y) is:
[0041] B(x,y)=∫ Θ B(θ) approx dΘ=B ∞ [1-t(x,y)] (5)
[0042] Where B(θ) approx is the backscattering volume element function, Θ represents the set of all possible scattering angles for a certain water volume element, B ∞ It is the background light intensity at infinity underwater.
[0043] 2. Non-uniform illumination image enhancement technology based on Retinex and CLAHE
[0044] 1) Multi-scale Retinex algorithm with color restoration
[0045] The Retinex theory was first proposed by American physicist Edwin Land. The theory is based on the color constancy theory. The so-called color constancy refers to the fact that human judgment of the color of an object will not change due to changes in lighting conditions, showing constancy. The Retinex theory believes that the color of an object perceived by humans is not affected by external lighting conditions, but only related to the reflective properties of the object itself. Therefore, we can use the Retinex theory to remove the color offset of underwater blurred images and achieve color fidelity. The Retinex theory regards the image S(x,y) as the product of the illumination component L(x,y) determined by environmental factors and the reflection component R(x,y) determined by the essential properties of the object, as shown in the following formula:
[0046] S(x,y)=L(x,y)·R(x,y) (6)
[0047] The illumination component L(x,y) determines the dynamic range of the image S(x,y), and the reflection component L(x,y) reflects the reflection properties of the object itself. The essence of the Retinex algorithm is to regard the image as the sum of the illumination component and the reflection component, and then remove the illumination component to eliminate the influence of external conditions on the image, and obtain the reflection component, that is, the original property of the object.
[0048] The principle of the multi-scale Retinex algorithm for color restoration is:
[0049]
[0050]
[0051] Among them, a k (x,y) represents the nonlinear adjustment coefficient, N represents the number of bands, I k (x, y) represents the image of the kth band, and β represents the gain adjustment coefficient. It can be seen that compared with the general multi-scale Retinex algorithm, the multi-scale Retinex algorithm for color restoration adds an adjustment factor a on its basis. k (x,y). Adjustment factor a k The role of (x, y) is to adjust the R, G, B components of the image enhanced by the multi-scale Retinex algorithm according to the ratio of the R, G, B components of the original image to avoid color distortion.
[0052] 2) CLAHE histogram equalization
[0053] The histogram distribution of low-light underwater images is narrow and the values are generally low. CLAHE can reshape the distribution of the histogram so that the pixel levels are distributed between 0 and 255, making it uniform, removing the color cast to a certain extent, and improving the contrast to show more details. The specific implementation steps of the CLAHE method are as follows:
[0054] The input image is divided into M×N non-repeating sub-regions.
[0055] b) The gray level histogram of each segmented sub-region is averaged and the pixels with the same result are calculated. That is, the average pixel is
[0056]
[0057] Among them, N gray Represents the total number of pixels with gray levels in the sub-region, N x Indicates the number of pixels on the X axis of each small area, N y Indicates the number of pixels on the Y axis of each small area.
[0058] c) In each small area histogram, the The number of pixels to be intercepted is ∑N, where K represents the interception coefficient, and the total number of pixels of the intercepted part is calculated. v , and then find the number of pixels allocated to each gray level from the total number of intercepted images:
[0059]
[0060] d) Repeat steps (a), (b), and (c) for each sub-region, perform histogram equalization on the new histogram obtained after cropping each sub-region, and use the transformation function to obtain the new grayscale value.
[0061] 3) New idea based on fusion of CLARE and multi-scale Retinex
[0062] In order to solve the problem of underwater images affected by uneven illumination, the present invention takes into account the shortcomings of the Retinex algorithm, that is, for images with large illumination changes, a halo phenomenon will occur, causing the color of the color image to be whitened and distorted. However, the CLAHE algorithm can enhance the local contrast of the image and reduce the illumination change through histogram limiting. Therefore, the two algorithms are fused to form a new algorithm for image enhancement.
[0063] Usually, the fused image needs to increase the amount of detail information as much as possible while retaining the advantages of the original image. Image fusion levels can be divided into three: decision-level fusion, feature-level fusion, and data-level fusion. Data-level fusion is the basis of high-level image fusion. It directly processes the data collected by the sensor in multiple scenes and retains the original scene as much as possible. Providing texture information that other layers cannot provide is also one of the focuses of current fusion image research. Commonly used methods include weighted averaging and wavelet transform. Among them, the weighted averaging method (Weighted Averaging, WA) can directly fuse several image information of different qualities. It has the advantages of simplicity, intuitiveness, and fast calculation speed. At the same time, it can improve the signal-to-noise ratio of the image. Therefore, the present invention selects the pixel weighted averaging method to fuse the images processed by CLARE and multi-scale Retinex algorithms respectively.
[0064] 3. Image dehazing algorithm based on multi-scale convolutional network MSCNN
[0065] The performance of existing image dehazing methods is limited by hand-extracted features such as dark channels, color differences, and maximum contrast, as well as complex fusion schemes. Ren et al. trained the MSCNN network by learning the mapping between the foggy image and its corresponding transmittance image. The trained network can directly estimate the corresponding transmittance from the fog image. In the dehazing stage, the trained transmittance estimation model is directly used to obtain the transmittance of the fog image, and then the formula I(x) = J(x)t(x) + A[1-t(x)] is used to solve the image to be restored. For the network estimating transmittance, the network designed by Ren et al. is divided into two parts: one is the transmittance coarse estimation network, and the other is the transmittance refinement network. The input of the second convolutional layer of the refinement network is combined with the output of the coarse estimation network, and both sub-networks are trained separately. Experiments show that MSCNN performs better than the state-of-the-art methods for both synthetic and real-world images in terms of quality and speed.
[0066] 1) Coarse-scale convolutional neural network
[0067] The task of the coarse-scale network is to estimate the overall transmittance map of the scene. The coarse-scale network contains four operations: convolution, maximum pooling, upsampling, and linear combination.
[0068] Convolutional Layer: This network takes RGB images as input. The response of each convolutional layer is given by the following equation.
[0069]
[0070] in and are the feature maps of layer l and layer l+1 respectively. In addition, k is the convolution kernel, the index (m,n) represents the mapping from the current m-th layer to the n-th layer, and * represents the convolution operator. The function σ(·) represents the rectified linear unit (ReLU) of the filter response, and b is the bias.
[0071] Max pooling layer: Each convolutional layer is followed by a max pooling layer with an upsampling factor of 2.
[0072] Upsampling: In the overall framework, the size of the ground truth transmissibility map is the same as the input. The size of the feature map is reduced by half after max pooling. Therefore, an upsampling layer is added to ensure that the size of the output transmissibility map and the input foggy image are equal. Although one can choose to remove the max pooling layer and the upsampling layer to achieve the same goal, this approach will reduce the nonlinearity of the network, which is not very effective. The upsampling layer follows the pooling layer to restore the size of the subsampled feature map while maintaining the nonlinearity of the network. The response of each upsampling layer can be defined as
[0073]
[0074] Where l represents the layer number, (x, y) represents the two-dimensional spatial coordinates of the image, and the essence of this formula is to copy the pixel value at (x, y) to its corresponding 2×2 area. The linear combination layer is essentially the result of calculating all feature maps of the second-to-last layer using a convolution kernel with a spatial size of 1×1.
[0075] Structurally, the network is an end-to-end regression network for estimating the transmittance of fog images, and uses the output of the coarse estimation network as an important feature map to be merged into the fine estimation network, fully reflecting the multi-scale characteristics of the network. However, due to the connection method of the upsampling layer, the local receptive field of each layer in the network is small, which has certain limitations in learning the global features of fog images. In addition, the use of synthetic data based on indoor images as training data also affects the network's estimation effect on the transmittance of fog images to a certain extent.
[0076] 2) Fine-scale convolutional neural network:
[0077] The task of the fine-scale convolutional neural network is to refine the rough transmittance map output by the above coarse-scale convolutional neural network. By inputting the foggy image and the rough transmittance map into the fine-scale convolutional neural network, a more accurate transmittance map can be obtained. The fine-scale convolutional neural network model includes three convolutional layers of different scales, with convolution kernel sizes of 7×7, 5×5 and 3×3 respectively. The pooling kernel size in the pooling layer is the same as that in the coarse-scale convolutional network, both of which are 2×2. Finally, after the refinement of the fine-scale convolutional neural network, the output of the model is the refined transmittance map.
[0078] The estimation of atmospheric illumination in the MSCNN algorithm is based on the dark channel prior theory. After obtaining the refined transmittance map, select the darker pixel points in the transmittance map and calculate the intensity of the corresponding pixel points in the original foggy image. The value with the maximum intensity is the atmospheric illumination value. Then substitute the output transmittance map of the MSCNN model and the solved atmospheric illumination value into the atmospheric scattering model formula to obtain the defogged image.
[0079] In order to combine the advantages of multi-scale Retinex and CLAHE algorithms with the MSCNN algorithm, the present invention proposes an improved MSCNN image dehazing algorithm.
[0080] like Figure 1 As shown, the specific process is as follows:
[0081] Step 1: Extract the R, G, and B channels of the image, normalize them, and process them using the CLAHE algorithm. Then, combine the processed three-channel images into one image to enhance the contrast and details of the image. In the process, use 8×8 rectangular blocks to divide the image into multiple different sub-areas, and the Clip-Limit is 0.01.
[0082] Step 2: Extract the R, G, and B channels of the image, process the three-channel images using the MSRCR algorithm in the Retinex algorithm, and then combine the processed three-channel images into one image. When merging the R, G, and B three-channel images, the Gaussian functions with scales of σ=20, σ=100, and σ=200 are used.
[0083] Step 3: Perform weighted fusion on the images processed in step 1) and step 2). By adjusting the weighted fusion coefficient, a better visual effect can be obtained. The formula is as follows:
[0084] D=tA+(1-t)B (13)
[0085] Where D represents the final fused image, A represents the image after CLAHE processing, B represents the image after MSRCR processing, and t is the fusion weighting coefficient. After a large number of experiments, the optimal fusion coefficient t is 0.95.
[0086] Step 4: The processed image D is preprocessed by white balance, contrast enhancement and gamma enhancement, wherein the gamma value is determined to be 0.9 after repeated experiments.
[0087] Step 5: Figure 2 As shown, the MSCNN network is trained, and then the processed image D is imported into the trained MSCNN model to estimate its transmittance t(x).
[0088] Among them, when the MSCNN network is trained, a CNN network that can estimate the image depth of field d(x) is first used, and the transmittance of the input clear image e is estimated using the formula t(x)=e-βd(x) (where β is the atmospheric absorption coefficient). Then the model randomly generates a background light value A, and uses the formula I(x)=J(x)t(x)+A[1-t(x)] to synthesize a foggy image, and then the synthesized foggy image is sent to the MSCNN network for training.
[0089] Step 6: Use the dark channel prior method to estimate the maximum illumination value of image D and obtain the background light value C.
[0090] Step 7: Substitute into the formula I(x)=J(x)t(x)+C[1-t(x)] to restore the haze-free image I(x).
[0091] The effect diagram of the defogging method of the present invention is shown in FIG3 , and the evaluation results are shown in Table 1:
[0092] Table 1 Underwater image quality evaluation results
[0093] Information Entropy Average Gradient UCIQE Visible edge gradient ratio Original image 7.0339 1.1158 0.2312 1 This method 7.0870 5.5931 0.3264 5.0402
[0094] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An image dehazing algorithm, characterized in that: The steps include: Step 1: extract the R, G, and B channels of the image, perform normalization, and process them using the CLAHE algorithm respectively, and then synthesize the processed three-channel images into one image; Step 2: extract the R, G, and B channels of the image, process the three-channel images using the MSRCR algorithm in the Retinex algorithm, and then synthesize the processed three-channel images into one image; Step 3: Perform weighted fusion on the images processed in step 1 and step 2 to obtain a fused image, which is set as image D; Step 4: preprocess the image D; Step 5: Train the MSCNN network, and then import the processed image D into the trained MSCNN model to estimate its transmittance t(x); Step 6: Use the dark channel prior method to estimate the maximum illumination value of image D to obtain the background light value C; Step 7: Substitute the formula I(x)=J(x)t(x)+C[1-t(x)] to restore the haze-free image I(x); Among them, when the MSCNN network is trained, a CNN network that can estimate the image depth of field d(x) is first used, and the transmittance of the input clear image e is estimated using the formula t(x)=e-βd(x). Then the model randomly generates a background light value A, and uses the formula I(x)=J(x)t(x)+A[1-t(x)] to synthesize a foggy image. The synthesized foggy image is then sent to the MSCNN network for training, where β is the atmospheric absorption coefficient.
2. An image defogging algorithm as claimed in claim 1, characterized in that: In step 1, the image is divided into a plurality of different sub-regions using 8×8 rectangular blocks.
3. The image defogging algorithm according to claim 1, characterized in that: In step 1, Clip-Limit is 0.
01.
4. The image defogging algorithm according to claim 1, characterized in that: In the step 2, when fusing the R, G, B three-channel images, Gaussian functions with scales of σ=20, σ=100, and σ=200 are used respectively.
5. The image defogging algorithm according to claim 1, characterized in that: In step 3, the fused image is: D=tA+(1-t)B Where D represents the final fused image, A represents the image after CLAHE processing, B represents the image after MSRCR processing, and t is the fusion weight coefficient, which is 0.
95.
6. The image defogging algorithm according to claim 1, characterized in that: In step 4, the preprocessing includes white balance.
7. An image defogging algorithm as claimed in claim 6, characterized in that: In step 4, the preprocessing includes contrast enhancement.
8. An image defogging algorithm as claimed in claim 7, characterized in that: In step 4, the preprocessing includes gamma enhancement.
9. An image defogging algorithm as claimed in claim 7, characterized in that: In step 4, the gamma value used in gamma enhancement is 0.9.
