A Domain-Adaptive Image Sharpening Method for Coal Mine Dust and Fog with Partial Parameter Sharing

Abstract

This invention proposes a domain-adaptive method for dehazing coal mine dust and fog images with partial parameter sharing. The method includes: constructing a coal mine dust and fog image dataset; building a domain-adaptive dehazing network framework with partial parameter sharing; supervised training of a loss function for synthetic dust and fog images; enhancement of real dust and fog images followed by unsupervised adversarial training; designing an enhancement module for real dust and fog images followed by supervised training; and obtaining the dehazed coal mine dust and fog images. The method provided by this invention effectively alleviates the problem of poor generalization ability of deep networks trained on synthetic data on real coal mine dust and fog images, thus improving the visual effect of the images.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to a method for enhancing and sharpening realistic dust and fog images in low-level vision. Background Technology

[0002] Video surveillance systems are crucial for the intelligent operation of coal mines, and intelligent analysis and understanding of video and images are widely used in coal mine production and management. Accurate signal analysis and precise detection rely on high-quality, clear surveillance images. However, the harsh underground environment of coal mines, with the interaction of dust and moisture, causes videos and images to appear "dusty and foggy," greatly hindering image analysis and understanding and significantly impacting the monitoring and assessment of the underground production environment by relevant coal mine monitoring systems. Therefore, how to appropriately enhance the clarity of surveillance videos and images in light of the actual coal mine production environment is of significant research value and importance for improving the performance of monitoring systems and enhancing the safety of underground coal mine operations.

[0003] In recent years, compared with traditional methods [1][2], deep learning-based dehazing methods [3][4] have made great progress on synthetic data. However, the synthetic dataset used for training has made appropriate idealizations and assumptions about the atmospheric scattering model, and there is a large distribution difference between it and the real scene foggy image. Most dehazing algorithms based on synthetic datasets have poor generalization in real scenes and the dehazing effect is not ideal. References [5][6][7] proposed training methods such as semi-supervised, domain adaptation, and fine-tuning to train the network, which improved the dehazing effect in real scenes to a certain extent. However, existing methods mostly use a completely shared network structure for synthetic and real images, and do not pay enough attention to the complexity of image distribution and degradation factors in real scenes. When the domain distribution difference is large, the effect is not ideal, which restricts the practical application. Therefore, it is necessary to study more flexible network structures and training strategies to clarify coal mine dust and fog images and improve image quality.

[0004] References:

[0005] 1. Kaiming He, Jian Sun and Xiaoou. Tang, Single image haze removal using dark channel prior, IEEE Transactions on Pattern Analysis and MachineIntelligence, 33(12): 2341-2353, 2011.

[0006] 2.Dana Berman, Tali Treibitz, and Shai Avidan. Non-local imagedehazing, In IEEE Conference on Computer Vision and Pattern Recognition,pages:1674-1682, 2016.

[0007] 3.Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng.Aod-net: all-in-one dehazing network. In IEEE International Conference onComputer Vision, pages 4770–4778, 2017.

[0008] 4.Yanyun Qu, Yizi Chen, Jingying Huang, and Yuan Xie. Enhancedpix2pix dehazing network. In IEEE Conference on Computer Vision and PatternRecognition, pages 8160–8168, 2019.

[0009] 5.Lerenhan Li, Yunlong Dong, Wenqi Ren, Jinshan Pan, Changxin Gao,Nong Sang, and Ming-Hsuan Yang. Semi-supervised image dehazing. IEEETransactions on Image Processing, 29: 2766–2779, 2020.

[0010] 6.Yuanjie Shao, Lerenhan Li, Wenqi Ren, Changxin Gao, and Nong Sang.Domain adaptation for image dehazing. In IEEE Conference on Computer Visionand Pattern Recognition, pages 2808–2817, 2020.

[0011] 7.Zeyuan Chen, Yangchao Wang, Yang Yang, and Dong Liu, “PSD: Principled synthetic-to-real dehazing guided by physical priors,” In IEEEConference on Computer Vision and Pattern Recognition (CVPR). Jun. 2021, pp.7180–7189 Summary of the Invention

[0012] This invention addresses the problem of poor generalization ability of existing dehazing networks on real dust and fog images in coal mines, and proposes an adversarial domain adaptive dehazing algorithm with partial parameter sharing, mainly including the following steps:

[0013] Step S1: Construct a coal mine dust and fog image dataset;

[0014] Step S2: Build a domain-adaptive defogging network framework with partially shared parameters;

[0015] Step S3: Supervised training of the design loss for the synthesized dust and fog image;

[0016] Step S4: Enhance the real dust and fog image and perform unsupervised adversarial training;

[0017] Step S5: Design an enhancement module for real dust and fog images and perform supervised training;

[0018] Step S6: Obtain the result of clarifying the coal mine dust and fog image;

[0019] Compared to current dehazing methods that use fully parameter-sharing, this invention first performs appropriate enhancement processing on the real dust and fog image, designs a domain-adaptive dehazing network with partially shared parameters, and introduces semi-cyclic reconstruction constraints on the unsupervised real image dehazing results. This effectively alleviates the problem of overfitting on synthetic fog and poor generalization ability on real coal mine dust and fog images caused by excessive differences in the distribution of synthetic fog and real fog. The working environment in coal mines is harsh, and image degradation is severe. This invention first enhances the real dust and fog image to improve image quality and mitigate the impact of factors such as illumination and noise on the dust and fog image. Secondly, the locally shared dehazing network structure further alleviates the problem of mutual influence and interference when the distribution difference between synthetic and real images is too large. The semi-cyclic reconstruction constraints on the unsupervised dehazing results can avoid distortion and loss of structure and details in the dehazed image. Finally, for ease of testing, this invention introduces a feature enhancement module into the dehazing network for real dust and fog images, using proprietary network parameters to learn the image enhancement operations in preprocessing. In practical applications, dehazing operations can be directly performed on coal mine dust and fog images. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the process of the present invention;

[0021] Figure 2 This is a schematic diagram of the framework of the present invention;

[0022] Figure 3 This is a schematic diagram of the domain-dependent dual-branch residual module of the present invention;

[0023] Figure 4 This is the feature enhancement module of the present invention.

[0024] Figure 5 For comparison of the defogging visual effects of the present invention

[0025] Figure 6 This is the quantitative defogging result of the present invention. Detailed Implementation

[0026] like Figure 1 and Figure 2 The diagram shown is a flowchart and framework diagram of the photo style transfer method based on style corpus constraints of the present invention. Figure 3 and Figure 4 This diagram illustrates the domain-dependent dual-branch residual module and feature enhancement module in a dehazing network. The main steps include: constructing a coal mine dust and fog image dataset; building a domain-adaptive dehazing network framework with partially shared parameters; designing a loss function for synthetic dust and fog images and performing supervised training; enhancing real dust and fog images and performing unsupervised adversarial training; designing an enhancement module for real dust and fog images and performing supervised training; and obtaining stylized photographs. The specific implementation details of each step are as follows:

[0027] Step S1: Construct a coal mine dust and fog image dataset, as follows:

[0028] Step S11: Use explosion-proof cameras or surveillance videos to acquire 2000 real-world dust and fog images of coal mines, of which 1500 are used for training and 500 are used for testing, and perform CP-AHE enhancement preprocessing.

[0029] Step S12: Collect 1650 clear images of the coal mine and obtain their corresponding depth maps using a pre-trained depth estimation network. Synthesize a foggy image according to the atmospheric scattering model formula, where the atmospheric light parameter A is selected within the range [0.7, 1.0]. Select within the range [0.6, 1.8]. Select 6500 pairs of synthetic fog and their corresponding clear images as training data, and 1400 pairs of data as test data.

[0030] Step S2: Build a domain-adaptive dehazing network framework with partially shared parameters, as follows:

[0031] Step S21: Select a multi-scale enhanced network with dense feature fusion as the base dehazing network G_base, and select the Unet model as the degenerate network denoted as T.

[0032] Step S22: Improve the Bottleneck layer of network G_base by introducing a domain-dependent dual-branch residual module structure as shown in the figure, for feature extraction. By performing average pooling and max pooling operations in the horizontal and vertical directions, and then merging them, features related to both directions can be obtained. and .

[0033] Step S23: and Passing through The convolution operation reduces the number of channels from 256 to 16. To encourage sufficient interaction of feature information in the horizontal and vertical directions, the following is achieved: and After calculating the activation function, a multiplication operation is performed to obtain the intermediate feature F containing location information, which can be expressed by the formula.

[0034] F= , F

[0035] Step S24: Characterize the synthesized fog by passing it through 1 step. The convolution operation restores the features from 16 channels to the original 256 channels before dimensionality reduction. Then, a shuffle operation is used to facilitate interaction between features, and finally, the sigmoid function is used to obtain the corresponding weight information. The same operation is applied to the features of the real fog, as shown in the following formula:

[0036]

[0037]

[0038] ,

[0039] The parameters of convolution Conv_S and convolution Conv_T in the formula are not shared, in order to handle the unique information between the synthetic domain and the real domain.

[0040] Step S3: Supervised training of the loss design for the synthesized fog image. Since the synthesized fog has a corresponding clear image, supervised pixel loss and perceptual loss are used during training to make the defogging image as close as possible to the clear image, as expressed in the following formula.

[0041]

[0042] in and These represent the dehazed image and the clear image after output, respectively. This represents the feature map of the corresponding layer in the pre-trained network VGG-19.

[0043] Step S4: Enhance the real dust and fog image and perform unsupervised adversarial training, as follows:

[0044] Step S41: Process the real fog image The enhanced image is obtained after CP-AHE enhancement. ,Will The result is obtained after feeding it into the defogging network G. ,Will The corresponding output is obtained by feeding the degenerate network T. .

[0045] Step S42: For the enhanced real fog image Since there is no supervision, an adversarial network is used during training to constrain the dehazing results. The adversarial loss function for the generator and discriminator D is defined as follows:

[0046]

[0047]

[0048] in Represents the distribution of sharp images. This represents the distribution of the dehazed image. Through adversarial training, the discriminator tries to distinguish between different images as much as possible. and The defogging network makes as much as possible It is consistent with the distribution of clear images, thus achieving a dehazing effect.

[0049] Step S43: To avoid structural distortion and detail loss in images during adversarial training, a loss function is constructed such that... Image obtained after passing through degenerate network T It should be able to be restored The loss function is defined as , here parameters It was set to 1e-3.

[0050] Step S5: Design an enhancement module for real dust and fog images and perform supervised training, as follows:

[0051] Step S51: Real Fog And the pre-processed enhanced dust and fog image Its characteristics are respectively used and This indicates that the two input features are differentiated in the shallow layers of the network. They first pass through three common residual modules consisting of Conv-ReLU-Conv, and then the features of the real fog image are input into an additional residual block. Through a dedicated residual block, the neural network learns the functionality of the preprocessing operation.

[0052] Step S52: For real fog images Design a loss function to make the real fog and enhanced realistic fog Changes caused by preprocessing can also be learned by neural networks. The loss function is defined as:

[0053]

[0054] in and This indicates that after introducing the feature enhancement module and Its characteristics.

[0055] Step S6: Obtain the result of the coal mine dust and fog image after sharpening. Specifically, the network is trained according to the above loss function. During testing, a real dust and fog image in the coal mine scene is selected as the input to obtain the corresponding sharpened image.

[0056] The working environment in coal mines is harsh, and images are severely degraded. Compared with the current dehazing method that uses a fully shared network structure, this invention first enhances the real dust and fog image to improve image quality and alleviate the influence of factors such as illumination and noise on the dust and fog image. Secondly, the use of a locally shared dehazing network structure can further alleviate the problem of mutual influence and interference when the distribution difference between the synthetic image and the real image is too large. The semi-cyclic reconstruction constraint on the unsupervised dehazing result can avoid the distortion and loss of structure and details of the image after dehazing. Finally, for the convenience of testing, this invention introduces a feature enhancement module into the dehazing network for real dust and fog images and uses proprietary network parameters to learn the image enhancement operation in the preprocessing. In practical applications, it can directly perform dehazing operation on coal mine dust and fog images. Compared with other methods, it further improves the dehazing effect of real fog in coal mines. The qualitative and quantitative comparison results are shown in Figure [5] and Figure [6].

Claims

1. A domain-adaptive method for sharpening coal mine dust and fog images with partial parameter sharing, characterized in that, The method specifically includes the following steps: Step S1: Construct a coal mine dust and fog image dataset, as follows: Step S11: Use explosion-proof cameras or surveillance videos to acquire 2000 real-world dust and fog images of coal mines, of which 1500 are used for training and 500 are used for testing, and perform CP-AHE enhancement preprocessing. Step S12: Collect 1650 clear images of the coal mine, obtain their corresponding depth maps using a pre-trained depth estimation network, and synthesize a foggy image according to the atmospheric scattering model formula, where the atmospheric light parameter A is selected within the range of [0.7, 1.0]. Within the range of [0.6, 1.8], 6500 pairs of synthetic fog and their corresponding clear images were selected as training data, and 1400 pairs of data were selected as test data; Step S2: Build a domain-adaptive dehazing network framework with partially shared parameters, as follows: Step S21: Select a multi-scale enhanced network with dense feature fusion as the base dehazing network G_base, and select the Unet model as the degenerate network denoted as T; Step S22: Improve the Bottleneck layer of network G_base by introducing a domain-dependent dual-branch residual module structure to extract features. By performing average pooling and max pooling operations in the horizontal and vertical directions, and then merging them, features related to both directions can be obtained. and ; Step S23: and Passing through The convolution operation reduces the number of channels from 256 to 16. To encourage sufficient interaction of feature information in the horizontal and vertical directions, the following is achieved: and After calculating the activation function, a multiplication operation is performed to obtain the intermediate feature F containing location information, which can be expressed by the formula. F= , F ; Step S24: Characterize the synthesized fog by passing it through 1 step. The convolution operation restores the feature channel count from 16 to 256 before dimensionality reduction. Then, a shuffle operation is used to promote interaction between features. Finally, the sigmoid function is used to obtain the corresponding weight information. The same operation is applied to the features of the real fog. The formula is expressed as follows: ； ； ， The parameters of convolution Conv_S and convolution Conv_T in the formula are not shared, in order to handle the unique information between the synthetic domain and the real domain; Step S3: Supervised training is performed on the design loss of the synthesized fog image. Since the synthesized fog has a corresponding clear image, supervised pixel loss and perceptual loss are used during training to make the dehazed image as close as possible to the clear image. The formula is expressed as follows: ； in and These represent the dehazed image and the clear image after output, respectively. This represents the feature map of the corresponding layer in the pre-trained network VGG-19; Step S4: Enhance the real dust and fog image and perform unsupervised adversarial training, as follows: Step S41: Process the real fog image The enhanced image is obtained after CP-AHE enhancement. ,Will The result is obtained after feeding it into the defogging network G. ,Will The corresponding output is obtained by feeding the degenerate network T. ; Step S42: For the enhanced real fog image Since there is no supervision, an adversarial network is used during training to constrain the dehazing results. The adversarial loss function for the generator and discriminator D is defined as follows: ； ； in Represents the distribution of sharp images. This represents the distribution of the dehazed image. Through adversarial training, the discriminator tries to distinguish between different images as much as possible. and The defogging network makes as much as possible The distribution is consistent with that of a clear image, thus achieving a dehazing effect; Step S43: To avoid structural distortion and detail loss in images during adversarial training, a loss function is constructed such that... Image obtained after passing through degenerate network T It should be able to be restored The loss function is defined as , here parameters It was set to 1e-3; Step S5: Design an enhancement module for real dust and fog images and perform supervised training, as follows: Step S51: Real Fog And the pre-processed enhanced dust and fog image Its characteristics are respectively used and This means that the two input features are differentiated in the shallow layer of the network. They first pass through three common residual modules consisting of Conv-ReLU-Conv. Then, the features of the real fog image are input into an additional residual block. The neural network learns the function of the preprocessing operation through the special residual block. Step S52: For real fog images Design a loss function to make the real fog and enhanced realistic fog The changes caused by preprocessing can also be learned by a neural network, and its loss function is defined as: ； in and This indicates that after introducing the feature enhancement module and Features; Step S6: Obtain the result of the coal mine dust and fog image after sharpening. Specifically, the network is trained according to the above loss function. During testing, a real dust and fog image in the coal mine scene is selected as the input to obtain the corresponding sharpened image.

Images