Maritime image defogging method based on multi-level transmissivity guidance

Abstract

The invention discloses a sea image defogging method based on multistage transmissivity guidance, and relates to the field of computer vision and artificial intelligence. Comprising the steps that a generative adversarial network is constructed, a generator network comprises a basic defogging network, a multi-level transmissivity estimation network and a transmissivity guiding defogging engine, and a discriminator network adopts a PatchGAN architecture; training the generative adversarial network based on the training data set; and obtaining a to-be-processed foggy maritime image and inputting the foggy maritime image to the trained generator network, thereby generating a clear maritime image. According to the method, the basic defogging image and the multi-level transmissivity image are respectively generated through the double-branch generator, transmissivity deviation caused by multiple scattering is corrected in a step-by-step iteration mode, and high-quality and high-fidelity image restoration under the offshore complex fog condition is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and artificial intelligence, in particular to an image processing method based on deep learning, and especially to an image defogging method applied to marine monitoring, unmanned ship visual navigation and the like. BACKGROUND

[0002] Unmanned ships play an increasingly important role in marine observation, environmental monitoring and water transportation, and the images collected by their visual systems are the key source of obtaining external information. However, the marine atmospheric environment is complex and changeable, and water vapor, salt fog and other factors are prone to form fog, which leads to a decrease in image contrast, color distortion and detail blurring, seriously affecting the performance of visual algorithms and limiting the application of intelligent devices such as unmanned ships. Therefore, it is of great theoretical value and practical significance to study an efficient and reliable marine image defogging algorithm.

[0003] Existing image defogging algorithms can be roughly divided into prior-based methods and data-driven methods. Prior-based methods usually rely on certain statistical rules or physical assumptions, such as dark channel prior (DCP), to estimate the transmission rate and atmospheric light by constructing an atmospheric scattering model, and then restore the haze-free image. Data-driven methods, especially deep learning-based methods, directly or indirectly learn the mapping relationship from hazy images to haze-free images by training neural networks on large datasets.

[0004] However, whether it is a traditional method or an existing deep learning method, most of them are designed for land environments, and their models are usually based on the single scattering assumption, i.e. the light only scatters once with the fog particles before reaching the camera. Compared with the land environment, the particle radius of the sea fog is larger and the scattering coefficient is higher, which causes the light to experience multiple scattering. This multiple scattering effect makes the estimation of the atmospheric scattering model extremely complex, resulting in a serious deviation in the estimation of the transmission rate. When existing algorithms are applied to marine images, they often cannot effectively handle this effect, resulting in insufficient defogging, blurred long-range view, overall dark image, color drift and other problems. Therefore, how to effectively deal with the unique multiple scattering effect of the marine environment and achieve accurate estimation of the transmission rate has become a key bottleneck problem in the development of marine image defogging technology. SUMMARY

[0005] The present application aims to provide an image defogging method that can effectively deal with the unique multiple scattering effect of the marine foggy environment, and aims to solve the problems of insufficient defogging, color distortion and detail loss caused by ignoring this effect in the prior art. The present application generates a basic defogging image and a multi-level transmission rate map through a double-branch generator, and corrects the transmission rate deviation caused by multiple scattering in a step-by-step iterative manner, achieving high-quality and high-fidelity image restoration in complex marine fog conditions.

[0006] The technical means adopted by the present application are as follows: The offshore image defogging method based on multi-level transmittance guidance comprises the following steps: A generative adversarial network is constructed, which comprises a generator network for generating a defogged image and a discriminator network for adversarial training with the generator; wherein the generator network comprises a basic defogging network, a multi-level transmittance estimation network and a transmittance-guided defogging engine, the basic defogging network generates a basic defogged image by processing a foggy image based on an encoder-decoder structure, the multi-level transmittance estimation network generates a multi-level transmittance map by processing the foggy image based on multi-level full convolution blocks, and the transmittance-guided defogging engine iteratively optimizes the basic defogged image based on the multi-level transmittance map to generate a defogged image; A training data set is constructed, which comprises clear offshore images and corresponding foggy offshore images; The generative adversarial network is trained based on the training data set; The foggy offshore image to be processed is input into the trained generator network to generate a clear offshore image.

[0007] Further, the encoder of the basic defogging network comprises five convolution blocks for down-sampling the input foggy image to extract deep features, and correspondingly, the decoder of the basic defogging network comprises five deconvolution blocks.

[0008] Further, the multi-level transmittance estimation network comprises seven full convolution blocks, and jump connections are introduced between the first layer and the sixth layer, between the second layer and the fifth layer, and between the third layer and the fourth layer to enhance the feature transmission capability, and the seventh full convolution block adopts a combination of a convolution layer and a Tanh activation function when outputting the transmittance map.

[0009] Further, the transmittance-guided defogging engine optimizes the basic defogged image n times through the following iterative formula:

[0010] wherein, is the defogged result after the n-th guidance, is the result of the last iteration, and the initial state is the basic defogged image, is the n-th transmittance map, is an empirical parameter.

[0011] Further, the discriminator network adopts a PatchGAN architecture, including five serial convolutional blocks, each of which includes a convolutional layer with a step of 2, a BN layer and a LeakyReLU activation function.

[0012] Further, the generative adversarial network is trained based on the training data set, including: The foggy sea image in the training data set is input into the constructed generator network, and the generator network outputs the final defogging image after processing; The defogging image output by the generator and the corresponding real fog-free image in the training data set are simultaneously input into the discriminator network, the discriminator network discriminates the two images, and calculates the loss function; The network parameters of the discriminator network and the generator network are alternately updated by using the loss function through a back propagation algorithm until the model converges.

[0013] Further, the loss function is calculated according to the following formula:

[0014] Wherein, G is the generator network, D is the discriminator network, is the conditional adversarial loss, is the basic defogging network, L 1 reconstruction loss, is the input foggy sea image, is the basic defogging image generated by the basic defogging network, is the overall L 1 reconstruction loss of the generator, is the real fog-free image corresponding to the input image, is the perceptual loss, is a weight coefficient.

[0015] Compared with the prior art, the present application has the following advantages: The application discloses a marine image defogging method based on multistage transmittance guidance, solves the multiple scattering effect caused by large water mist particle radius and high scattering coefficient in a marine foggy environment, and the effect causes the problems of poor defogging effect, easy defogging deficiency or color distortion of existing defogging algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to make the technical personnel of the present technology better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by the ordinary skilled in the art without creative labor should belong to the scope of protection of the present application.

[0017] Figure 1 The model architecture diagram used for the marine image defogging method based on multistage transmittance guidance in the embodiments of the present application.

[0018] Figure 2 The model training flowchart used for the marine image defogging method based on multistage transmittance guidance in the embodiments of the present application.

[0019] Figure 3 The model test flowchart used for the marine image defogging method based on multistage transmittance guidance in the embodiments of the present application. DETAILED DESCRIPTION

[0020] In order to make the technical personnel of the present technology better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by the ordinary skilled in the art without creative labor should belong to the scope of protection of the present application.

[0021] The application provides an offshore image defogging method based on multi-level transmittance guidance, and a network structure thereof is shown in the figure. Figure 1 The network is composed of a double-branch multi-level transmittance guidance generator network (hereinafter referred to as a generator) and a discriminator network (hereinafter referred to as a discriminator). The specific implementation steps of the method will be described in detail below.

[0022] Step 1: Preparation of a data set First, the embodiment constructs a data set for model training. Since it is difficult to collect foggy and fog-free image pairs of the same scene and the same moment in the real world, the method generates synthetic foggy images from clear offshore images using a generative adversarial network (GAN), thereby constructing a paired training sample set. At the same time, real, unpaired offshore foggy images are collected as a test set to verify the generalization ability and effectiveness of the model in real scenarios.

[0023] Step 2: Construction and implementation of the generator network As shown in the generator network part, Figure 1 The embodiment constructs a double-branch generator network. The generator receives a foggy offshore image as input, and its internal structure includes three core components: a basic defogging network (BD-net), a multi-level transmittance estimation network (MTE-net), and a transmittance-guided defogging engine.

[0024] Basic defogging network (BD-net): The basic defogging network is used to generate a basic defogging image with global features and color balance. In the embodiment, the BD-net adopts an encoder-decoder structure, and the encoder is composed of five convolutional blocks. The first and fifth convolutional blocks include a convolutional layer and a LeakyReLU activation function, and the second to fourth convolutional blocks include a convolutional layer, a batch normalization (BN) layer, and a LeakyReLU activation function in sequence. In the embodiment, the convolution kernel size is set to 4x4, the step is 2, and the output channel number is 32, 128, 256, 256, and 256 in sequence. The decoder part corresponds to the encoder structure and is composed of five deconvolutional blocks. Each deconvolutional block includes a deconvolutional layer, a BN layer, and a ReLU activation function, the convolution kernel size is 4x4, the step is 2, and the output channel number is 256, 256, 128, 64, and 3 in sequence. Finally, the defogging result is output through a Tanh activation function. The role of the BD-net is to generate a basic defogging image with color balance and good global structure, providing a stable initial state for subsequent refinement.

[0025] Multi-level Transmittance Estimation Network (MTE-net): The MTE-net generates multi-level transmittance maps to guide the progressive dehazing process. In this embodiment, MTE-net consists of seven fully convolutional blocks, with skip connections introduced between symmetrical layers to enhance feature transfer capability. Each convolutional block uses a combination of convolutional layers and the LeakyReLU activation function, with a kernel size of 3×3, a stride of 1, and output channels of 32, 32, 32, 64, 64, and 64 respectively. The seventh convolutional block outputs a transmittance map using a combination of convolutional layers and the Tanh activation function, with 24 output channels. To maintain spatial correlation between adjacent pixels, downsampling layers or batch normalization layers are not used throughout the network. In this embodiment, the input is a concatenation of an initial transmittance map (estimated using a dark channel prior algorithm) and a hazy image along the channel dimension. This network outputs multi-level transmittance maps to guide the progressive optimization of the basic dehazing network. Furthermore, the generator network also includes a DCP transmittance estimator, which provides prior guidance for transmittance estimation. Specifically, an initial transmittance map of the input hazy image is computed using the Dark Channel Prior (DCP) algorithm. This initial transmittance map is then concatenated with the original hazy image along the channel dimension to form the input data for the multi-level transmittance estimation network, thereby assisting the network to converge more quickly and accurately.

[0026] Transmittance-guided dehazing engine: This engine is the core of this invention for achieving progressive optimization. It receives the base dehazing image generated by BD-net and the multi-level transmittance map generated by the multi-level transmittance estimation network. The base dehazed image is optimized n times using the following iterative formula: (1) In the formula, This is the defogging result after the nth guidance. The initial state is the result of the previous iteration. For the base dehazed image, This is the nth transmittance map. This is an empirical parameter, set to 0.5 in this embodiment, and the number of iterations n is set to 8.

[0027] Step 3: Construction and Implementation of the Discriminator Network like Figure 1 As shown in the discriminator network section, this embodiment constructs a discriminator network for adversarial training against the generator. The discriminator network distinguishes between generated and real images, and its structure adopts the PatchGAN architecture, consisting of five concatenated convolutional blocks. Each convolutional block includes a convolutional layer with a stride of 2, a batch normalization (BN) layer, and a LeakyReLU activation function. The kernel size is 4×4, and the number of output channels is 32, 64, 128, 256, and 1, respectively. This network performs image processing based on local receptive fields.N x N Block discrimination can effectively capture local high-frequency features, and improve the realism and detail fidelity of the defogged image.

[0028] Step 4: Training of the model Referring to Figure 2 , the figure is a model training flowchart of the method of the present application.

[0029] First, the foggy sea image in the training set is input into the constructed generator, and the generator outputs the final defogged image after processing.

[0030] Second, the defogged image output by the generator and the corresponding real fog-free image in the data set are input into the discriminator at the same time. The discriminator discriminates the authenticity of the two images and calculates the discrimination loss.

[0031] At the same time, in order to effectively supervise the training of the generator, a composite loss function is used for optimization. The function is composed of conditional adversarial loss , the L 1 loss acting on the output of the BD-net, L 1 loss acting on the final generated result, and perception loss . These loss functions work together to ensure that the model achieves an optimal balance in pixel accuracy, global structure, and perceptual authenticity.

[0032] Specifically, the loss function is calculated according to the following formula:

[0033] wherein, G is the generator network, D is the discriminator network, is the conditional adversarial loss, is the L 1 reconstruction loss of the basic defogging network, is the input foggy sea image, is the basic defogging image generated by the basic defogging network, L is the 1 reconstruction loss of the generator as a whole, is the real fog-free image corresponding to the input image, is the perception loss,

[0034] is the weight coefficient. Specifically, the conditional adversarial loss function is expressed as follows:

[0035] The goal of the generator is to minimize and discriminator perform the opposite operation. In this adversarial way, the two branches can constantly improve their respective capabilities.

[0036] To make the generator learn the global information better, a 1 loss is adopted to constrain the basic dehazing network, whose expression is:

[0037] At the same time, in order to make ( ) consistent with the true value , a 1 loss is also applied to ( ), whose expression is:

[0038] wherein, symbols , and represent pixel values.

[0039] Finally, a perceptual loss is used to measure the distance between the dehazed image ( ) and the feature representation of the reference image . The perceptual loss is constructed based on the VGG-19 network pre-trained on the ImageNet dataset, and its expression is as follows:

[0040] wherein, represents the feature map obtained by the layer convolution of the VGG-19 network. Specifically, the output of the 5-2 layer of the VGG-19 network is selected to calculate the perceptual loss.

[0041] Finally, by using the above loss, the network parameters of the discriminator and the generator are alternately updated by the back propagation algorithm until the model converges.

[0042] In the present embodiment, the basic dehazing network (BD-net), the multi-level transmittance estimation network (MTE-net) and the discriminator network can all be implemented under the PyTorch deep learning framework, so as to facilitate the rapid construction and parameter optimization of the model. In the training stage, the input image is uniformly adjusted to a resolution of 256x256x3, and the pixel value is normalized to ​1,1] interval, thereby ensuring the numerical scale consistency between different samples. In the model training process, the Adam optimizer is used to update the network parameters, the learning rate is set to 0.0001, and the negative slope of the LeakyReLU activation function is set to 0.2, so as to ensure the balance between the convergence speed and stability of the model. The above parameters such as the size of the convolution kernel, the step and the number of output channels can be adjusted according to the complexity of the task and the hardware conditions, and the technical solutions of the present application can be flexibly realized by the person skilled in the art.

[0043] Step five: image defogging test Referring to Figure 3 The figure is a test flow chart of the method of the present application. When the model training is completed, only the trained generator network is used for image defogging.

[0044] An image to be processed is input.

[0045] The image is sent to the trained generator. The image is passed through the basic defogging network and the multi-level transmittance estimation network in parallel, and the output results are iteratively optimized by the transmittance guided defogging engine.

[0046] The final clear and high-quality defogging image is output, and the defogging process is completed.

[0047] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A sea image defogging method based on multi-level transmittance guidance, characterized in that, The method comprises the following steps: a generative adversarial network is constructed, the generative adversarial network comprising a generator network for generating a defogged image and a discriminator network for adversarial training with the generator; wherein the generator network comprises a basic defogging network, a multi-level transmittance estimation network and a transmittance-guided defogging engine, the basic defogging network processing a foggy image based on an encoder-decoder structure to generate a basic defogged image, the multi-level transmittance estimation network processing the foggy image based on multi-level fully convolutional blocks to generate a multi-level transmittance map, and the transmittance-guided defogging engine iteratively optimizing the basic defogged image based on the multi-level transmittance map to generate the defogged image; a training data set is constructed, the training data set comprising clear sea images and corresponding foggy sea images; the generative adversarial network is trained based on the training data set; a foggy sea image to be processed is input into the trained generator network to generate a clear sea image.

2. The offshore image defogging method based on multi-level transmittance guidance according to claim 1, characterized in that, The encoder of the basic defogging network comprises five convolutional blocks for down-sampling the input foggy image to extract deep features, and the decoder of the basic defogging network comprises five deconvolutional blocks.

3. The offshore image defogging method based on multi-level transmittance guidance according to claim 1, characterized in that, The multi-level transmittance estimation network comprises seven fully convolutional blocks, and a skip connection is introduced between the first layer and the sixth layer, between the second layer and the fifth layer, and between the third layer and the fourth layer to enhance the feature transmission capability, and a convolutional layer and a Tanh activation function are combined when the seventh fully convolutional block outputs the transmittance map.

4. The offshore image defogging method based on multi-level transmittance guidance according to claim 1, characterized in that, The transmittance-guided defogging engine optimizes the basic defogged image n times through the following iterative formula: In the formula, is the defogging result after the n th guidance, is the result of the last iteration, initial state is the base defogging image, is the n th transmittance map, is an empirical parameter.

5. The offshore image defogging method based on multi-level transmittance guidance according to claim 1, characterized in that, The discriminator network adopts a PatchGAN architecture, comprising five serial convolutional blocks, each comprising a convolutional layer with a step size of 2, a BN layer and a LeakyReLU activation function.

6. The offshore image defogging method based on multi-level transmittance guidance according to claim 1, characterized in that, Training the generative adversarial network based on the training data set comprises: inputting the foggy sea images in the training data set into the constructed generator network, and outputting the final defogged image after processing by the generator network; inputting the defogged image output by the generator and the corresponding real fog-free image in the training data set into the discriminator network at the same time, the discriminator network performing authenticity discrimination on the two images and calculating a loss function; updating the network parameters of the discriminator network and the generator network alternately by using the loss function through a back propagation algorithm until the model converges.

7. The offshore image defogging method based on multi-level transmittance guidance according to claim 6, characterized in that, The loss function is calculated according to the following formula: wherein, G is a generator network, D is a discriminator network, is a conditional adversarial loss, is a loss of the base dehazing network L 1 reconstruction loss, is an input hazy sea image, is a base dehazed image generated by the base dehazing network, is a loss of the generator as a whole L 1 reconstruction loss, is a real haze-free image corresponding to the input image, is a perceptual loss, is a weight coefficient.

Images