Underwater image enhancement method and device based on generative adversarial network multi-scale fusion

Abstract

The invention relates to an underwater image enhancement method and device based on generative adversarial network multi-scale fusion, and the method comprises the steps: obtaining an original underwater image, and carrying out the preprocessing of the original underwater image; inputting a preprocessing result into an underwater image enhancement model based on generative adversarial network multi-scale fusion to obtain an enhanced underwater image; the underwater image enhancement model comprises a generator and a discriminator, and the generator is obtained by replacing the last downsampled convolution module in an encoder of the U-Net network with a multi-scale feature fusion module; and the multi-scale feature fusion module is used for extracting multi-scale fusion features by adopting multi-scale convolution, residual connection, dense connection and a global attention mechanism. The method can adapt to different underwater scenes and enhance the image quality, the color cast and brightness recovery of the underwater image are improved, the robustness of the image is kept, and color distortion and noise artifacts are avoided.

Description

Technical Field

[0001] The present application relates to the field of image enhancement technology, and in particular to an underwater image enhancement method and device based on multi-scale fusion of a generative adversarial network. Background Art

[0002] At present, the wide application of underwater tasks has made underwater image enhancement and target detection research driven by application needs. In the fields of marine science and ecology, it is crucial to use underwater vision technology to observe and study marine life, coral reefs and underwater terrain. In addition, underwater tasks also involve underwater mineral exploration, pipeline maintenance, and underwater robot tasks. Understanding the complexity and uniqueness of the underwater environment is the key to success in the field of underwater vision. The intensity of underwater light decreases with increasing depth, resulting in the obvious lack of regional illumination and dark colors in the collected images. The scattering of light and the absorption of light of different wavelengths caused by particles and dissolved substances in the water cause the image to be blurred and the color to be distorted. These characteristics make underwater target acquisition extremely challenging, and traditional image processing technology has limited application effects underwater, manifested in problems such as low contrast, image blur, overlapping target confusion and color distortion. In recent years, the advancement of deep learning technology has provided new possibilities for underwater image enhancement. Deep learning models can learn complex underwater feature representations, thereby improving the performance of image enhancement.

[0003] Underwater image enhancement methods improve image quality by using some general image processing methods. They do not consider the imaging process of the image and some hyperparameters of the imaging process, that is, they only process the image forward. At present, the technical methods for underwater optical image enhancement include but are not limited to image enhancement technology based on spatial domain, image enhancement technology based on transform domain, image enhancement technology based on color constancy, image enhancement technology based on convolutional neural network, and image enhancement technology based on image fusion. However, the existing technology is not ideal for the quality restoration of underwater optical images, and color distortion and noise artifacts may occur. Summary of the invention

[0004] Based on this, it is necessary to provide an underwater image enhancement method and device based on multi-scale fusion of generative adversarial networks to address the above technical problems.

[0005] An underwater image enhancement method based on multi-scale fusion of a generative adversarial network, the method comprising: The original underwater image of the captured image is obtained, and the original underwater image is preprocessed.

[0006] The preprocessing results are input into the underwater image enhancement model based on multi-scale fusion of generative adversarial network to obtain the enhanced underwater image; the underwater image enhancement model based on multi-scale fusion of generative adversarial network includes a generator and a discriminator. The generator is obtained by replacing the last down-sampling convolution module in the encoder of the U-Net network with a multi-scale feature fusion module, and the discriminator is used to guide the generator to generate high-quality images; the multi-scale feature fusion module is used to use multiple convolution kernels of different scales for feature extraction, introduce residual connection and global attention mechanism to extract detail features, splice features of different scales in different channel dimensions, and then output the multi-scale fusion features after passing through the point convolution layer.

[0007] Residual connection and global attention mechanism are introduced to extract detailed features, features of different scales are spliced ​​in different channel dimensions, and then the output is controlled to output multi-scale fusion features after passing through the point convolution layer.

[0008] The underwater image enhancement method and device based on the multi-scale fusion of the generative adversarial network include: obtaining the original underwater image of the captured image, preprocessing the original underwater image; inputting the preprocessing result into the underwater image enhancement model based on the multi-scale fusion of the generative adversarial network to obtain the enhanced underwater image; the underwater image enhancement model based on the multi-scale fusion of the generative adversarial network includes a generator and a discriminator, the generator is obtained by replacing the last down-sampled convolution module in the encoder of the U-Net network with a multi-scale feature fusion module; the multi-scale feature fusion module is used to extract features using multiple convolutions of different scales, introduce residual connections and global attention mechanisms to extract detail features, splice features of different scales in different channel dimensions, and then output the multi-scale fusion features after passing through the point convolution layer. This method can adapt to different underwater scenes and enhance the image quality. While improving the color deviation and brightness restoration of underwater images, it maintains the robustness of the image and avoids the appearance of color distortion and noise artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of a flow chart of an underwater image enhancement method based on multi-scale fusion of a generative adversarial network in one embodiment; Figure 2 A schematic diagram of a generator structure in another embodiment; Figure 3 This is a schematic diagram of the structure of a multi-scale feature fusion module in another embodiment; Figure 4 A schematic diagram of a discriminator structure in another embodiment; Figure 5 It is a schematic diagram of comparison of underwater image enhancement visualization results on a synthetic dataset SUID in another embodiment; Figure 6 It is a schematic diagram of the comparison of underwater image enhancement visualization results on the real dataset URPC2020 in another embodiment; Figure 7 It is a schematic diagram showing the comparison of underwater image enhancement visualization results on a real dataset Trash-ICRA19 in another embodiment; Figure 8 A schematic diagram showing a comparison of visualization results of an ablation experiment of an MF-GAN underwater image enhancement model in another embodiment; Fig. 9 It is a structural block diagram of an underwater image enhancement device based on multi-scale fusion of a generative adversarial network in one embodiment. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0011] An underwater image enhancement method based on multi-scale fusion of generative adversarial networks, referred to as MF-GAN underwater image enhancement method.

[0012] An underwater image enhancement model based on multi-scale fusion of generative adversarial networks, referred to as the MF-GAN underwater image enhancement model.

[0013] In one embodiment, Figure 1 As shown, an underwater image enhancement method based on multi-scale fusion of a generative adversarial network is provided, and the method comprises the following steps: Step 100: Acquire the original underwater image of the captured image and pre-process the original underwater image.

[0014] Specifically, conventional preprocessing is performed on the original underwater image, for example, the data size is adjusted to 256 pixels×256 pixels.

[0015] Step 102: Input the preprocessing result into the underwater image enhancement model based on multi-scale fusion of generative adversarial network to obtain an enhanced underwater image; the underwater image enhancement model based on multi-scale fusion of generative adversarial network includes a generator and a discriminator. The generator is obtained by replacing the last down-sampled convolution module in the encoder of the U-Net network with a multi-scale feature fusion module, and the discriminator is used to guide the generator to generate high-quality images; the multi-scale feature fusion module is used to use multiple convolution kernels of different scales for feature extraction, introduce residual connection and global attention mechanism to extract detail features, splice features of different scales in different channel dimensions, and then output the multi-scale fusion features after passing through the point convolution layer.

[0016] Specifically, the underwater image enhancement model based on multi-scale fusion of generative adversarial networks is based on the generative adversarial network (GAN) architecture, and introduces multi-scale operations, dense connection operations, residual connections, and attention operation mechanisms. In the GAN framework system, the two most important functional modules are the generator and the discriminator. The main task of the generator is to generate data distribution similar to the real data, and the task of the discriminator is to distinguish between generated data and real data. The training process of GAN is to let the generator and the discriminator compete with each other, so that the generator generates more realistic data, and the discriminator becomes more accurate.

[0017] The discriminator can be but is not limited to Patch-GAN or Spectral Normalization discriminator.

[0018] The Spectral Normalization discriminator controls the complexity of the model by introducing spectral normalization in the convolutional layer of the discriminator to prevent the gradient from disappearing or exploding during training. This normalization method can stabilize the training process and improve the convergence speed and generation quality of the Generative Adversarial Network (GAN). The Spectral Normalization discriminator performs well when processing high-resolution images and can better capture the details and textures of the image.

[0019] like Figure 2 As shown in the figure, the generator adopts the U-Net framework as a whole, with some minor changes in details. In the encoding process, after a series of feature extraction stages, 3×3 convolution is used to extract features from the input image, and the output channel is expanded to obtain batch feature maps. Leaky ReLU is used as the activation function to provide nonlinear mapping relationships and enhance the expressiveness of features. Then batch normalization is used to enhance the stability of the network. The downsampling process uses a 2×2 maximum pooling method to extract the main features and reduce the risk of overfitting. A multi-scale feature fusion module (MFFB) is inserted in the middle. The upsampling process, i.e. the extension path of U-Net, uses deconvolution operations to double the feature size. A jump connection is established between the encoder and the decoder to connect the same level layers between the two, so that the network can directly transmit detail information and combine low-level and high-level feature information. The details and context information in the input image are effectively retained to improve the image generation quality.

[0020] The design of the generator needs to take into account the particularities of the target task environment. Underwater images are usually affected by factors such as lighting, scattering, and color distortion, resulting in poor image quality. In addition, there are usually targets of multiple scales in the underwater environment, and there will be overlapping parts between targets. Therefore, a multi-scale feature fusion module is introduced into the generator. This module uses multi-scale fusion, dense connectors, attention and other mechanisms to better help the generator process small underwater targets and subtle textures and overall structural features in underwater images.

[0021] In the underwater image enhancement method based on multi-scale fusion of generative adversarial network, the method includes: obtaining the original underwater image of the captured image, preprocessing the original underwater image; inputting the preprocessing result into the underwater image enhancement model based on multi-scale fusion of generative adversarial network to obtain the enhanced underwater image; the underwater image enhancement model based on multi-scale fusion of generative adversarial network includes a generator and a discriminator, the generator is obtained by replacing the last down-sampled convolution module in the encoder of the U-Net network with a multi-scale feature fusion module; the multi-scale feature fusion module is used to extract features using multiple convolutions of different scales, introduce residual connections and global attention mechanisms to extract detail features, splice features of different scales in different channel dimensions, and then output the multi-scale fusion features after passing through the point convolution layer. This method can adapt to different underwater scenes and enhance the image quality. While improving the color deviation and brightness restoration of underwater images, it maintains the robustness of the image and avoids the appearance of color distortion and noise artifacts.

[0022] In one embodiment, before step 102, a training process of an underwater image enhancement model based on a generative adversarial network multi-scale fusion is also included; the training process includes the following steps: Step 200: Obtain a real underwater image and a corresponding real classification label; Step 202: Construct a total loss function, which is: (1) in, L is the total loss function, To combat losses, For fidelity loss, , are the quality losses of the generator and the discriminator, respectively. , , are the balance parameters for fidelity loss, quality loss generator and quality loss of the discriminator, respectively.

[0023] Specifically, unsupervised image enhancement usually ensures the image enhancement effect through a loss function. Therefore, in order to ensure the stability of the network and restore the true normal light image, this application takes into account the difficulty in obtaining paired datasets of underwater images, and adopts adversarial loss and unsupervised target loss such as image fidelity loss and image quality loss to constrain model training.

[0024] Step 204: Input the real low-quality image into the generator to obtain the generated image. Step 206: Input the generated image and the real underwater image into the discriminator to obtain a discrimination result.

[0025] Step 208: Determine the total loss based on the discrimination result, the true classification label, the true low-quality image, the generated image, the true high-quality image and the total loss function.

[0026] Step 210: The underwater image enhancement model based on multi-scale fusion of a generative adversarial network is trained with the total loss as a constraint to obtain a trained underwater image enhancement model based on multi-scale fusion of a generative adversarial network.

[0027] In one embodiment, the adversarial loss is: (2) in, is the weight of the gradient penalty, are samples from random interpolation between the real data and the generated data, It is used for sampling The distribution of pdata is the distribution of the real data, is the distribution that generated the data, G ( z ) is the sample generated by the generator, D (.) is the discriminator network, D ( x ) is the output of the discriminator for the real sample.

[0028] The fidelity loss is: (3) in, J is the number of layers of the convolutional neural network used for feature extraction and classification, The convolutional neural network is used for feature extraction and classification. j The feature map of the layer.

[0029] The quality losses of the generator and discriminator are: (4) (5) in, x l , x h 、x g They are real low-quality images, real high-quality images and generated images respectively.

[0030] Specifically, the details of the three types of loss functions are as follows: (1) Adversarial Loss The original objective function of the Generative Adversarial Network consists of two parts, corresponding to the generator and the discriminator. They compete with each other to learn the data distribution of generating real samples. In the GAN originally published by Ian, its adversarial loss is defined based on the log-likelihood loss, as follows: (6) in, z is a random vector sampled from the latent space, G ( z ) is the sample generated by the generator, D (.) is the discriminator network. D ( x ) is the output of the discriminator for the real sample, pdata ( x )and pdata ( z ) are the distribution of real data and the distribution of generated data, respectively. During the training process, the goal of the generator is to minimize the generator loss, while the goal of the discriminator is to minimize the discriminator loss. In the actual training process, the network will have some problems that are difficult to ignore, such as imbalance between the generator and the discriminator, mode collapse, and training instability. In response to these problems, existing research has introduced the Wasserstein distance to solve problems such as instability in the traditional GAN ​​training process. It is pointed out that the main reason for the instability of traditional GAN ​​network training is the gradient disappearance caused by the imbalance of the generator and discriminator optimization process. The Wasserstein distance, also known as the earth moving distance, is an indicator used to measure the difference between two probability distributions. The specific definition is as follows: (7) in, P , Q are two given probability distribution variables, ( P , Q )yes P and Q The set of all possible joint distributions between d ( x , y) is the cost function defined on x and y. Compared with JS divergence and cross entropy as distance metrics of traditional GAN ​​loss functions, Wasserstein distance is continuous and differentiable, that is, when there is no overlap between the two distributions, Wasserstein distance can still measure the difference between the two probability distributions, thereby enabling the adversarial generative network to solve the problem of gradient disappearance at the beginning of training. Specifically, the loss function of the discriminator in WGAN is defined as follows: (8) The target loss of the generator is defined as follows: (9) Although WGAN has brought some improvements in stability and generation quality compared to traditional GAN, the weight clipping parameters it introduces, that is, limiting the parameters in the discriminator to be fixed in the range [0.01, 0.01], are very sensitive to the performance and stability of the model. Therefore, gradient penalty is introduced in WGAN-gp to improve the stability of training. The objective function is shown in formula (2). In formula (2), is the discriminant function D The loss function of WGAN-GP is The role of is to penalize the gradient of the discriminator, forcing the norm of the discriminator's gradient to be close to 1. This helps alleviate the problems of mode collapse and training instability, making the training of the generator and discriminator more stable.

[0031] (2) Fidelity loss Fidelity loss is a loss function that ensures the content consistency between the generated image and the input image in an unsupervised training method. The setting of this loss function can avoid the inconsistency between the generated image content and the input image content caused by model transition enhancement. Specifically, fidelity loss is defined as the "2-norm" distance between the feature map of the input low-quality image and the feature map of the high-quality image extracted by the pre-trained VGG network. The specific definition is shown in formula (3).

[0032] (3) Quality loss Quality loss refers to a loss function that is proposed based on the relativistic discriminator structure to guide the generator to generate images with higher quality than real images. Specifically, the discriminator of the traditional GAN ​​only judges the probability that the generated data is real data, which cannot well guide the generator to convert low-quality images into high-quality images. The reason is that the discriminator has no clear understanding of the quality of the image. Not only that, even different people have different subjective perceptions of the quality of the image. Therefore, in order to enhance the quality of the generated images, improvements are made based on the relativistic discriminator, and the discriminator is trained using real low-quality images and real high-quality images to increase the discriminator's ability to distinguish low-quality and high-quality images, thereby guiding the generator to play the role of image enhancement.

[0033] The quality loss of the generator and discriminator are shown in formula (4) and formula (5) respectively.

[0034] In one embodiment, the generator includes: an encoder and a decoder; wherein the encoder includes three convolution modules, three downsampling layers and a multi-scale feature fusion module; the decoder includes three convolution modules, three upsampling layers and a point convolution layer; the convolution module includes two convolution layers with convolution kernels of 3×3 and a ReLU activation function; step 204 includes: inputting the real low-quality image into the first convolution module of the encoder to obtain a first encoding feature; inputting the first encoding feature into the second convolution module of the encoder after being processed by the first downsampling layer to obtain a second encoding feature; inputting the second encoding feature into the third convolution module of the encoder after being processed by the second downsampling layer to obtain a third encoding feature; and The feature is processed by the third down-sampling layer and input into the multi-scale feature fusion module to obtain the multi-scale fusion feature; the multi-scale fusion feature is processed by the third upsampling layer and fused with the third encoding feature, and the fusion result is processed by the third convolution module of the decoder to obtain the third decoding feature; the third decoding feature is processed by the second upsampling layer and fused with the second encoding feature, and the fusion result is processed by the second convolution module of the decoder to obtain the second decoding feature; the second decoding feature is processed by the first upsampling layer and fused with the second encoding feature, and the fusion result is processed by the first convolution module of the decoder to obtain the first decoding feature; the first decoding feature is input into the point convolution layer to obtain the generated image.

[0035] In one embodiment, the multi-scale feature fusion module includes: two multi-scale convolution feature extraction modules, one global attention module and one first point convolution layer; the third encoded feature is input into the multi-scale feature fusion module after being processed by the third lower layer to obtain a multi-scale fusion feature, including: inputting the third encoded feature into the first multi-scale convolution feature extraction module to obtain a first multi-scale feature; splicing the first multi-scale feature with the third encoded feature to obtain an intermediate feature; inputting the intermediate feature into the second multi-scale convolution feature extraction module to obtain a second multi-scale feature; processing the third encoded feature through the global attention module to obtain an attention feature; splicing the third encoded feature, the second multi-scale feature and the attention feature, and processing the spliced ​​result through the first point convolution layer and the Leaky ReLU activation function to obtain a multi-scale fusion feature.

[0036] Specifically, Figure 3 As shown in the figure, the multi-scale feature fusion module passes the input feature map through three convolution kernels of sizes 7×7, 5×5, and 3×3 in sequence, so as to extract features at different scales, and then splices the features of different scales in the channel dimension. In this module, the step size of the convolution kernel is set to 1, which is convenient for the subsequent splicing operation of feature maps of different scales. At the bottleneck of this module, a 1×1 convolution kernel is used to control the number of output channels. At the same time, in order to extract more details, a residual connection is introduced, that is, the residual connection is connected to the global attention module. This is an attention mechanism module commonly used in convolutional network feature extraction. This module combines the channel attention module and the spatial attention module to obtain the joint attention weights in the channel and space, thereby more effectively learning and utilizing the information in the image.

[0037] The global attention module can be but is not limited to a CBAM module or a CA module.

[0038] The CA module accurately encodes channel relationships and long-range dependencies through two steps: coordinate information embedding and coordinate attention generation. The CA module obtains attention on image width and height through global average pooling, capturing long-term dependencies in spatial directions, which helps the model better locate objects of interest. ‌‌

[0039] Figure 3 In the neural network, the Leaky Relu activation function is introduced, and each layer can receive the feature maps of all previous layers, thereby enhancing the reuse and propagation of features.

[0040] In one embodiment, the multi-scale convolutional feature extraction module includes three parallel convolution branches, each convolution branch includes one convolution layer and a Leaky ReLU activation function; the convolution layers of the three parallel convolution branches are: a convolution layer with a convolution kernel of 7×7, a convolution layer with a convolution kernel of 5×5, and a convolution layer with a convolution kernel of 3×3.

[0041] In one embodiment, the global attention module is a CBAM module.

[0042] In one embodiment, the discriminator is Patch-GAN.

[0043] Specifically, when training a generative adversarial network, the output of the discriminator is usually scaled to the range [0,1] through a Sigmoid activation function, and the output is interpreted as the probability that the generated image is a real image, thereby minimizing the probability between the generated image and the real image. In the MF-GAN image enhancement model algorithm, in order to better capture the local features and texture of the image, Patch-GAN is introduced, such as Figure 4 As shown in the figure, the discriminator consists of five 4×4 and one 1×1 convolution kernels. The discriminator divides the image into many overlapping small blocks, and then judges the authenticity probability of each small block separately, which helps to improve the network's perception of the image. At the same time, compared with the global discriminator, the discriminator only needs to judge the small blocks of the image, which can reduce the computational cost.

[0044] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0045] In a verification embodiment, the network framework in the experimental environment is built using Pytorch1.8 and Python3.8. Other dependent libraries include NumPy, CV2, and Matplotlib. All experiments are conducted on a server equipped with four 2080Ti graphics cards. In order to more comprehensively evaluate the performance of the image enhancement method, enrich the diversity of data, and improve the generalization ability of the model, this patent uses the synthetic data set SUID, the real data set URPC2020, and the real data set Trash-ICRA19 for training. The training process of each data set includes three stages. The first is the initialization stage of training. The image needs to be preprocessed in a conventional way, such as adjusting the data size to 256×256, selecting Adam as the optimizer of the network, and randomly initializing the weights of the generator and the discriminator according to the normal distribution. In order to reasonably ensure the convergence of the network, the initialization learning rate is set to 0.0005, the batch size is set to 16, and the number of iterations is set to 200. Then enter the adversarial training stage, continuously optimize the generator's ability to generate high-quality underwater image samples, and reach the convergence stage. Finally, fine-tune the network.

[0046] (1) Quantitative and qualitative analysis In order to verify the effect of the model, the image enhancement effect of the network model was compared with six image enhancement methods, including two traditional image enhancement methods (WB, CLAHE), two physical prior-based image enhancement methods (UDCP, ULAP), and three deep learning-based image enhancement methods (UGAN, FGAN, Ucolor). The specific comparative analysis method includes qualitative and quantitative analysis methods, that is, qualitative comparative analysis of the image enhancement effects of different models is performed by using visualization, and image quality is quantitatively evaluated using commonly used image quality evaluation indicators.

[0047] Specifically, for the synthetic dataset SUID, since it has data samples for paired tests, this section uses the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) to evaluate the image quality enhancement effect. For real datasets URPC2020 and Trash-ICRA19 without high-quality image references, this patent uses two no-reference evaluation indicators in the field of natural image enhancement and two no-reference evaluation indicators specifically for underwater image quality, including the Image Evaluation Spatial Quality Evaluator (BRISQUE), the Natural Image Quality Evaluator NIQE (Natural Image Quality Evalution), the Underwater Image Quality Meter UIQM (Underwater Image Quality Measure) and the Underwater Color Image Quality Evaluator UCIQE (Underwater Color Image Quality Evalution) indicators for evaluation.

[0048] Peak signal-to-noise ratio (PSNR) measures the degree of image distortion based on the error between pixels and is defined by the mean square error. The PSNR result is in decibels, and the value is usually between 10 and 50, with the higher the value, the better the image quality. However, PSNR only considers the global mean square error, and may not provide an accurate assessment of detail changes that the human eye is not sensitive to. Therefore, in some scenarios, the structural similarity index can be used to obtain a more comprehensive image quality assessment.

[0049] The structural similarity index (SSIM) comprehensively considers the brightness, contrast and structure of the image. By comparing the similarity of two images from these three dimensions, it provides a more comprehensive image quality evaluation. The value range of SSIM is between -1 and 1. The closer the value is to 1, the more similar the two images are, and the closer the value is to -1, the greater the difference between the two images.

[0050] Image Spatial Quality Evaluator (BRISQUE) is a global reference-free image quality assessment method. Its principle is to extract the feature vector of the local normalized features of the image, establish a regression model for it, and fit the mixed feature distribution from the high-quality and low-quality image training data to obtain a quality score. The lower the score, the better the image quality. Specifically, this method extracts more adaptive features in the image to be tested by introducing the MSCN (Mean Substracted Contrast Normalized) coefficient, and uses the multiplicative relationship between adjacent pixels to capture the connection between adjacent pixels, that is, the adjacent relationship is modeled from four directions: horizontal, vertical, main diagonal, and sub-diagonal. In the process of calculating the feature vector, the fitted generalized Gaussian distribution GGD (Generalized Gaussian Distribution) model parameters are used to obtain the GGD distribution of the MSCN parameters. At the same time, the asymmetric generalized Gaussian distribution (AGGD) is used to fit the adjacent raw product parameters to obtain a complete feature vector. Finally, the feature vector is input into a learning algorithm such as a support vector machine for training to obtain the final quality score of each distortion.

[0051] The Natural Image Quality Evaluator (NIQE) is used to evaluate the degree of degradation of the image under test compared to the natural image. This is a no-reference evaluation model. The image quality evaluation process of this model includes spatial NSS feature extraction, image block local feature extraction, Gaussian mixture model fitting training and quality scoring. The quality of the distorted image is evaluated by calculating the multivariate Gaussian distribution MVG distance of the natural image and distorted image features. The lower the NIQE score, the closer the distorted image is to the natural image, that is, the higher the image quality. The design goal of this image evaluation method is to be applicable to a wide range of natural images, not just limited to specific types of images.

[0052] The Underwater Image Quality Meter (UIQM) is a method specifically used to evaluate underwater image quality. This method uses a combination of color measurement (UICM), contrast measurement (UISM), and clarity (UIConM) to evaluate underwater image quality features such as clarity and color reproduction. The higher the UIQM value, the better the image quality.

[0053] The Underwater Color Image Quality Evaluator (UCIQE) is usually used to evaluate the quality of underwater color images. This evaluation method quantifies the color cast and blur of the image through chromaticity, contrast and saturation. The higher the UCIQE value, the better the image quality.

[0054] Tables 1, 2, and 3 are the comparison results of quantitative indicator evaluation on the underwater synthetic dataset SUID, the real underwater dataset URPC2020, and Trash-ICRA19, respectively. Figure 5, Figure 6 and Figure 7 The following are visualization comparisons of the enhancement results of some images in the three datasets. In general, compared with other image enhancement methods, the MF-GAN underwater image enhancement method proposed in this application can comprehensively consider factors such as color balance and contrast according to specific circumstances, and will not produce red artifacts, showing good results in both synthetic and real datasets.

[0055] According to Table 1 and Figure 5 Analysis shows that on the synthetic dataset SUID, the WB, UGAN, and FGAN methods repaired the color cast of the unprocessed images to varying degrees, and the CLAHE, UDCP, UGAN, Ucolor, and FGAN methods defogged the original images to varying degrees, but the UDCP processing method made the overall brightness of the image darker, which made its experimental results not meet the expectations of underwater image quality enhancement. In comparison, the MF-GAN underwater image enhancement method proposed in this application achieves defogging, color correction, and brightness balance of the original image, effectively enhancing the overall quality of the underwater image and making it more suitable for human eye observation.

[0056] According to Table 2 and Figure 6 Analysis shows that on the URPC2020 underwater real dataset, the WB, CLAHE, Ucolor, and FGAN methods performed different degrees of color correction and dehazing during image processing, but the WB method made the image color brighter, and the UDCP method still lacked brightness in image processing. The UGAN method improved the image detail performance to a certain extent, making the structure, texture, and features of the image clearer. Compared with these methods, the MF-GAN underwater image enhancement method proposed in this application performed slightly better in this dataset. Compared with the Ucolor method, this method makes the image color richer in color correction processing.

[0057] According to Table 3 and Figure 7 According to the analysis, on the Trash-ICRA19 underwater real dataset, CLAHE, UGAN, Ucolor, and FGAN methods dehazed the original images to varying degrees. UDCP used dark channel priors to estimate the conduction blur and color shift in underwater images, resulting in a reduction in the overall image brightness, especially in the exposed area. ULAP made a strong adjustment to the color of underwater images during processing, resulting in the overall image appearing red. Compared with traditional methods, methods based on deep learning are less likely to suffer from over-color correction. The method proposed in this application also performed well on this dataset.

[0058] Table 1 Comparison of full-reference evaluation index results on the synthetic dataset SUID

[0059] Table 2 Comparison of no-reference evaluation index results on the real dataset URPC2020

[0060] Table 3 Comparison of no-reference evaluation index results on the real dataset Trash-ICRA19

[0061] 2) Ablation experiment In order to verify the effectiveness of each module, this embodiment conducts ablation experiments on the model components on the URPC2020 dataset and evaluates the corresponding experimental results. Specifically, for the overall structure of the MF-GAN underwater image enhancement model, this embodiment will evaluate the experimental results of the model when removing the residual learning operation (-RL), removing the attention mechanism operation (-CBAM), removing the dense cascade operation (-DC) and removing the multi-scale feature fusion module (-MFFB). The experimental results are shown in Table 4. Comparison of the visualization results of the ablation experiment of the MF-GAN underwater image enhancement model Figure 8 shown.

[0062] From the overall experimental results, when the residual operation, attention mechanism operation, dense connection operation and multi-scale structure operation are removed, the model's underwater image enhancement effect is significantly reduced under different evaluation indicators. At the same time, it can be observed that the residual connection has a significant impact on the model, and it is effective in improving the UIQM and UCIQE quality evaluation indicators of underwater images. Both indicators can reflect comprehensive information such as color, clarity and contrast of the image. Compared with the residual operation, the attention operation focuses on specific areas in the image and enriches the details of the image. From the experimental results, it is observed that the attention mechanism has little effect on the underwater image quality enhancement effect. Because it is effective in emphasizing the detailed structural features of the image, but has little effect on the global enhancement of the image. Dense connection (DC) performs well in the evaluation indicators NIQE and BRISQUE, and performs poorly in the UIQM and UCIQE evaluation indicators, which shows that dense connection helps to eliminate image artifacts, but it is difficult to improve the color deviation problem of underwater images. The multi-scale fusion module (MFFB) has a significant improvement in the evaluation indicators, because this module contains residual operation, attention operation and dense operation. And the overall consistency of the image is maintained through multi-scale characteristics.

[0063] Table 4 Comparison of ablation test results of MF-GAN underwater image enhancement model under different components

[0064] In one embodiment, Fig. 9 As shown, an underwater image enhancement device based on multi-scale fusion of a generative adversarial network is provided, comprising: an original underwater image acquisition module and an underwater image enhancement module, wherein: The original underwater image acquisition module is used to acquire the original underwater image of the captured image and pre-process the original underwater image; The underwater image enhancement module is used to input the preprocessing results into the underwater image enhancement model based on multi-scale fusion of generative adversarial networks to obtain an enhanced underwater image; the underwater image enhancement model based on multi-scale fusion of generative adversarial networks includes a generator and a discriminator. The generator is obtained by replacing the last down-sampling convolution module in the encoder of the U-Net network with a multi-scale feature fusion module, and the discriminator is used to guide the generator to generate high-quality images; the multi-scale feature fusion module is used to use multiple convolution kernels of different scales for feature extraction, introduce residual connections and global attention mechanisms to extract detail features, splice features of different scales in different channel dimensions, and then output the multi-scale fusion features after passing through the point convolution layer.

[0065] In one embodiment, the underwater image enhancement module also includes a training module of an underwater image enhancement model based on a generative adversarial network multi-scale fusion, which is used to obtain a real underwater image and a real classification label for use; construct a total loss function, and the total loss function is shown in formula (1). The real low-quality image is input into the generator to obtain a generated image; the generated image and the real underwater image are input into the discriminator to obtain a discrimination result; the total loss is determined according to the discrimination result, the real classification label, the real low-quality image, the generated image, the real high-quality image and the total loss function; the underwater image enhancement model based on the generative adversarial network multi-scale fusion is trained with the total loss as a constraint to obtain a trained underwater image enhancement model based on the generative adversarial network multi-scale fusion.

[0066] In one embodiment, the adversarial loss in the training module of the underwater image enhancement model based on the generative adversarial network multi-scale fusion is shown in formula (2). The fidelity loss is shown in formula (3). The quality loss of the generator and the discriminator are shown in formula (4) and formula (5) respectively.

[0067] In one embodiment, the generator includes: an encoder and a decoder; wherein the encoder includes three convolution modules, three downsampling layers and a multi-scale feature fusion module; the decoder includes three convolution modules, three upsampling layers and a point convolution layer; the convolution module includes two convolution layers with convolution kernels of 3×3 and a ReLU activation function; the training module of the underwater image enhancement model based on the multi-scale fusion of the generative adversarial network is also used to input the real low-quality image into the first convolution module of the encoder to obtain the first encoded feature; the first encoded feature is input into the second convolution module of the encoder after being processed by the first downsampling layer to obtain the second encoded feature; the second encoded feature is input into the third convolution module of the encoder after being processed by the second downsampling layer to obtain The third encoding feature; the third encoding feature is input into the multi-scale feature fusion module after being processed by the third down-sampling layer to obtain the multi-scale fusion feature; the multi-scale fusion feature is fused with the third encoding feature after being processed by the third upsampling layer, and the fusion result is processed by the third convolution module of the decoder to obtain the third decoding feature; the third decoding feature is fused with the second encoding feature after being processed by the second upsampling layer, and the fusion result is processed by the second convolution module of the decoder to obtain the second decoding feature; the second decoding feature is fused with the second encoding feature after being processed by the first upsampling layer, and the fusion result is processed by the first convolution module of the decoder to obtain the first decoding feature; the first decoding feature is input into the point convolution layer to obtain the generated image.

[0068] In one embodiment, the multi-scale feature fusion module includes: two multi-scale convolution feature extraction modules, one global attention module and one first point convolution layer; a training module of an underwater image enhancement model based on multi-scale fusion of a generative adversarial network is also used to input the third coding feature into the first multi-scale convolution feature extraction module to obtain a first multi-scale feature; after splicing the first multi-scale feature with the third coding feature, an intermediate feature is obtained; the intermediate feature is input into the second multi-scale convolution feature extraction module to obtain a second multi-scale feature; after processing the third coding feature through the global attention module, an attention feature is obtained; the third coding feature, the second multi-scale feature and the attention feature are spliced, and the spliced ​​result is processed through the first point convolution layer and the Leaky ReLU activation function to obtain a multi-scale fusion feature.

[0069] In one embodiment, the multi-scale convolutional feature extraction module includes three parallel convolution branches, each convolution branch includes one convolution layer and a Leaky ReLU activation function; the convolution layers of the three parallel convolution branches are: a convolution layer with a convolution kernel of 7×7, a convolution layer with a convolution kernel of 5×5, and a convolution layer with a convolution kernel of 3×3.

[0070] In one embodiment, the global attention module is a CBAM module.

[0071] In one of the embodiments, the discriminator in the underwater image enhancement module is Patch-GAN.

[0072] For the specific definition of the underwater image enhancement device based on generative adversarial network multi-scale fusion, please refer to the definition of the underwater image enhancement method based on generative adversarial network multi-scale fusion mentioned above, which will not be repeated here. Each module in the above-mentioned underwater image enhancement device based on generative adversarial network multi-scale fusion can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0073] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An underwater image enhancement method based on multi-scale fusion of generative adversarial networks, characterized in that: The method comprises: Acquire an original underwater image of the captured image, and preprocess the original underwater image; The preprocessing result is input into an underwater image enhancement model based on multi-scale fusion of a generative adversarial network to obtain an enhanced underwater image; the underwater image enhancement model based on multi-scale fusion of a generative adversarial network includes a generator and a discriminator, the generator is obtained by replacing the last down-sampling convolution module in the encoder of the U-Net network with a multi-scale feature fusion module, and the discriminator is used to guide the generator to generate high-quality images; the multi-scale feature fusion module is used to use multiple convolution kernels of different scales for feature extraction, introduce residual connections and global attention mechanisms to extract detail features, splice features of different scales in different channel dimensions, and then output the control output multi-scale fusion features after passing through the point convolution layer.

2. The underwater image enhancement method based on multi-scale fusion of generative adversarial network according to claim 1 is characterized in that: The preprocessing result is input into an underwater image enhancement model based on a generative adversarial network multi-scale fusion to obtain an enhanced underwater image, and the step also includes a training process of the underwater image enhancement model based on a generative adversarial network multi-scale fusion; The training process includes: Obtain real underwater images and their corresponding real classification labels; Construct a total loss function, which is: in, L is the total loss function, To combat losses, For fidelity loss, , are the quality losses of the generator and the discriminator, respectively. , , are the balance parameters for the fidelity loss, quality loss generator, and quality loss of the discriminator, respectively; Input the real low-quality image into the generator to obtain the generated image; Inputting the generated image and the real underwater image into a discriminator to obtain a discrimination result; Determine the total loss according to the discrimination result, the true classification label, the true low-quality image, the generated image, the true high-quality image and the total loss function; The total loss is used as a constraint to train the underwater image enhancement model based on multi-scale fusion of a generative adversarial network, thereby obtaining a trained underwater image enhancement model based on multi-scale fusion of a generative adversarial network.

3. The underwater image enhancement method based on multi-scale fusion of generative adversarial networks according to claim 2 is characterized in that: The adversarial loss is: in, is the weight of the gradient penalty, are samples from random interpolation between the real data and the generated data, It is used for sampling The distribution of pdata is the distribution of the real data, is the distribution that generated the data, G ( z ) is the sample generated by the generator, D (.) is the discriminator network, D ( x ) is the output of the discriminator for the real sample; The fidelity loss is: in, J is the number of layers of the convolutional neural network used for feature extraction and classification, The convolutional neural network is used for feature extraction and classification. j Feature map of the layer; The quality losses of the generator and discriminator are: in, x l , x h 、x g They are real low-quality images, real high-quality images and generated images respectively.

4. The underwater image enhancement method based on multi-scale fusion of generative adversarial networks according to claim 2 is characterized in that: The generator includes: an encoder and a decoder; wherein the encoder includes three convolution modules, three downsampling layers and one multi-scale feature fusion module; the decoder includes three convolution modules, three upsampling layers and one point convolution layer; the convolution module includes two convolution layers with a convolution kernel of 3×3 and a ReLU activation function; Input the real low-quality image into the generator to obtain the generated image, including: Input the real low-quality image into the first convolution module of the encoder to obtain the first encoded feature; Inputting the first coding feature into a second convolutional module of the encoder after being processed by the first lower layer to obtain a second coding feature; Inputting the second encoded feature into a third convolutional module of the encoder after being processed by a second lower layer to obtain a third encoded feature; Inputting the third encoded feature into the multi-scale feature fusion module after being processed by the third lower layer to obtain a multi-scale fusion feature; The multi-scale fusion feature is processed by the third upsampling layer and then fused with the third encoding feature, and the fusion result is processed by the third convolution module of the decoder to obtain a third decoding feature; Processing the third decoding feature through the second upsampling layer and fusing it with the second encoding feature, and processing the fusion result through the second convolution module of the decoder to obtain a second decoding feature; The second decoding feature is processed by the first upsampling layer and then fused with the second encoding feature, and the fusion result is processed by the first convolution module of the decoder to obtain a first decoding feature; The first decoded feature is input into the point convolution layer to obtain a generated image.

5. The underwater image enhancement method based on multi-scale fusion of generative adversarial networks according to claim 4 is characterized in that: The multi-scale feature fusion module includes: two multi-scale convolution feature extraction modules, a global attention module and a first point convolution layer; The third encoded feature is input into the multi-scale feature fusion module after being processed by the third lower layer to obtain a multi-scale fusion feature, including: Inputting the third encoded feature into a first multi-scale convolutional feature extraction module to obtain a first multi-scale feature; After concatenating the first multi-scale feature with the third encoding feature, an intermediate feature is obtained; Inputting the intermediate features into a second multi-scale convolutional feature extraction module to obtain second multi-scale features; Processing the third encoding feature through the global attention module to obtain an attention feature; The third encoding feature, the second multi-scale feature and the attention feature are spliced, and the splicing result is processed by the first point convolution layer and the Leaky ReLU activation function to obtain a multi-scale fusion feature.

6. The underwater image enhancement method based on multi-scale fusion of generative adversarial networks according to claim 5 is characterized in that: The multi-scale convolution feature extraction module includes three parallel convolution branches, each of which includes a convolution layer and a Leaky ReLU activation function; the convolution layers of the three parallel convolution branches are: a convolution layer with a convolution kernel of 7×7, a convolution layer with a convolution kernel of 5×5, and a convolution layer with a convolution kernel of 3×3.

7. The underwater image enhancement method based on multi-scale fusion of generative adversarial networks according to claim 5 is characterized in that: The global attention module is a CBAM module.

8. The underwater image enhancement method based on multi-scale fusion of generative adversarial networks according to claim 1 is characterized in that: The discriminator is Patch-GAN.

9. An underwater image enhancement device based on multi-scale fusion of generative adversarial networks, characterized in that: The device comprises: An original underwater image acquisition module is used to acquire the original underwater image of the captured image and pre-process the original underwater image; The underwater image enhancement module is used to input the preprocessing results into the underwater image enhancement model based on multi-scale fusion of generative adversarial networks to obtain an enhanced underwater image; the underwater image enhancement model based on multi-scale fusion of generative adversarial networks includes a generator and a discriminator, the generator is obtained by replacing the last down-sampling convolution module in the encoder of the U-Net network with a multi-scale feature fusion module, and the discriminator is used to guide the generator to generate high-quality images; the multi-scale feature fusion module is used to use multiple convolution kernels of different scales for feature extraction, introduce residual connections and global attention mechanisms to extract detail features, splice features of different scales in different channel dimensions, and then output the multi-scale fusion features after passing through the point convolution layer.

Images