An image enhancement method based on underwater attention generative adversarial network

Abstract

This invention discloses an image enhancement method based on underwater attention generative adversarial networks (UAGANs), comprising the following steps: acquiring an underwater image to be enhanced; constructing an attention generative adversarial network (UAGAN) by extracting low-level and high-level semantic information using a convolutional neural network through supervised learning; training the UAGAN using the underwater image to be enhanced as input with generative adversarial loss, loss function, and structural similarity loss to obtain a trained UAGAN, and outputting the enhanced underwater image. The supervised learning method, by extracting low-level and high-level semantic information using a convolutional neural network, enables the designed UAGAN framework to have stronger generalization capabilities. Simultaneously, the proposed UAGAN framework avoids the problems of parameter estimation and underwater imaging modeling. Combining positional attention and PatchGAN techniques helps capture long-range dependency information, thereby preventing over-enhancement of the generated underwater image.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of underwater visual intelligent perception, and relates to an image enhancement method based on an underwater attention generative adversarial network. BACKGROUND

[0002] Underwater optical imaging technology can provide high-resolution images consistent with human visual perception, and has become an effective means for exploring and understanding the ocean. It is noted that there are a large number of dissolved substances, suspended substances and particulate matters in complex marine environments, and suspended particles absorb, scatter and refract light, resulting in color distortion, detail blurring and low contrast in captured images, which seriously reduces the quality of underwater image imaging, inevitably bringing severe challenges to underwater tasks based on vision.

[0003] At present, aiming at the low contrast, detail blurring and noise problems unique to underwater images, the mainstream underwater image enhancement methods include: non-physical model-based image enhancement method, physical model-based underwater image restoration method and data-driven image-to-image conversion method.

[0004] (1) Non-physical model-based underwater image enhancement method;

[0005] Generally, the non-physical model-based underwater image enhancement method realizes the clarification of the image by directly adjusting the pixel value, which includes histogram equalization and its evolution method, white balance series algorithm, domain transformation method. The histogram equalization framework mainly solves the problem of small dynamic range of pixel distribution and concentrated distribution. It mainly uses the gray scale transformation function to make the distribution range of the histogram wider, thereby effectively improving the contrast of the underwater image; the limited contrast adaptive histogram equalization method divides the underwater image into multiple regions, and applies different gray mapping functions to different image regions, thereby effectively solving the problem of over-enhancement or under-enhancement of local image regions; in addition, the white balance series algorithm mainly adjusts the image contrast and corrects the color based on the idea of color temperature; in addition, researchers also convert the underwater optical image from the spatial domain to the transform domain, fully utilize the related characteristics of the transform domain for processing, and then convert the underwater optical image from the transform domain to the spatial domain, thereby realizing the enhancement of the underwater optical image.

[0006] (2) Physical model-based underwater image restoration method;

[0007] By exploring the existence of a certain amount of dark pixels in the local area of the haze-free image (that is, the pixel value of a certain channel in the three channels is close to 0), a dark channel prior method is proposed, which can effectively restore the image color and improve the image contrast; combined with the information that the red channel of the underwater optical image decays most seriously, an underwater dark channel enhancement framework is proposed, which mainly uses green channel and blue channel information in the process of solving the dark channel. By using the color information related to the depth change of the scene to estimate the background light, a generalized dark channel algorithm is proposed, which can effectively adjust the color of the image. By fusing the scoring mechanism of multiple prior knowledge in the process of estimating the backscattering light, a generalized underwater dark channel prior method is proposed, which can more effectively estimate the transmission map, thereby helping to improve the enhancement of underwater images.

[0008] (3) Data-driven image-to-image conversion method

[0009] The U-Net network structure has been proven to be extremely effective in the field of image enhancement. With the help of encoding and decoding structure, the use of convolution and deconvolution mode, combined with the jump connection operation can effectively save the low-level information of the image, and prevent gradient disappearance at the same time; the UGAN framework can effectively guide the generator to generate images that conform to the distribution of real images by means of the generation of an adversarial mechanism, thereby effectively avoiding the cumbersome loss function design; the Wasserstein GAN method can make the generated image have characteristics consistent with human perception by means of perceptual loss. The GAN-RS method realizes image content preservation and noise filtering by using the adversarial branch and the evaluation branch respectively, and the formation of the true sample is mainly based on the filtering method.

[0010] The underwater image enhancement technology in complex multi-domain scenes mainly has the following defects: (1) the method of directly adjusting the pixel value to realize the underwater image enhancement does not have generalization, and the method of realizing image enhancement based on the underwater imaging model has modeling difficulty, transmission map difficult to obtain, and absorption and scattering coefficient difficult to accurately estimate; (2) in the process of feature extraction, speckle noise, Gaussian noise and impulse noise are inevitably introduced, from the channel level, the existing method has not effectively suppressed the above-mentioned mixed noise, so that it is difficult to produce images that conform to human visual perception; (3) the strategy of relying only on a single activation value of the last fully connected layer or the convolution layer patch activation value to judge the underwater sample generated by the generator is easy to make the generator produce over-enhancement or under-enhancement phenomenon. SUMMARY

[0011] In order to solve the above problems, the technical scheme adopted by the present application is as follows: an image enhancement method based on underwater attention generative adversarial network, comprising the following steps:

[0012] An underwater image to be enhanced is obtained.

[0013] By means of the supervised learning mode, the attention generative adversarial network is constructed by means of a convolutional neural network to extract low-level and high-level semantic information;

[0014] The underwater image to be enhanced is taken as input, the attention generative adversarial network is trained by using a generative adversarial loss, a loss and a structural similarity loss, a trained attention generative adversarial network is obtained, and an enhanced underwater image is output.

[0015] Further, the attention generative adversarial network combines a single-hidden-layer neural network architecture by means of global max-pooling and average-pooling operations to form a cascaded dense channel attention module, so as to realize channel weight redistribution.

[0016] Further, the attention generative adversarial network captures long-distance dependency information by combining position attention and PatchGAN technology.

[0017] Further, the output of the cascaded dense channel attention module is obtained in the following manner:

[0018] A dense connection network is used to extract discriminative noise features and enhanced feature flow, which can be expressed as:

[0019]

[0020] Wherein: represents the output of the lth basic module in the dense network, represents the original input feature, represents the output of the dense network, T l (·) is a nonlinear conversion including batch normalization, LeakyReLU and convolution operation, and O(·) represents a 1x1 convolution operation;

[0021] Statistical information of each feature map is obtained by using global average pooling and global maximum pooling operations, which can be expressed as:

[0022]

[0023]

[0024] Wherein, h, w and c represent the height, width and number of layers of the channel feature map respectively, k = 1, 2, …, c, represents the kth channel feature map, and represent the outputs of the global average pooling and the global maximum pooling respectively;

[0025] Two completely independent fully connected layers are used to calculate the channel weight, which can be expressed as:

[0026]

[0027] Where *∈{GAP,GMP}, and These are learnable fully connected weight parameters, where R(·) and S(·) are the ReLU and Sigmoid activation functions, respectively. Clearly, the channel weights... It is restricted to the range (0,1);

[0028] The output of the entire cascaded dense channel attention module can be expressed as:

[0029]

[0030] in, It is the output of the entire cascaded dense channel attention module.

[0031] Furthermore: the attention-based generative adversarial network calculates the interactivity between any two non-local spatial locations through positional attention, which can be expressed as:

[0032]

[0033] Where i = 1, 2, ..., n and j = 1, 2, ..., n, and These respectively represent features derived from discriminative features. The reconstructed feature map, where n = hw is the total number of features. Represents the location attention map, β j,i Used to measure the positional weights between the i-th and j-th positions in the same feature map;

[0034] Positional attention can be represented as:

[0035]

[0036] in: This represents the predicted positional attention feature at the j-th position. The representation is the location attention feature map, γ is the weighting factor, and h i and a j These are reconstructed feature maps. and The i-th and j-th elements.

[0037] Furthermore, the optimization function of the attention-based generative adversarial network is obtained as follows:

[0038] To avoid gradient vanishing and mode collapse during training, the classic optimization function WGAN-GP is used, which is described as follows:

[0039]

[0040] in and These represent the probability distributions of real images and generated images, respectively. It is the penalty domain, λ gp G(·) and D(·) represent weighting factors, respectively, and the generator and discriminator are the generator and discriminator.

[0041] Secondly, the L1 loss is used to capture low-frequency information between the reference image and the generated image, and it can be expressed as:

[0042]

[0043] The distance between the reference image and the generated image is calculated using structural similarity loss in terms of brightness, contrast, and structure, and is expressed as follows:

[0044]

[0045] Where μ r and σ r These are the mean and standard deviation of the reference image, μ g and σ g These are the mean and standard deviation of the generated images, σ rg c1 and c2 represent the covariance between the reference image and the generated image, and are constants.

[0046] Finally, combining the loss function described above, the final optimization function can be expressed as:

[0047]

[0048] Where, λ L1 and λ SSIM It is a weighting factor, l t This represents the final optimization loss.

[0049] An image enhancement device based on underwater attention generative adversarial networks includes:

[0050] Acquisition module: Used to acquire underwater images to be enhanced;

[0051] Module for building: Used to construct attention-based generative adversarial networks by extracting low-level and high-level semantic information through convolutional neural networks using supervised learning modes;

[0052] Training module: The underwater image to be enhanced is taken as input, and the attention generative adversarial network is trained using generative adversarial loss, loss and structural similarity loss. The trained attention generative adversarial network is obtained and the enhanced underwater image is output.

[0053] The image enhancement method based on underwater attention generative adversarial networks provided by this invention has the following advantages:

[0054] By employing a supervised learning model and leveraging convolutional neural networks to extract low-level and high-level semantic information, the designed UAGAN framework exhibits stronger generalization capabilities. Furthermore, the proposed UAGAN framework avoids the problems of parameter estimation (global atmospheric light value, absorption coefficient, scattering coefficient, transmission map, etc.) and underwater imaging modeling.

[0055] By leveraging global max pooling and average pooling operations, combined with a single hidden layer neural network architecture, a cascaded dense channel attention module is formed to achieve channel weight redistribution, which helps to suppress underwater noise features and improve the flexibility of low-level to high-level feature stacking.

[0056] Combining positional attention and PatchGAN techniques helps capture long-range dependency information, thus avoiding over-enhancement of the generated underwater images. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 The proposed UAGAN framework diagram;

[0059] Figure 2 Cascaded dense channel attention module diagram;

[0060] Figure 3 Location attention module diagram;

[0061] Figure 4 Visual comparison charts on the UIEB dataset;

[0062] Figure 5 Visualization comparison plots on the URPC dataset;

[0063] Figure 6 Performance comparison chart of data-driven methods on the UIEB dataset;

[0064] Figure 7 Loss function comparison chart;

[0065] Figure 8 Visualization comparison of the feature layers at the end of the discriminator. Detailed Implementation

[0066] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0069] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0070] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0071] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0072] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0073] An image enhancement method based on underwater attention-based generative adversarial networks includes the following steps:

[0074] S1: Obtain the underwater image to be enhanced;

[0075] S2: Through supervised learning, attention-based generative adversarial networks are constructed by extracting low-level and high-level semantic information using convolutional neural networks;

[0076] S3: Take the underwater image to be enhanced as input, and train the attention generative adversarial network using generative adversarial loss, loss and structural similarity loss. The trained attention generative adversarial network is obtained and the enhanced underwater image is output.

[0077] Steps S1 / S2 / S3 are executed sequentially;

[0078] The attention generative adversarial network utilizes global max pooling and average pooling operations, combined with a single hidden layer neural network architecture, to form a cascaded dense channel attention module, thereby achieving channel weight redistribution.

[0079] This invention achieves the sharpening of degraded images by utilizing high-quality images, which can be represented as:

[0080]

[0081] in The representative end-to-end underwater image enhancement method aims to enhance a given degraded underwater image. Based on the prediction of enhanced underwater images

[0082] In the UNet architecture, skip connections and stacking operations are represented as:

[0083]

[0084] in This represents the output of the i-th layer in the decoder. This represents the output of the i-th layer in the encoder. These are the convolutional weights of the i-th layer in the decoder, where R(·) represents the ReLU type activation function. This represents a splicing operation. This represents a convolution operation. It's important to note that skip connections can transfer low-level features from the encoder to the decoder.

[0085] In this invention, such as Figure 1 As shown, the proposed UAGAN framework is systematically composed of cascaded dense channel attention and position attention modules.

[0086] like Figure 2 As shown, the attention generative adversarial network captures long-range dependency information by combining positional attention and PatchGAN techniques.

[0087] The process of obtaining the output of the cascaded dense channel attention module is as follows:

[0088] The use of densely connected networks to extract discriminative noise features and enhance feature flow can be described as follows:

[0089]

[0090] in: This represents the output of the l-th basic module in the dense network. Represents the original input features. T represents the output of a dense network. l(·) is a non-linear transformation that includes batch normalization, LeakyReLU and convolution operations, and O(·) represents a 1×1 convolution operation;

[0091] By utilizing global average pooling and global max pooling operations to obtain statistical information for each feature map, it can be represented as:

[0092]

[0093]

[0094] Where h, w, and c represent the height, width, and number of layers of the channel feature map, respectively, and k = 1, 2, ..., c. This represents the feature map of the k-th channel. and These represent the outputs of global average pooling and global max pooling, respectively.

[0095] Note that without considering the interdependencies between channels, it's difficult to directly determine the contribution of each channel using only global average pooling and max pooling techniques. In this case, using two completely independent fully connected layers to calculate the channel weights can be expressed as:

[0096]

[0097] Where *∈{GAP,GMP}, and These are learnable fully connected weight parameters, where R(·) and S(·) are the ReLU and Sigmoid activation functions, respectively. Clearly, the channel weights... The number of hidden nodes is limited to (0,1); it should be noted that the number of hidden nodes in a fully connected layer is equal to the number of input and output nodes.

[0098] The output of the entire cascaded dense channel attention module can be expressed as:

[0099]

[0100] in, It is the output of the entire cascaded dense channel attention module.

[0101] Note that over-enhancement is highly likely to occur when multiple convolutional operations are stacked, mainly because the discriminator relies entirely on padding to perform the discrimination operation. Instead of being limited to adjacent positions, the positional attention used to compute the interaction between any two non-local spatial positions can be expressed as:

[0102]

[0103] Where i = 1, 2, ..., n and j = 1, 2, ..., n, and These respectively represent features derived from discriminative features. The reconstructed feature map, where n = hw is the total number of features. Represents the location attention map, β j,i Used to measure the positional weights between the i-th and j-th positions in the same feature map;

[0104] like Figure 3 As shown, positional attention can be represented as:

[0105]

[0106] in: This represents the predicted positional attention feature at the j-th position. The representation is the location attention feature map, γ is the weighting factor, and h i and a j These are reconstructed feature maps. and The i-th and j-th elements.

[0107] Furthermore, the optimization function of the attention generative adversarial network is obtained as follows:

[0108] To avoid gradient vanishing and mode collapse during training, the classic optimization function WGAN-GP is used, which is described as follows:

[0109]

[0110] in and These represent the probability distributions of real images and generated images, respectively. It is the penalty domain, λ gp G(·) and D(·) represent weighting factors, respectively, and the generator and discriminator are the generator and discriminator.

[0111] Secondly, the L1 loss is used to capture low-frequency information between the reference image and the generated image, and it can be expressed as:

[0112]

[0113] The distance between the reference image and the generated image is calculated using structural similarity loss in terms of brightness, contrast, and structure, and is expressed as follows:

[0114]

[0115] Where μ r and σ r These are the mean and standard deviation of the reference image, μg and σ g These are the mean and standard deviation of the generated images, σ rg c1 and c2 represent the covariance between the reference image and the generated image, and are constants.

[0116] Finally, combining the loss function described above, the final optimization function can be expressed as:

[0117]

[0118] Where, λ L1 and λ SSIM It is a weighting factor, l t This represents the final optimization loss.

[0119] An image enhancement device based on underwater attention generative adversarial networks includes:

[0120] Acquisition module: Used to acquire underwater images to be enhanced;

[0121] Module for building: Used to construct attention-based generative adversarial networks by extracting low-level and high-level semantic information through convolutional neural networks using supervised learning modes;

[0122] Training module: The underwater image to be enhanced is taken as input, and the attention generative adversarial network is trained using generative adversarial loss, loss and structural similarity loss. The trained attention generative adversarial network is obtained and the enhanced underwater image is output.

[0123] The training dataset used in this invention includes 6128 image pairs generated by unsupervised style transfer and 4000 image pairs simulated based on the Jerlov model. Hyperparameters and are both set to 10. Furthermore, the aforementioned datasets were scaled to 256 x 256 before training.

[0124] To effectively demonstrate the effectiveness and superiority of the proposed UAGAN framework, a comprehensive comparison is conducted between the proposed UAGAN method and model-based augmentation methods (including UCM and IBLA), the physics-based recovery framework (UDCP), and data-driven transformation methods (including UGAN and UWCNN) in various degraded underwater scenarios. The corresponding comparison results are as follows: Figure 4 As shown, from a subjective visual comparison perspective, we can clearly see that data-driven methods (UGAN, UWCNN, and UAGAN) achieve better results than model-free augmentation methods (UCM and IBLA) and model-based restoration methods (UDCP). The proposed UAGAN framework achieves better color restoration performance. Figure 5 and Figure 6Almost identical performance can be achieved. Specifically, the UCM framework struggles to achieve satisfactory enhancement performance. Furthermore, IBLA and UDCP tend to exacerbate degradation characteristics. Additionally, UGAN and UWCNN methods introduce significant artificial noise, greatly weakening visual perception. From the perspective of observing marine life, the proposed UAGAN framework achieves superior enhancement performance.

[0125] Table 1

[0126]

[0127]

[0128] To make the comparison results more statistically significant, Table 1 summarizes the mean and variance of the enhancement performance of different methods on the UIEB and URPC test sets, with the best and second-best values ​​marked in bold and underline, respectively. Note that the full-reference metrics SSIM and PSNR could not be obtained due to the lack of corresponding reference images in the URPC dataset. Specifically, in the evaluation of the UIEB dataset, particularly for PSNR, UCIQE, UIQM, and UISM metrics, the developed UAGAN framework achieves superior enhancement performance compared to other similar methods. This indicates that the enhanced underwater images have a higher peak signal-to-noise ratio, and the UAGAN method achieves a better balance in chroma, saturation, and sharpness. It is important to emphasize that the UAGAN framework achieves more competitive generalization performance with smaller variances in SSIM, UCIQE, UIQM, and UISM, clearly demonstrating that the UAGAN framework can maintain consistency in the enhancement results across the test set images. Regarding the augmentation performance of UAGAN on the URPC dataset, compared to model-free augmentation methods (UCM) and physically-based color restoration methods (including IBLA and UDCP), UGAN, UWCNN, and UAGAN achieve superior augmentation performance from a comprehensive perspective. It should be noted that by using cascaded dense channel attention and positional attention strategies, and based on the UIQM and UISM metrics, the UAGAN framework with generative adversarial training mechanisms can make the augmented underwater images more consistent with human visual perception. Furthermore, in addition to the UICM metric, the proposed UAGAN method achieves optimal or near-optimal variance.

[0129] To measure the correlation between objective evaluation metrics and average subjective evaluation scores, this invention conducted a correlation experiment on 30 randomly selected images. The correlation coefficients used primarily included KRCC, PLCC, and SRCC. The correlation results are summarized in Tables 2-5. From these, it is clear that the full-reference metrics SSIM and PSNR generally have larger correlation coefficients than the non-reference metrics (i.e., UCIQE and UIQM). Furthermore, regarding non-reference metrics, UIQM and UISM tend to obtain higher correlation coefficients, meaning that using UIQM and UISM for objective evaluation will yield results highly consistent with subjective visual quality.

[0130] Table 2

[0131] Metrics KRCC PLCC SRCC SSIM 0.4725 0.7013 ]]> ​ 0.5721 ]]> ​ PSNR 0.4101 0.6106 0.5121 UCIQE 0.2095 0.3184 0.2803 UIQMs 0.4376 0.6243 0.5028 UICM 0.3917 0.5455 0.5013 UISM 0.4549 ]]> ​ 0.7165 0.5913 UIConM 0.3390 0.4424 0.3788

[0132] Table 3

[0133] Metrics KRCC PLCC SRCC UCIQE 0.2950 0.4164 0.3819 UIQMs 0.6345 0.8074 0.7435 UICM 0.1484 0.2494 0.2381 UISM 0.5363 ]]> ​ 0.6829 ]]> ​ 0.6577 ]]> ​ UIConM 0.3085 0.6679 0.4096

[0134] Table 4

[0135] Metrics KRCC PLCC SRCC SSIM 0.4616 0.6682 ]]> ​ 0.5204 PSNR 0.4151 0.5171 0.4923 UCIQE 0.3505 0.4820 0.4235 UIQMs 0.4627 ]]> ​ 0.5356 0.5211 ]]> ​ UICM 0.0283 0.0056 0.0302 UISM 0.5849 0.6788 0.6411 UIConM 0.2061 0.3671 0.2905

[0136] Table 5

[0137]

[0138]

[0139] Comparison of single-image non-reference underwater image quality assessment

[0140] The non-reference underwater image quality assessment results for individual images in the URPC and UIEB datasets are shown in Tables 6 and 7. For individual images in the URPC dataset, data-driven techniques, including UGAN, UWCNN, and UAGAN, achieve superior enhancement performance compared to model-based restoration methods and model-free enhancement frameworks (i.e., UCM, IBLA, and UDCP). It is worth emphasizing that among the aforementioned data-driven methods, the proposed UAGAN method achieves the best performance in terms of underwater color, sharpness, and contrast, suggesting that the CDCA module can effectively suppress underwater environmental noise and enhance useful features. Similarly, almost identical results can be obtained from the UIEB dataset.

[0141] Table 6

[0142]

[0143] Table 7

[0144]

[0145]

[0146] To comprehensively evaluate the enhancement effect based on visual perception quality, this invention conducted a subjective performance evaluation experiment. Specifically, 11 participants were selected to subjectively evaluate the image quality; 6 participants had experience in image processing and computer vision, while the other 5 had no relevant experience. During the evaluation, zooming in and out was permitted without time limits. It was noted that images with clear visuals, natural colors, and realistic textures should be given higher scores. Conversely, images with low contrast, color cast, artifacts, and / or blur should be given lower scores. Furthermore, the highest score was the same as the number of methods compared, and the lowest score was set to 1.

[0147] Subjective evaluation results on the URPC and UIEB datasets are summarized in Tables 8 and 9, respectively. Table 8 shows that methods including UGAN, UWCNN, and UAGAN achieved higher scores due to the strong learning capabilities of data-driven frameworks. In Table 9, UGAN's visual performance (underlined) is weaker than the proposed UAGAN framework due to the absence of CDCA and PA modules. The lack of a generative adversarial mechanism results in a lower score for UWCNN. By using a cascaded dense network with channel attention and a positional attention strategy, the proposed UAGAN framework achieves optimal performance in most cases (bold).

[0148] Table 8

[0149] Methods Figure 5 (a) Figure 5 (b) Figure 5 (c) Figure 5 (d) Figure 5 (e) Figure 5 (f) UCM 3.4545 3.9091 4.4545 ]]> ​ 5.6364 3.7273 5.1818 ]]> ​ IBLA 2.8182 2.0909 1.7273 1.1818 1.4545 1.1818 UDCP 1.0909 1.1818 2.2727 2.0000 2.4545 1.9091 UGAN 4.9091 ]]> ​ 4.8182 ]]> ​ 2.5455 3.5455 4.0909 ]]> ​ 3.4545 UWCNN 2.8182 3.1818 4.3636 3.2727 3.4545 3.7273 UAGAN 5.9091 5.7273 5.6364 5.3636 ]]> ​ 5.8182 5.5455

[0150] Table 9

[0151] Methods Figure 6 (a) Figure 6 (b) Figure 6 (c) Figure 6 (d) Figure 6 (e) Figure 6 (f) Figure 6 (g) WaterNet 2.0909 1.8182 1.1818 1.4545 2.2727 ]]> ​ 1.5455 2.4545 ]]> ​ UGAN 2.3636 ]]> ​ 3.1818 3.0909 ]]> ​ 3.1818 ]]> ​ 1.7273 2.3636 ]]> ​ 1.6364 uWCNN 1.6364 2.2727 2.0909 1.9091 2.0909 2.1818 2.0909 UAGAN 3.9091 2.7273 ]]> ​ 3.6364 3.4545 3.7273 3.9091 3.8182

[0152] To reveal the role of each module introduced into the UAGAN framework, this invention conducted three ablation experiments on the UIEB dataset: without cascaded dense channel attention, without positional attention, and without both cascaded dense channel attention and positional attention. The corresponding evaluation results are presented in Table 10. From this, we can see that the CDCA module is closely related to the PSNR and UISM metrics. In other words, using the CDCA module can improve the sharpness of the enhanced image and reduce image noise. Furthermore, the CDCA module has an opposite effect to the UICM and UIConM metrics. It should be noted that the PA module only weakens the UCIQE metric. Combining the CDCA and PA modules helps in constructing the UAGAN framework. Ultimately, the developed UAGAN framework achieves optimal SSIM, PSNR, UCIQE, UIQMs, and UISM metrics.

[0153] Table 10

[0154]

[0155]

[0156] To fully reveal the contribution of the CDCA module, the sum and loss curves with and without CDCA (w / o) are presented. Figure 7 As we can clearly see, the curves with and without the CDCA module show very similar trends for either of the two loss types. The main reason is that no shuffling operation was used when acquiring batch training samples. In other words, the order of the paired samples used to train the UAGAN framework is completely consistent in each round. Keeping other training parameters consistent, only adding or removing the corresponding module (i.e., with or without CDCA) can affect the trend of the curve. In this case, the two curves show very similar trends. Clearly, with the CDCA module, the loss curve exhibits a more pronounced convergence trend.

[0157] To more intuitively demonstrate the function of the PA module, in Figure 8 The image presents the feature visualization results of the final layer of the discriminator, where columns (a)-(d) represent the enhanced image, the feature map of the enhanced image, the feature map of the reference image, and the difference between the enhanced image and the reference image, respectively. Figure 8 As can be seen, without the PA module, the local response values ​​are very pronounced. Conversely, when the PA module is able to capture long-range dependencies, the discriminator's output no longer exhibits local clustering. Furthermore, comparing the second and fourth columns reveals that the discriminator with the PA module is more focused on foreground objects and better determines the quality of the generated image.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image enhancement method based on underwater attention-based generative adversarial networks, characterized in that: Includes the following steps: Acquire underwater images to be enhanced; By using a supervised learning model, an attention-based generative adversarial network is constructed by extracting low-level and high-level semantic information through a convolutional neural network. The attention generative adversarial network utilizes global max pooling and average pooling operations, combined with a single hidden layer neural network architecture, to form a cascaded dense channel attention module, thereby achieving channel weight redistribution. The process of obtaining the output of the cascaded dense channel attention module is as follows: The use of densely connected networks to extract discriminative noise features and enhance feature flow can be described as follows: in: This represents the output of the l-th basic module in the dense network. Represents the original input features. T represents the output of a dense network. l (·) is a non-linear transformation that includes batch normalization, LeakyReLU and convolution operations, and O(·) represents a 1×1 convolution operation; By utilizing global average pooling and global max pooling operations to obtain statistical information for each feature map, it can be represented as: Where h, w, and c represent the height, width, and number of layers of the channel feature map, respectively, and k = 1, 2, ..., c. This represents the feature map of the k-th channel. and These represent the outputs of global average pooling and global max pooling, respectively. The channel weights are calculated using two completely independent fully connected layers, as follows: Where *∈{GAP,GMP}, and These are learnable fully connected weight parameters, where R(·) and S(·) are the ReLU and Sigmoid activation functions, respectively; clearly, the channel weights... It is restricted to the range (0,1); The output of the entire cascaded dense channel attention module can be expressed as: in, It is the output of the entire cascaded dense channel attention module; The attention generative adversarial network captures long-range dependency information by combining positional attention and PatchGAN techniques; The attention-based generative adversarial network calculates the interactivity between any two non-local spatial locations using positional attention, which can be represented as: Where i = 1, 2, ..., n and j = 1, 2, ..., n, and These respectively represent the features derived from discriminative features. The reconstructed feature map, where n = hw is the total number of features. Represents the location attention map, β j,i Used to measure the positional weights between the i-th and j-th positions in the same feature map; Positional attention can be represented as: in: This represents the predicted positional attention feature at the j-th position. The representation is the location attention feature map, γ is the weighting factor, and h i and a j These are reconstructed feature maps. and The i-th and j-th elements; The underwater image to be enhanced is used as input, and the attention-based generative adversarial network is trained using generative adversarial loss, loss, and structural similarity loss to obtain the trained attention-based generative adversarial network, which outputs the enhanced underwater image.

2. The image enhancement method based on underwater attention-based generative adversarial networks according to claim 1, characterized in that: The optimization function of the attention-based generative adversarial network is obtained as follows: To avoid gradient vanishing and mode collapse during training, the classic optimization function WGAN-GP is used, which is described as follows: in and These represent the probability distributions of real images and generated images, respectively. It is the penalty domain, λ gp G(·) and D(·) represent weighting factors, respectively, and the generator and discriminator are the generator and discriminator. Secondly, the L1 loss is used to capture low-frequency information between the reference image and the generated image, and it can be expressed as: The distance between the reference image and the generated image is calculated using structural similarity loss in terms of brightness, contrast, and structure, and is expressed as follows: Where μ r and σ r These are the mean and standard deviation of the reference image, μ g and σ g These are the mean and standard deviation of the generated images, σ rg c1 and c2 represent the covariance between the reference image and the generated image, and are constants. Finally, combining the loss function described above, the final optimization function can be expressed as: Where, λ L1 and λ SSIM It is a weighting factor, l t This represents the final optimization loss.

3. An image enhancement device based on underwater attention-based generative adversarial networks, characterized in that: include: Acquisition module: Used to acquire underwater images to be enhanced; Module for building: Used to construct attention-based generative adversarial networks by extracting low-level and high-level semantic information through convolutional neural networks using supervised learning modes; The attention generative adversarial network utilizes global max pooling and average pooling operations, combined with a single hidden layer neural network architecture, to form a cascaded dense channel attention module, thereby achieving channel weight redistribution. The process of obtaining the output of the cascaded dense channel attention module is as follows: The use of densely connected networks to extract discriminative noise features and enhance feature flow can be described as follows: in: This represents the output of the l-th basic module in the dense network. Represents the original input features. T represents the output of a dense network. l (·) is a non-linear transformation that includes batch normalization, LeakyReLU and convolution operations, and O(·) represents a 1×1 convolution operation; The statistical information of each feature map is obtained by using global average pooling and global max pooling operations, which is represented as follows: Where h, w, and c represent the height, width, and number of layers of the channel feature map, respectively, and k = 1, 2, ..., c. This represents the feature map of the k-th channel. and These represent the outputs of global average pooling and global max pooling, respectively. The channel weights are calculated using two completely independent fully connected layers, which can be expressed as: Where *∈{GAP,GMP}, and These are learnable fully connected weight parameters, where R(·) and S(·) are the ReLU and Sigmoid activation functions, respectively. Clearly, the channel weights... It is restricted to the range (0,1); The output of the entire cascaded dense channel attention module can be expressed as: in, It is the output of the entire cascaded dense channel attention module; The attention generative adversarial network captures long-range dependency information by combining positional attention and PatchGAN techniques; The attention-based generative adversarial network calculates the interactivity between any two non-local spatial locations using positional attention, which can be represented as: Where i = 1, 2, ..., n and j = 1, 2, ..., n, and These respectively represent the features derived from discriminative features. The reconstructed feature map, where n = hw is the total number of features. Represents the location attention map, β j,i Used to measure the positional weights between the i-th and j-th positions in the same feature map; Positional attention can be represented as: in: This represents the predicted positional attention feature at the j-th position. The representation is the location attention feature map, γ is the weighting factor, and h i and a j These are reconstructed feature maps. and The i-th and j-th elements; Training module: The underwater image to be enhanced is taken as input, and the attention generative adversarial network is trained using generative adversarial loss, loss and structural similarity loss. The trained attention generative adversarial network is obtained and the enhanced underwater image is output.

Images