Underwater image enhancement method and system based on generative adversarial network, and electronic equipment

Abstract

The invention discloses an underwater image enhancement method and system based on a generative adversarial network, and electronic equipment. The method comprises the following steps: acquiring an underwater image data set and a land image data set; determining a training set according to the underwater image data set and the land image data set; constructing a multi-scale parallel encoder comprising a plurality of branches, and constructing a multi-scale fusion decoder comprising a multi-scale feature dense fusion module; an underwater image enhancement model based on the generative adversarial network is constructed, the underwater image enhancement model comprises two generators and two discriminators, and each generator is constructed based on a multi-scale parallel encoder and a multi-scale fusion decoder; and training an underwater image enhancement model based on the generative adversarial network by using the training set to obtain a trained underwater image enhancement model, so as to enhance the underwater image to be enhanced by using the trained underwater image enhancement model. According to the invention, the image enhancement effect can be improved, and the image enhancement accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an underwater image enhancement method, system and electronic equipment based on a generative adversarial network. Background Art

[0002] The defects of traditional underwater image enhancement methods mainly come from the idealization and simplification of their assumptions. For example, physical models often rely on conditions such as water uniformity and stable light sources, while non-physical models (such as Retinex theory) have limited effects in dealing with color distortion and light scattering that are unique to underwater. In addition, these methods are very sensitive to parameter settings and it is difficult to achieve consistent results in different environments. Coupled with the complexity and variability of the underwater environment, it is difficult for traditional methods to comprehensively and accurately enhance the quality of underwater images.

[0003] The main drawbacks of deep learning-based generative adversarial networks (GANs) in underwater image enhancement are the lack of diversity and quality in the training dataset, which results in limited generalization of the model when faced with complex or changing underwater scenes that have never been seen before. In addition, the images generated by GANs often strongly depend on the distribution of the training data, which may ignore small details or cause certain visual distortions, and the model is prone to mode collapse or instability during training, which limits its consistent performance in different underwater environments.

[0004] It can be seen from this that the image enhancement effect of the existing underwater image enhancement method is relatively poor, and the accuracy of the enhanced image is not high. Summary of the invention

[0005] The present application aims to propose an underwater image enhancement method, system and electronic equipment based on a generative adversarial network, which can improve the image enhancement effect and improve the accuracy of image enhancement.

[0006] In a first aspect, an embodiment of the present application provides an underwater image enhancement method based on a generative adversarial network, the method comprising:

[0007] Obtain underwater image datasets and land image datasets;

[0008] Determine a training set according to the underwater image dataset and the land image dataset;

[0009] Construct a multi-scale parallel encoder containing multiple branches, and construct a multi-scale fusion decoder containing a multi-scale feature dense fusion module;

[0010] Constructing an underwater image enhancement model based on a generative adversarial network comprising two generators and two discriminators, wherein each of the generators is constructed based on the multi-scale parallel encoder and the multi-scale fusion decoder;

[0011] The training set is used to train the underwater image enhancement model based on the generative adversarial network to obtain a trained underwater image enhancement model, so that the trained underwater image enhancement model can be used to perform image enhancement on the underwater image to be enhanced.

[0012] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0013] The method constructs a multiscale parallel encoder including multiple branches, and a multiscale fusion decoder including a multiscale feature dense fusion module; constructs an underwater image enhancement model based on a generative adversarial network including two generators and two discriminators, wherein each generator is constructed based on a multiscale parallel encoder and a multiscale fusion decoder; uses a training set to train the underwater image enhancement model based on the generative adversarial network to obtain a trained underwater image enhancement model, so as to use the trained underwater image enhancement model to perform image enhancement on the underwater image to be enhanced. In this way, encoding using a multiscale parallel encoder including multiple branches can continue to maintain high-resolution feature maps and reduce information loss, and then decoding using a multiscale fusion decoder including a multiscale feature dense fusion module can avoid sudden changes in feature map size and dimension, and prevent the loss of important information in the feature map, so that the trained underwater image enhancement model can improve the image enhancement effect and improve the accuracy of image enhancement.

[0014] In some embodiments, when the trained underwater image enhancement model is used to perform image enhancement on the underwater image to be enhanced, the method further includes:

[0015] Acquiring an underwater image to be enhanced;

[0016] Using the multi-scale parallel encoder to encode the underwater image to be enhanced, to obtain encoding results with multiple resolutions, wherein the resolutions of the encoding results with multiple resolutions are different;

[0017] The encoding results of the multiple resolutions are input into the multi-scale fusion decoder for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder.

[0018] In some implementations, the multi-scale parallel encoder includes a first convolutional layer filled with 1 and multiple downsampling modules, and the multi-scale parallel encoder is used to encode the underwater image to be enhanced to obtain encoding results of multiple resolutions, including:

[0019] Inputting the underwater image to be enhanced into the first convolution layer filled with 1 to obtain a convolution result;

[0020] The convolution result is encoded using multiple downsampling modules to obtain encoding results of multiple resolutions.

[0021] In some implementations, the using of multiple downsampling modules to encode the convolution result to obtain encoding results of multiple resolutions includes:

[0022] Divide the multiple downsampling modules into three parallel-connected branches, the three parallel-connected branches are respectively a first branch, a second branch and a third branch, each branch comprises a plurality of downsampling modules, the downsampling modules comprise a plurality of second convolutional layers filled with 1 and an unfilled third convolutional layer, or the downsampling modules comprise a plurality of second convolutional layers filled with 1;

[0023] Inputting the convolution result into a plurality of second convolution layers filled with 1 of the first branch to obtain a first output result;

[0024] Inputting the first output result into the unfilled third convolutional layer of the first branch to obtain a second output result;

[0025] Inputting the first output result into a plurality of second convolutional layers filled with 1 of the first branch to obtain an encoding result of a first resolution output by the first branch;

[0026] Inputting the second output result into a plurality of second convolutional layers filled with 1 of the second branch to obtain a third output result;

[0027] Inputting the third output result into the unfilled third convolutional layer of the second branch to obtain a fourth output result;

[0028] Inputting the third output result into a plurality of second convolutional layers filled with 1 of the second branch to obtain an encoding result of a second resolution output by the second branch;

[0029] The fourth output result is input into a plurality of second convolutional layers filled with 1 of the third branch to obtain an encoding result of a third resolution output by the third branch.

[0030] In some implementations, inputting the encoding results of the multiple resolutions into the multi-scale fusion decoder for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder includes:

[0031] Performing the same convolution operation on the encoding result of each resolution of the encoding results of the multiple resolutions to obtain convolution output results of the multiple resolutions;

[0032] The convolution output results of the multiple resolutions are input into the multi-scale feature dense fusion module for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder.

[0033] In some embodiments, the multi-scale feature dense fusion module includes upsampling, and the convolution output results of the multiple resolutions are input into the multi-scale feature dense fusion module for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder, including:

[0034] Upsampling a convolution output result of a first resolution among the convolution output results of the multiple resolutions to obtain a first upsampling result, wherein the convolution output result of the first resolution is a convolution output result of the lowest resolution;

[0035] Performing pixel-by-pixel addition and fusion of the first upsampling result and the convolution output result of the second resolution to obtain a first fused feature map, wherein the resolution of the convolution output result of the second resolution is higher than the resolution of the convolution output result of the first resolution;

[0036] If there is a convolution output result with a third resolution higher than the resolution of the convolution output result with the second resolution, upsampling the first fused feature map to obtain a second upsampling result;

[0037] Adding features of the second upsampling result and the first upsampling result to obtain a feature addition result;

[0038] The feature addition result and the convolution output result of the third resolution are added and fused pixel by pixel to obtain a second fused feature map, and so on, until the convolution output result with the highest resolution among the convolution output results of the multiple resolutions is added and fused pixel by pixel with up-sampling results corresponding to all the convolution output results with low resolutions to obtain a target enhanced feature map;

[0039] A convolution operation is performed on the target enhanced feature map to obtain an enhanced image output by the multi-scale fusion decoder.

[0040] In some implementations, the convolution output result with the highest resolution among the convolution output results with the multiple resolutions is added and fused pixel by pixel with up-sampled results corresponding to all the convolution output results with the low resolutions, including:

[0041] α j =F i-1

[0042]

[0043] F i=Φ(α i ,β i )

[0044] Among them, α j and F i-1 Represents the fused feature map corresponding to the convolution output result of the i-1th resolution, F i represents the fused feature map corresponding to the convolution output result of the i-th resolution, β i represents the upsampling result corresponding to the convolution output result of the i-th resolution, U represents upsampling, Φ represents pixel-by-pixel addition, and α i Represents the convolution output result of the i-th resolution.

[0045] In a second aspect, an embodiment of the present application further provides an underwater image enhancement system based on a generative adversarial network, the system comprising:

[0046] A data acquisition unit, used to acquire an underwater image data set and a land image data set;

[0047] A training set determination unit, used to determine a training set according to the underwater image data set and the land image data set;

[0048] A first construction unit is used to construct a multi-scale parallel encoder including multiple branches, and to construct a multi-scale fusion decoder including a multi-scale feature dense fusion module;

[0049] A second construction unit is used to construct an underwater image enhancement model based on a generative adversarial network, which includes two generators and two discriminators, wherein each of the generators is constructed based on the multi-scale parallel encoder and the multi-scale fusion decoder;

[0050] The image enhancement unit is used to use the training set to train the underwater image enhancement model based on the generative adversarial network to obtain a trained underwater image enhancement model, so as to use the trained underwater image enhancement model to perform image enhancement on the underwater image to be enhanced.

[0051] In a third aspect, an embodiment of the present application also provides an electronic device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute an underwater image enhancement method based on a generative adversarial network as described above.

[0052] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the underwater image enhancement method based on a generative adversarial network as described above.

[0053] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0055] Figure 1 It is a flowchart of an embodiment of an underwater image enhancement method based on a generative adversarial network provided by the present application;

[0056] Figure 2 It is a schematic diagram of the basic principle of CycleGAN in the best embodiment of the underwater image enhancement method based on generative adversarial network provided by the present application;

[0057] Figure 3 : is a schematic diagram comparing the CycleGAN structure and the MFD-CycleGAN structure in the best embodiment of the underwater image enhancement method based on the generative adversarial network provided by the present application;

[0058] Figure 4 It is a structural schematic diagram of the MSFDF module in the best embodiment of the underwater image enhancement method based on the generative adversarial network provided by the present application;

[0059] Figure 5 It is a schematic diagram of an experimental data set sample in the best embodiment of the underwater image enhancement method based on the generative adversarial network provided by the present application;

[0060] Figure 6 : is a schematic diagram comparing underwater image enhancement effects of different models in the best embodiment of the underwater image enhancement method based on a generative adversarial network provided by the present application;

[0061] Figure 7 It is a schematic diagram comparing the output results of different models and the Canny edge detection results in the best embodiment of the underwater image enhancement method based on the generative adversarial network provided by the present application;

[0062] Figure 8 is a schematic structural diagram of an embodiment of an underwater image enhancement system based on a generative adversarial network provided by the present application;

[0063] Fig. 9 It is a schematic diagram of the structure of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0065] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0066] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0067] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0068] The defects of traditional underwater image enhancement methods mainly come from the idealization and simplification of their assumptions. For example, physical models often rely on conditions such as water uniformity and stable light sources, while non-physical models (such as Retinex theory) have limited effects in dealing with color distortion and light scattering that are unique to underwater. In addition, these methods are very sensitive to parameter settings and it is difficult to achieve consistent results in different environments. Coupled with the complexity and variability of the underwater environment, it is difficult for traditional methods to comprehensively and accurately enhance the quality of underwater images.

[0069] The main drawbacks of deep learning-based GAN networks in underwater image enhancement are the lack of diversity and quality in the training dataset, which results in limited generalization of the model when faced with complex or variable underwater scenes that have never been seen before. In addition, the images generated by GANs often strongly depend on the distribution of the training data, which may ignore small details or cause certain visual distortions, and the model is prone to mode collapse or instability during training, which limits its consistent performance in different underwater environments.

[0070] It can be seen from this that the image enhancement effect of the existing underwater image enhancement method is relatively poor, and the accuracy of the enhanced image is not high.

[0071] In order to solve the problem that the image enhancement effect of the above-mentioned existing underwater image enhancement methods is relatively poor and the accuracy of the enhanced images is not high, the present application proposes an underwater image enhancement method, system and electronic device based on a generative adversarial network.

[0072] Reference Figure 1 The embodiment of the present application provides an underwater image enhancement method based on a generative adversarial network, the method comprising the following steps:

[0073] Step S100, acquiring an underwater image dataset and a land image dataset;

[0074] Step S200, determining a training set according to the underwater image data set and the land image data set;

[0075] Step S300, constructing a multi-scale parallel encoder including multiple branches, and constructing a multi-scale fusion decoder including a multi-scale feature dense fusion module;

[0076] Step S400, constructing an underwater image enhancement model based on a generative adversarial network including two generators and two discriminators, wherein each generator is constructed based on a multi-scale parallel encoder and a multi-scale fusion decoder;

[0077] Step S500: Use the training set to train the underwater image enhancement model based on the generative adversarial network to obtain a trained underwater image enhancement model, so as to use the trained underwater image enhancement model to perform image enhancement on the underwater image to be enhanced.

[0078] In this embodiment, a multiscale parallel encoder including multiple branches is constructed, and a multiscale fusion decoder including a multiscale feature dense fusion module is constructed; an underwater image enhancement model based on a generative adversarial network is constructed, which includes two generators and two discriminators, wherein each generator is constructed based on a multiscale parallel encoder and a multiscale fusion decoder; a training set is used to train the underwater image enhancement model based on the generative adversarial network to obtain a trained underwater image enhancement model, so that the trained underwater image enhancement model is used to enhance the underwater image to be enhanced. In this way, encoding using a multiscale parallel encoder including multiple branches can continue to maintain high-resolution feature maps and reduce information loss, and decoding using a multiscale fusion decoder including a multiscale feature dense fusion module can avoid sudden changes in feature map size and dimension, and prevent the loss of important information in the feature map, so that the trained underwater image enhancement model can improve the image enhancement effect and improve the accuracy of image enhancement.

[0079] The training set is determined based on the underwater image dataset and the land image dataset, and the training set may be constructed by using the underwater image dataset and the land image dataset containing the same number of images.

[0080] The above-mentioned underwater image enhancement model based on a generative adversarial network, which includes two generators and two discriminators, can be an underwater image enhancement model based on a generative adversarial network constructed based on the CycleGAN model. The loss function in the underwater image enhancement model based on a generative adversarial network in this embodiment is the same as the loss function in the CycleGAN model. The two generators and two discriminators in the underwater image enhancement model based on a generative adversarial network are the same as the CycleGAN model, except that the structure of the generator is different from the generator structure in the CycleGAN model.

[0081] The above-mentioned use of a training set to train the underwater image enhancement model based on a generative adversarial network to obtain a trained underwater image enhancement model can be to input the training set into the underwater image enhancement model based on a generative adversarial network, so that the underwater image enhancement model based on the generative adversarial network performs adversarial learning until the loss function converges or reaches a preset convergence condition, thereby obtaining a trained underwater image enhancement model.

[0082] In some embodiments, when using the trained underwater image enhancement model to perform image enhancement on the underwater image to be enhanced, the method further includes:

[0083] Acquiring an underwater image to be enhanced;

[0084] A multi-scale parallel encoder is used to encode the underwater image to be enhanced, and encoding results of multiple resolutions are obtained, and the resolutions of the encoding results of the multiple resolutions are different;

[0085] The encoding results of multiple resolutions are input into the multi-scale fusion decoder for image enhancement, and the enhanced image output by the multi-scale fusion decoder is obtained.

[0086] In this embodiment, a multi-scale parallel encoder is used to encode the underwater image to be enhanced to obtain multiple encoding results with different resolutions, and then the multiple encoding results with different resolutions are used for image enhancement using a multi-scale fusion decoder. This can better maintain the resolution of the feature map and reduce information loss, thereby improving the image enhancement effect.

[0087] In some embodiments, the multi-scale parallel encoder includes a first convolution layer filled with 1 and multiple downsampling modules, and the multi-scale parallel encoder is used to encode the underwater image to be enhanced to obtain encoding results of multiple resolutions, including:

[0088] The underwater image to be enhanced is input into the first convolution layer filled with 1 to obtain the convolution result;

[0089] Multiple downsampling modules are used to encode the convolution results to obtain encoding results with multiple resolutions.

[0090] In this embodiment, multiple downsampling modules are used to encode the convolution results to obtain encoding results of multiple resolutions. By encoding the convolution results through multiple downsampling modules, the resolution of the feature map can be better maintained, information loss can be reduced, and thus the image enhancement effect can be improved.

[0091] In some implementations, multiple downsampling modules are used to encode the convolution result to obtain encoding results of multiple resolutions, including:

[0092] Divide the multiple downsampling modules into three parallel-connected branches, the three parallel-connected branches are respectively a first branch, a second branch, and a third branch, each branch comprises a plurality of downsampling modules, the downsampling modules comprise a plurality of second convolutional layers filled with 1 and an unfilled third convolutional layer, or the downsampling modules comprise a plurality of second convolutional layers filled with 1;

[0093] Input the convolution result into a plurality of second convolution layers filled with 1 of the first branch to obtain a first output result;

[0094] Input the first output result into the unfilled third convolutional layer of the first branch to obtain a second output result;

[0095] Input the first output result into a plurality of second convolutional layers filled with 1 of the first branch to obtain an encoding result of the first resolution output by the first branch;

[0096] Input the second output result into a plurality of second convolutional layers filled with 1 of the second branch to obtain a third output result;

[0097] Inputting the third output result into the unfilled third convolutional layer of the second branch to obtain a fourth output result;

[0098] Input the third output result into a plurality of second convolutional layers filled with 1 of the second branch to obtain an encoding result of a second resolution output by the second branch;

[0099] The fourth output result is input into multiple second convolutional layers filled with 1 of the third branch to obtain an encoding result of a third resolution output by the third branch.

[0100] In this embodiment, the use of multiple second convolutional layers with padding of 1 can maintain the resolution of the feature map and will not reduce the size of the feature map. The use of an unfilled third convolutional layer will reduce the size of the feature map and reduce the resolution of the feature map. Therefore, the output of the unfilled third convolutional layer is used as the input of the next branch, and the output of the current branch uses the second convolutional layer with padding of 1, which can not only better extract the feature information, but also maintain the resolution of the feature map, and multiple branches are used to obtain feature maps of different resolutions. The multi-branch structure is used to construct an underwater image enhancement model, which retains more image details and texture information, and can better learn the information in feature maps of different resolutions, thereby improving the image enhancement effect and improving the accuracy of image enhancement.

[0101] The above-mentioned multiple second convolutional layers with padding of 1 can have the same convolution kernel size as the multiple first convolutional layers with padding of 1, that is, they all use 3×3 convolutions with padding of 1, and each 3×3 convolution is followed by a layer of BN (BatchNormalization) and a layer of ReLU activation function.

[0102] The unfilled third convolutional layer may be an unfilled 3×3 convolutional layer.

[0103] In some implementations, inputting the encoding results of multiple resolutions into a multi-scale fusion decoder for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder includes:

[0104] Performing the same convolution operation on the encoding result of each resolution in the encoding results of multiple resolutions to obtain convolution output results of multiple resolutions;

[0105] The convolution output results of multiple resolutions are input into the multi-scale feature dense fusion module for image enhancement, and the enhanced image output by the multi-scale fusion decoder is obtained.

[0106] In this embodiment, the same convolution operation is performed on the encoding results of each resolution of the encoding results of multiple resolutions, so that the convolution output results of the obtained multiple resolutions have the same number of channels, and then the convolution output results of the multiple resolutions are input into the multi-scale feature intensive fusion module for image enhancement. By fusing the convolution output results of different resolutions (i.e., the feature maps of low resolution), the multi-scale feature intensive fusion module can better handle the differences and noise between images of different resolutions, thereby generating an output image with clearer details, thereby improving the image enhancement effect and the accuracy of the enhanced image.

[0107] In some embodiments, the multi-scale feature dense fusion module includes upsampling, and the convolution output results of multiple resolutions are input into the multi-scale feature dense fusion module for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder, including:

[0108] Upsampling a convolution output result of a first resolution among the convolution output results of multiple resolutions to obtain a first upsampling result, wherein the convolution output result of the first resolution is a convolution output result of the lowest resolution;

[0109] The first upsampling result and the convolution output result of the second resolution are added and fused pixel by pixel to obtain a first fused feature map, wherein the resolution of the convolution output result of the second resolution is higher than the resolution of the convolution output result of the first resolution;

[0110] If there is a convolution output result with a third resolution higher than the convolution output result with the second resolution, upsampling the first fused feature map to obtain a second upsampling result;

[0111] Adding features of the second upsampling result and the first upsampling result to obtain a feature addition result;

[0112] The feature addition result and the convolution output result of the third resolution are added and fused pixel by pixel to obtain a second fused feature map, and so on, until the convolution output result of the highest resolution among the convolution output results of multiple resolutions is added and fused with the up-sampling results corresponding to all the convolution output results of the low resolutions pixel by pixel to obtain a target enhanced feature map;

[0113] The target enhanced feature map is convolved to obtain the enhanced image output by the multi-scale fusion decoder.

[0114] In this embodiment, by upsampling the low-resolution convolution output result (i.e., the low-resolution feature map) and then performing matrix addition fusion or pixel-by-pixel addition with the high-resolution convolution output result, the information in the original image can be better expressed, thereby achieving clearer underwater image generation and improving the image enhancement effect and the accuracy of the enhanced image.

[0115] In some implementations, the convolution output result with the highest resolution among the convolution output results with multiple resolutions is pixel-by-pixel added and fused with up-sampled results corresponding to all the convolution output results with low resolutions, including:

[0116] α j =F i-1

[0117]

[0118] F i=Φ(α i ,β i )

[0119] Among them, α j and F i-1 Represents the fused feature map corresponding to the convolution output result of the i-1th resolution, F i represents the fused feature map corresponding to the convolution output result of the i-th resolution, β i represents the upsampling result corresponding to the convolution output result of the i-th resolution, U represents upsampling, Φ represents pixel-by-pixel addition, and α i Represents the convolution output result of the i-th resolution.

[0120] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0121] The existing underwater image enhancement methods have the following disadvantages:

[0122] 1. Traditional underwater image enhancement methods usually perform poorly in dealing with color distortion, uneven lighting, environmental changes, and complex scenes, resulting in unstable and unnatural enhancement effects.

[0123] Reason: The defects of traditional underwater image enhancement methods mainly stem from the idealization and simplification of their assumptions. For example, physical models often rely on conditions such as water uniformity and stable light sources, while non-physical models (such as Retinex theory) have limited effects in dealing with color distortion and light scattering unique to underwater. In addition, these methods are very sensitive to parameter settings and it is difficult to achieve consistent results in different environments. Coupled with the complexity and variability of the underwater environment, it is difficult for traditional methods to comprehensively and accurately enhance the quality of underwater images.

[0124] 2. The underwater image enhancement method based on deep learning GAN network may have shortcomings in terms of insufficient training data, limited model generalization ability, and the generated results are too dependent on the distribution of the training set, resulting in detail distortion and unnaturalness.

[0125] Reason: The defects of deep learning-based GAN networks in underwater image enhancement mainly stem from the insufficient diversity and quality of the training data set, which leads to limited generalization ability of the model when facing complex or changeable underwater scenes that have not been seen before; in addition, the images generated by GAN often strongly depend on the distribution of training data, which may ignore small details or cause certain visual distortions, and the model is prone to mode collapse or instability during training, which limits its consistent performance in different underwater environments.

[0126] In order to solve the problems existing in the above-mentioned existing underwater image enhancement methods, this embodiment proposes a method based on multi-scale feature fusion (MFD-CycleGAN, that is, the underwater image enhancement model based on the generative adversarial network of this embodiment) to improve the clarity and contrast of underwater low-resolution images and restore their color characteristics. This is to solve the problem of blurred underwater images, low visibility and serious color deviation caused by the attenuation of light propagation caused by water. This embodiment realizes underwater image enhancement by constructing a cross-domain mapping relationship between underwater images and land images. This method uses a generative network and a discriminant network, does not require paired images for network training, can overcome the difficulty of collecting paired images, and use a wider range of underwater image data for training to adapt to various complex underwater scenes. The technical solution of this embodiment specifically includes the following contents:

[0127] (1) Introduction to the CycleGAN model.

[0128] The CycleGAN model is a deep learning model that achieves the mutual mapping between two images X and Y through adversarial learning. The principle diagram of the CycleGAN model is as follows Figure 2 As shown in the figure. The model is actually composed of two mirror-symmetric GANs, which form a complete cycle. The CycleGAN model contains two pairs of generative networks and discriminative networks, and can achieve mutual conversion between the two domains through adversarial learning. In this process, the cycle consistency loss function (Cycleloss) is introduced to ensure the consistency of the cycle. The CycleGAN model contains a total of 4 loss functions, including two generative loss functions and two discriminative loss functions.

[0129] exist Figure 2 In the example, the input image X can be mapped to the image G(X) in the output image Y through the encoding and decoding structure of the generator G, and the corresponding discriminator is Dx. Since CycleGAN is based on the principle of symmetry, the image Y in the output image Y also needs to be mapped back to the F(Y) in the input image X through a generator F, and the corresponding discriminator is Dy. If after the mapping of G and F, G(F(x))≈X and F(G(y))≈Y, then the conversion process of the CycleGAN model is completed.

[0130] The training process of CycleGAN is slightly different from that of traditional GAN. It requires optimizing the loss functions of two generators and two discriminators at the same time. Among them, the cyclic loss function is an important part of CycleGAN, which can help the model achieve cycle consistency and thus improve the performance of the model.

[0131] (2) Construct the MFD-CycleGAN of this embodiment.

[0132] The MFD-CycleGAN of this embodiment is built based on CycleGAN. For the residual network used in the CycleGAN generator, its main function is to increase the depth of the network, so that the network can more accurately learn the mapping relationship in the image conversion task, thereby improving the performance of the generator. The residual network can effectively solve the problems of gradient disappearance and gradient explosion through cross-layer connection, making the network easier to train. In CycleGAN, the network structure of the generator is relatively shallow, with only 9 layers of residual modules, which is to avoid the problems of overfitting and unstable training. Although the advantages of deep networks in fields such as image recognition and target detection need to be shown in deeper network structures, in image conversion tasks, since the mapping relationship between input and output is very complex, even a relatively shallow network structure can achieve good performance. In addition, the CycleGAN generator adopts a full convolution connection method, which enables the network to process input images of any size, thereby increasing the flexibility and scope of application of the network. This full convolution connection method can also avoid the problems of excessive parameters and excessive calculations caused by the use of traditional fully connected layers, making the network easier to train and optimize.

[0133] However, the images generated by CycleGAN in complex scenes are not clear enough. First, the encoding structure of the CycleGAN generator is too simple. It only reduces the image resolution through convolution and pooling, which degrades the high-resolution image and causes permanent loss of some image features, resulting in blurred generated images. Second, the decoding part of the CycleGAN generator directly performs deconvolution operations on the low-resolution feature map to restore the image size. The sudden increase in the feature map size will also cause image information loss, resulting in checkerboard artifacts and abnormal ripples in the generated image. Such images are contrary to the visual habits of the human eye. By studying the traditional CycleGAN network structure, this embodiment improves the encoding and decoding parts, combines the multi-resolution parallel network and the multi-scale feature dense fusion module, and constructs a multi-scale feature fusion underwater image enhancement network MFD-CycleGAN.

[0134] (3) Construct a multi-scale parallel encoder in the MFD-CycleGAN of this embodiment.

[0135] The CycleGAN generator encoding uses a multi-layer downsampling structure to extract image features. However, each convolution and pooling operation will cause some information to be permanently lost, which in turn affects the clarity and contrast of the generated image. To solve the information loss problem, this embodiment improves the encoding structure of the CycleGAN generator and constructs a multi-resolution sub-network parallel structure that retains high-resolution images, such as Figure 3 shown.

[0136] The encoding structure of the CycleGAN generator first receives the input image with a 3×3 convolution layer filled with 1, and then uses three downsampling modules to extract features from the image. Each downsampling module has three layers of convolution, the first two layers are 3×3 convolution layers filled with 1, and each convolution is followed by a layer of BN (Batch Normalization) and a layer of ReLU activation function. The last layer uses an unfilled 3×3 convolution layer to reduce the size of the feature map to 1 / 2 of the original. However, the encoding structure of the CycleGAN generator will permanently lose some information when downsampling.

[0137] Each time the feature map is reduced in size, MFD-CycleGAN retains the feature map of the original resolution, thereby generating a parallel network (i.e., a multi-scale parallel encoder). The high-resolution feature map continues to perform 3×3 convolutions with a padding of 1 to maintain high resolution, and the low-resolution feature map is obtained by performing unfilled 3×3 convolutions on the high-resolution feature map.

[0138] Specifically, the multi-scale parallel encoder of MFD-CycleGAN includes three parallel branches of different resolutions, each branch contains several downsampling modules, and one downsampling module in each branch contains three layers of convolution. The first two layers of the first downsampling module in the high-resolution branch are 3×3 convolution layers with padding of 1, and each convolution is followed by a layer of BN (Batch Normalization) and a layer of ReLU activation function. The last layer uses an unfilled 3×3 convolution layer to reduce the size of the feature map to 1 / 2 of the original size. Then, the three layers of convolution of other downsampling modules corresponding to each branch use 3×3 convolution layers with padding of 1 to maintain the resolution of the current branch. That is, the high-resolution branch first performs two 3×3 convolution layers with padding of 1, and then performs an unfilled 3×3 convolution layer to reduce the feature map size to 1 / 2 of the original size. The output feature map of the unfilled 3×3 convolution layer is used as the input of the low-resolution branch. Then, the output feature maps of the two 3×3 convolution layers with padding of 1 are input into other downsampling modules of the current high-resolution branch (that is, the three convolution layers of other downsampling modules all use 3×3 convolution layers with padding of 1), and multiple 3×3 convolution layers with padding of 1 are continued to be convolved until the result of the high-resolution branch is output, so that the high-resolution branch continues to maintain high resolution. In the downsampling module of the lowest resolution branch, the three convolution layers are all 3×3 convolution layers with padding of 1. Because the lowest resolution branch is already the last branch and there is no branch with lower resolution, there is no need to use unfilled 3×3 convolution layers, and only the resolution of the lowest resolution branch needs to be maintained.

[0139] Finally, by fusing the multi-resolution feature map (i.e., including the high-resolution feature map and the low-resolution feature map) in the MSFDF module, a clearer underwater image can be generated, so that the information in the original image can be better expressed.

[0140] (4) Construct the Multi-Scale Feature Dense Fusion Module (MSFDF module) in the MFD-CycleGAN of this embodiment.

[0141] Regarding the decoding structure of the generator in the MFD-CycleGAN of this embodiment, the multi-scale fusion decoder of this embodiment including the multi-scale feature intensive fusion module is different from the decoding structure of the original CycleGAN model. The decoding structure of the original CycleGAN model obtains the output image through two layers of deconvolution, while this embodiment adds a multi-scale feature intensive fusion module to the upsampling part. The module structure is as follows: Figure 4 As shown in the figure, conv means convolution, upsmaple means upsampling, and Feature map means feature map. The encoder inputs each resolution in the MSFDF module to adjust the number of channels through a 1×1 convolution layer, and then inputs the MSFDF module. The low-resolution feature map is restored to the same size as the high-resolution feature map through multiple deconvolution operations, and then matrix-added and fused with the high-resolution feature map to obtain a new high-resolution feature map. After the feature maps at all levels are fused, the output feature map is deconvolved and reshaped to obtain the enhanced feature map (i.e., the fused feature map, including Figure 4 F2 and F3 in the above example), and finally convolution operation is performed on F3 to obtain the final enhanced image. This also realizes the multi-level skip connection of upsampling. Compared with the traditional CycleGAN, the process of increasing the feature map size in the MFD-CycleGAN of this embodiment is smoother. By fusing the high-resolution feature map with the low-resolution feature map, the detail information of the original image is retained and the important features of the image are enhanced.

[0142] The expressions of the input formulas of feature maps at all levels in the MSFDF module are as follows:

[0143] α i =C(f i )

[0144] Among them, f i represents the i-th level feature map of the encoder input MSFDF module. The C function refers to the 1×1 convolution operation on the input feature map to achieve the same number of channels. i is the convolution output feature image (i.e., the convolution output result of the i-th resolution), Figure 4α1 in is obtained by inputting the 1st level feature map of the MSFDF module through the encoder, α2 is obtained by inputting the 2nd level feature map of the MSFDF module through the encoder, and α3 is obtained by inputting the 3rd level feature map of the MSFDF module through the encoder.

[0145] α j =F i-1

[0146]

[0147] The above formula is used to fuse feature maps at all levels. The U function represents the upsampling operation, which only fuses low-level features. That is, the above operation process uses multiple deconvolution to enlarge the low-level feature map after the number of channels is unified, and enlarge it to the same size as the high-level feature map, and then perform feature addition and fusion. The value range of j is 1 to i-1. By adding and fusing the feature maps, β is obtained. i (i.e. the result of adding the features).

[0148] F i =Φ(α i ,β i )

[0149] Among them, Φ represents the pixel addition operation corresponding to the feature map, and the convolution output feature map α i Fusion output feature map β with low level i Perform Φ operation to obtain the output feature map F i (i.e. fused feature map).

[0150] The MFD-CycleGAN model has three levels of output. The MSFDF module is used to combine the three levels of feature maps, avoiding sudden changes in the size and dimension of the feature maps and preventing the loss of important information in the feature maps. The role of the MSFDF module has been proven in experiments to enhance the details of the network output image.

[0151] Compared with the prior art, this embodiment has the following advantages:

[0152] 1. The MFD-CycleGAN model of this embodiment retains feature maps of different resolutions through a multi-scale parallel encoder, which helps to retain more image details and texture information during the generation process. By adding a multi-scale feature dense fusion module (MSFDF module) in the decoding part of the generator, the model can better handle the differences and noise between images of different resolutions, thereby generating output images with clearer details. By comparing with other traditional and deep learning-based underwater image enhancement models, the experimental results show that MFD-CycleGAN performs better under multiple evaluation indicators, and the generated images have less noise and clearer details.

[0153] 2. Through the generation network and discriminator network in the MFD-CycleGAN model, a cross-domain mapping relationship between underwater images and land images is constructed, and the cyclic consistency loss function (Cyclic loss) is introduced to ensure the consistency of the cycle, so that the model can learn more accurate inter-domain mapping. Through experiments, the paper proves that the MFD-CycleGAN model can learn the mapping relationship between underwater images and land images without paired training data, and realize underwater image enhancement.

[0154] For better explanation, the following experiments were performed in this embodiment:

[0155] (1) Experimental environment of this embodiment.

[0156] The specific hardware and software configuration of the training and testing platform of the entire model in this embodiment is shown in Table 1 below.

[0157] Table 1 Experimental configuration information

[0158]

[0159] In this embodiment, the data set is divided into two parts, one part is the low-resolution image of the underwater scene, and the other part is the high-resolution image of the land scene. In theory, MFD-CycleGAN can be trained to enable the network to learn the mapping relationship between the two domains. The mapping relationship between the underwater scene and the land scene is learned through the MFD-CycleGAN model to achieve super-resolution reconstruction of underwater scene images. Among them, the underwater low-resolution images are from EUVP (URL is https: / / irvlab.cs.umn.edu / resources / euvp-dataset). The high-resolution land images are from LIVE (URL is http: / / live.ece.utexas.edu / rese arch / ChallengeDB / index.html). The training set trainA contains 1500 underwater degraded images, and the training set trainB contains 1500 land high-definition images. The test set consists of 100 underwater degraded images.

[0160] The experimental dataset is Figure 5 As shown, Figure 5 (a) is a set of low-resolution images of underwater scenes. Figure 5(b) is a set of high-resolution images of land scenes. The criteria for selecting underwater images in this article follow the principle of "completely random" to simulate real underwater conditions. Taking into account the limitations of contrast, light exposure, shooting environment and color distribution, this embodiment partially screens the land image data. Experiments have found that if bright light (such as white woven lights, sunlight, etc.) is introduced as a feature during the training process, the model will cause the small brightness differences in the underwater image to be over-amplified, and the output image will carry large areas of white light. In addition, there are large areas of pure color elements in images such as deserts, forests and oceans, and the qualitative differences between adjacent pixels are small, so features are difficult to extract, resulting in artifacts of multiple colors. This embodiment carefully screened the data set when selecting it to avoid the impact of the above problems.

[0161] In order to compare with other models, the size of underwater scene pictures is uniformly adjusted to 224×224. The entire network is encoded under the PaddlePaddle framework, and the training environment configuration is shown in Table 1. All algorithm studies in this embodiment are carried out under the same environment configuration. In order to compare the image enhancement performance of different models, this embodiment selects two evaluation indicators PSNR and NIQE that are widely used in image enhancement and restoration. At the same time, in order to further verify the performance of the MFD-CycleGAN model in image details and clarity, this embodiment uses Canny edge detection to analyze the image.

[0162] (2) Experimental results of the embodiments.

[0163] The image enhancement effects of MFD-CycleGAN, Retinex algorithm, histogram equalization method and traditional CycleGAN method were experimentally compared on the same dataset. The underwater image enhancement effects of different models are shown in Figure 2. Figure 6 shown.

[0164] Figure 6 (a) is a set of original images. Figure 6 (b) is a set of images enhanced by MFD-CycleGAN (i.e., the technical solution of this embodiment). Figure 6 (c) is a set of images enhanced by histogram equalization method. Figure 6 (d) is a set of images enhanced by the Retinex algorithm. Figure 6 (e) is a set of images enhanced by CycleGAN.

[0165] Analysis of two traditional underwater image enhancement models shows that the Retinex algorithm has a certain enhancement effect on underwater blurred images, solving the problem of underwater images being blue-green, but it also causes image distortion and obvious color loss, and a large number of noise points appear in the image. The histogram equalization method is slightly better than the Retinex algorithm, and the detail information restoration is more obvious, but the overall tone is also white, and there is obvious color loss. The CycleGAN model can restore color to a certain extent, but the restoration of blue-green background images is poor, and there are obvious red artifacts on the edges of objects.

[0166] In this embodiment, the MFD-CycleGAN model is proposed. After the encoding and decoding structure of the CycleGAN model generator is improved, the model shows good image color and detail restoration capabilities. Because the encoding structure retains the high-resolution feature map, the restored image color is more realistic and the details are richer. At the same time, MFD-CycleGAN introduces the MSFDF module. Compared with other algorithms, the underwater image enhanced by MFD-CycleGAN has less noise and clearer object edges. The experiments in this embodiment show that the MFD-CycleGAN model can obtain good color correction results in most scenarios, and its enhanced image effect is more in line with human visual perception.

[0167] Table 2 Comparison of underwater image enhancement evaluation indicators of different models

[0168]

[0169]

[0170] It can be seen from Table 2 that the MFD-CycleGAN model has certain advantages over the other three methods in terms of PSNR and NIQE indicators, which shows that the MFD-CycleGAN model of this embodiment has a certain effect on underwater image enhancement. The multi-scale parallel coding structure and the MSFDF module help the MFD-CycleGAN model to output enhanced images with less noise, higher restoration, and closer to the scenes in the real world.

[0171] In order to further verify the underwater image restoration effect of the MFD-CycleGAN model and verify the model's performance in restoring edge details, this example uses the Canny edge detection algorithm to process the output images of each model to compare the image restoration. Figure 7 As shown, Figure 7 (a) is the original image and its Canny edge detection result. Figure 7(b) is the image enhanced by MFD-CycleGAN and its Canny edge detection result. Figure 7 (c) is the image enhanced by the histogram equalization method and its Canny edge detection result. Figure 7 (d) is the image enhanced by the Retinex algorithm and its Canny edge detection result. Figure 7 (e) is the image enhanced by CycleGAN and its Canny edge detection result. The image restored by MFD-CycleGAN shows more key matching points and edge features, which means that the model can better retain the detailed information of the image. At the same time, the image restored by MFD-CycleGAN has less noise and is closer to the original image, which shows that the MFD-CycleGAN model performs well in processing noise and distortion of underwater images. The experimental results show that the MFD-CycleGAN model performs well in image details and clarity, and can better restore underwater images than other algorithms. Edge detection is a commonly used image analysis method that can effectively identify edge information in an image. Therefore, it is very reasonable and reliable to use Canny edge detection as a means of evaluating the quality of image restoration in this embodiment. It can be seen that MFD-CycleGAN is an effective underwater image restoration method that can greatly restore the quality and clarity of the image and has broad application prospects in practical applications.

[0172] This embodiment uses the MFD-CycleGAN model to achieve super-resolution reconstruction and enhancement of underwater scene images. In terms of data sets, this embodiment follows the principle of "completely random" to simulate real underwater conditions, and partially screens land image data. In the experiment, the MFD-CycleGAN model was compared with the Retinex algorithm, the histogram equalization method and the traditional CycleGAN method in the same data set for image enhancement effect experiments. The results show that the MFD-CycleGAN model performs well in image details and clarity compared with other methods, and has certain advantages in PSNR and NIQE indicators. In addition, the MFD-CycleGAN model adopts a multi-scale parallel coding structure and MSFDF module, which makes the model perform well in restoring image color and details. The experimental results also show that the image enhanced by MFD-CycleGAN shows more key matching points and edge features, which is closer to the original image, indicating that the MFD-CycleGAN model performs well in processing noise and distortion of underwater images. At the same time, this embodiment also uses the Canny edge detection algorithm to process the output image of each model to further verify the performance of the model in edge detail recovery and clarity enhancement. The experimental results show that the image enhanced by MFD-CycleGAN shows more key matching points and edge features, which means that the MFD-CycleGAN model can better preserve the detail information of the image.

[0173] Improving the quality of underwater images is crucial for underwater scientific research. The MFD-CycleGAN model of this embodiment can be used for underwater biological observation, underwater geological structure analysis, etc. Through enhanced underwater images, marine resources such as oil, natural gas, and minerals can be more accurately identified and evaluated. Through underwater image enhancement technology, underwater environmental changes can be better monitored, pollution can be assessed, and marine ecology can be protected. For the monitoring and management of fishery resources and the monitoring of aquaculture environments, enhanced underwater images can provide more detailed information.

[0174] Reference Figure 8 The embodiment of the present application also provides an underwater image enhancement system based on a generative adversarial network, the system comprising a data acquisition unit 100, a training set determination unit 200, a first construction unit 300, a second construction unit 400 and an image enhancement unit 500, wherein:

[0175] A data acquisition unit 100 is used to acquire an underwater image dataset and a land image dataset;

[0176] A training set determination unit 200, configured to determine a training set based on an underwater image data set and a land image data set;

[0177] A first construction unit 300 is used to construct a multi-scale parallel encoder including multiple branches, and to construct a multi-scale fusion decoder including a multi-scale feature dense fusion module;

[0178] A second construction unit 400 is used to construct an underwater image enhancement model based on a generative adversarial network, which includes two generators and two discriminators, wherein each generator is constructed based on a multi-scale parallel encoder and a multi-scale fusion decoder;

[0179] The image enhancement unit 500 is used to train the underwater image enhancement model based on the generative adversarial network using the training set to obtain the trained underwater image enhancement model, so as to use the trained underwater image enhancement model to perform image enhancement on the underwater image to be enhanced.

[0180] It should be noted that since the underwater image enhancement system based on a generative adversarial network in this embodiment and the above-mentioned underwater image enhancement method based on a generative adversarial network are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0181] Reference Fig. 9 The present application also provides an electronic device, which includes:

[0182] at least one memory;

[0183] at least one processor;

[0184] at least one program;

[0185] The program is stored in the memory, and the processor executes at least one program to implement the underwater image enhancement method based on the generative adversarial network implemented in the present disclosure.

[0186] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0187] The electronic device according to the embodiment of the present application is described in detail below.

[0188] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0189] The memory 1700 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the underwater image enhancement method based on the generative adversarial network of the embodiment of the present disclosure.

[0190] Input / output interface 1800, used to implement information input and output;

[0191] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0192] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0193] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0194] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned underwater image enhancement method based on a generative adversarial network.

[0195] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0196] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0197] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0198] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0200] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0201] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0202] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge possessed by ordinary technicians in the relevant technical field, various changes can be made without departing from the purpose of the present application.

Claims

1. An underwater image enhancement method based on a generative adversarial network, characterized in that: The method comprises: Obtain underwater image datasets and land image datasets; Determine a training set according to the underwater image dataset and the land image dataset; Construct a multi-scale parallel encoder containing multiple branches, and construct a multi-scale fusion decoder containing a multi-scale feature dense fusion module; Constructing an underwater image enhancement model based on a generative adversarial network comprising two generators and two discriminators, wherein each of the generators is constructed based on the multi-scale parallel encoder and the multi-scale fusion decoder; The training set is used to train the underwater image enhancement model based on the generative adversarial network to obtain a trained underwater image enhancement model, so that the trained underwater image enhancement model can be used to perform image enhancement on the underwater image to be enhanced.

2. The underwater image enhancement method based on generative adversarial network according to claim 1, characterized in that: When the trained underwater image enhancement model is used to perform image enhancement on the underwater image to be enhanced, the method further includes: Acquiring an underwater image to be enhanced; Using the multi-scale parallel encoder to encode the underwater image to be enhanced, to obtain encoding results with multiple resolutions, wherein the resolutions of the encoding results with multiple resolutions are different; The encoding results of the multiple resolutions are input into the multi-scale fusion decoder for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder.

3. The underwater image enhancement method based on generative adversarial network according to claim 2 is characterized in that: The multi-scale parallel encoder comprises a first convolution layer filled with 1 and a plurality of down-sampling modules, and the underwater image to be enhanced is encoded by the multi-scale parallel encoder to obtain encoding results of multiple resolutions, including: Inputting the underwater image to be enhanced into the first convolution layer filled with 1 to obtain a convolution result; The convolution result is encoded using multiple downsampling modules to obtain encoding results of multiple resolutions.

4. The underwater image enhancement method based on generative adversarial network according to claim 3 is characterized in that: The convolution result is encoded by using multiple downsampling modules to obtain encoding results of multiple resolutions, including: Divide the multiple downsampling modules into three parallel-connected branches, the three parallel-connected branches are respectively a first branch, a second branch and a third branch, each branch comprises a plurality of downsampling modules, the downsampling modules comprise a plurality of second convolutional layers filled with 1 and an unfilled third convolutional layer, or the downsampling modules comprise a plurality of second convolutional layers filled with 1; Inputting the convolution result into a plurality of second convolution layers filled with 1 of the first branch to obtain a first output result; Inputting the first output result into the unfilled third convolutional layer of the first branch to obtain a second output result; Inputting the first output result into a plurality of second convolutional layers filled with 1 of the first branch to obtain an encoding result of a first resolution output by the first branch; Inputting the second output result into a plurality of second convolutional layers filled with 1 of the second branch to obtain a third output result; Inputting the third output result into the unfilled third convolutional layer of the second branch to obtain a fourth output result; Inputting the third output result into a plurality of second convolutional layers filled with 1 of the second branch to obtain an encoding result of a second resolution output by the second branch; The fourth output result is input into a plurality of second convolutional layers filled with 1 of the third branch to obtain an encoding result of a third resolution output by the third branch.

5. The underwater image enhancement method based on generative adversarial network according to claim 2, characterized in that: The step of inputting the encoding results of the multiple resolutions into the multi-scale fusion decoder for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder comprises: Performing the same convolution operation on the encoding result of each resolution of the encoding results of the multiple resolutions to obtain convolution output results of multiple resolutions; The convolution output results of the multiple resolutions are input into the multi-scale feature dense fusion module for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder.

6. The underwater image enhancement method based on generative adversarial network according to claim 5, characterized in that: The multi-scale feature dense fusion module includes upsampling, and the convolution output results of the multiple resolutions are input into the multi-scale feature dense fusion module for image enhancement to obtain an enhanced image output by the multi-scale fusion decoder, including: Upsampling a convolution output result of a first resolution among the convolution output results of the multiple resolutions to obtain a first upsampling result, wherein the convolution output result of the first resolution is a convolution output result of the lowest resolution; Performing pixel-by-pixel addition and fusion of the first upsampling result and the convolution output result of the second resolution to obtain a first fused feature map, wherein the resolution of the convolution output result of the second resolution is higher than the resolution of the convolution output result of the first resolution; If there is a convolution output result with a third resolution higher than the resolution of the convolution output result with the second resolution, upsampling the first fused feature map to obtain a second upsampling result; Adding features of the second upsampling result and the first upsampling result to obtain a feature addition result; The feature addition result and the convolution output result of the third resolution are added and fused pixel by pixel to obtain a second fused feature map, and so on, until the convolution output result with the highest resolution among the convolution output results of the multiple resolutions is added and fused pixel by pixel with up-sampling results corresponding to all the convolution output results with low resolutions to obtain a target enhanced feature map; A convolution operation is performed on the target enhanced feature map to obtain an enhanced image output by the multi-scale fusion decoder.

7. The underwater image enhancement method based on generative adversarial network according to claim 6, characterized in that: The convolution output result with the highest resolution among the convolution output results with the multiple resolutions is added and fused pixel by pixel with up-sampling results corresponding to all the convolution output results with the low resolutions, including: α j ＝F i-1 F i =Φ(a i ,b i ) Among them, α j and F i-1 Represents the fused feature map corresponding to the convolution output result of the i-1th resolution, F i represents the fused feature map corresponding to the convolution output result of the i-th resolution, β i represents the upsampling result corresponding to the convolution output result of the i-th resolution, U represents upsampling, Φ represents pixel-by-pixel addition, and α i Represents the convolution output result of the i-th resolution.

8. An underwater image enhancement system based on a generative adversarial network, characterized in that: The system comprises: A data acquisition unit, used to acquire an underwater image data set and a land image data set; A training set determination unit, used to determine a training set according to the underwater image data set and the land image data set; A first construction unit is used to construct a multi-scale parallel encoder including multiple branches, and to construct a multi-scale fusion decoder including a multi-scale feature dense fusion module; A second construction unit is used to construct an underwater image enhancement model based on a generative adversarial network, which includes two generators and two discriminators, wherein each of the generators is constructed based on the multi-scale parallel encoder and the multi-scale fusion decoder; The image enhancement unit is used to use the training set to train the underwater image enhancement model based on the generative adversarial network to obtain a trained underwater image enhancement model, so as to use the trained underwater image enhancement model to perform image enhancement on the underwater image to be enhanced.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the underwater image enhancement method based on the generative adversarial network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the underwater image enhancement method based on a generative adversarial network as described in any one of claims 1 to 7.

Images