Underwater image enhancement method with wavelet transform and fusion attention mechanism

Abstract

The invention relates to the field of image processing, in particular to an underwater image enhancement method with wavelet transform and a fusion attention mechanism, and the method comprises the steps: obtaining underwater image data, analyzing the dominant feature information of the image data, calculating feature extraction characterization parameters, setting labels, and constructing an underwater image enhancement network. The generator receives the image data and then generates an output image, and the discriminator obtains the output image of the generator, judges whether to adjust the number of matrix grids based on the label so as to output a matrix corresponding to the number of matrix grids, judges a local area of the output image, and determines the construction state of the underwater image enhancement network. And training the underwater image enhancement network by using the underwater image of the training set through a composite loss function until convergence. According to the method, the underwater image enhancement network is constructed by using the generator and the discriminator, the degraded underwater image is enhanced, and the image quality and the reduction degree of underwater details are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an underwater image enhancement method with wavelet transform and fusion attention mechanism. Background Art

[0002] Underwater image enhancement is a key technology to solve underwater imaging problems. It mainly targets the chromatic aberration and low visibility problems of underwater images caused by light absorption and scattering. As an important means of obtaining water information, underwater optical images not only play a key role in scientific research and detection, but also have important value in the fields of national defense and military. They can intuitively display the underwater environment and provide reliable visual support for tasks such as submarine navigation, military reconnaissance, and sea area monitoring. However, the special medium properties of water bodies and impurity interference make light easily absorbed and scattered when propagating in water, resulting in image intensity attenuation and reduced clarity. The dynamic water environment further exacerbates the complexity of image degradation. This degradation not only affects the visual effect, but also brings challenges to information extraction, reducing the accuracy of underwater military security, resource monitoring, and submarine covert operations. Therefore, it is crucial to improve underwater image processing and enhancement technology.

[0003] Chinese patent publication number: CN117495687A, discloses an underwater image enhancement method, including the following steps: S1, obtain a high-definition underwater image data set including a reference image, establish a training set and a test set; S2, establish an adaptive histogram equalization model for image preprocessing to improve the brightness and contrast of the image; S3, establish an image enhancement model based on an improved cGAN network; S4, train the improved cGAN network using the training set until convergence; S5, input the high-definition underwater image to be analyzed into the converged improved cGAN network, and output an enhanced high-quality and clear underwater image. The method of the present invention obtains an enhanced high-quality and clear underwater image, which can effectively avoid the degradation phenomena such as color cast, detail loss, target blur and low contrast of the underwater image.

[0004] Chinese patent publication number: CN115034985A, discloses an underwater image enhancement method, including: inverting the original image taken underwater to obtain a dark channel image; calculating the global background light and underwater ambient light based on the dark channel image; calculating the transmittance of the water medium to light; calculating the preliminary defogging underwater image based on the global background light and the transmittance of the water medium to light; using a multi-scale Ret inex algorithm to perform image enhancement processing on the preliminary defogging underwater image; using a color restoration function to compensate for image distortion; using a histogram to intercept and balance the proportion of multi-scale Ret inex processing values ​​of different color channels in the final processed image to obtain a secondary processed underwater image; using a bilateral filter of Gaussian filtering and guided filtering to perform noise processing on the secondary processed underwater image to obtain the best image. The present invention improves the clarity of the original underwater image with foggy color cast and low quality, and reduces the degree of color deviation and noise.

[0005] However, there are still the following problems in the prior art:

[0006] In actual situations, although the existing underwater image enhancement methods can improve visual effects by adjusting grayscale and contrast, they are limited in effect when dealing with color distortion and uneven lighting in complex underwater environments. Although the physical model-based method has strong explanatory power in theory, its robustness depends on the model accuracy and input parameters. If the model accuracy and input parameters are inaccurate, the clarity of the underwater image will be low. Therefore, during the calculation process, due to insufficient consideration of underwater lighting characteristics and color distortion problems, the image quality is poor, and it is difficult to accurately restore underwater details, which affects the actual application effect. Summary of the invention

[0007] To this end, the present invention provides an underwater image enhancement method with wavelet transform and fusion attention mechanism to solve the problem that in actual situations, in the existing underwater image enhancement method, although the visual effect can be improved by adjusting the grayscale and contrast, the effect is limited when dealing with color distortion and uneven illumination in complex underwater environments. The method based on the physical model, although theoretically explanatory, its robustness depends on the model accuracy and input parameters. If the model accuracy and input parameters are inaccurate, the clarity of the underwater image will be low. Therefore, in the calculation process, due to insufficient consideration of the underwater illumination characteristics and color distortion problems, the image quality is poor, it is difficult to accurately restore the underwater details, and the actual application effect is affected.

[0008] To achieve the above object, the present invention provides an underwater image enhancement method with wavelet transform and fusion attention mechanism, which comprises:

[0009] Acquire underwater image data to construct an underwater image dataset, including degraded underwater images and their corresponding clear reference images, and divide the dataset into a test set, a training set, and a validation set in proportion;

[0010] Extracting image data from a test set, analyzing explicit feature information of the image data, and calculating feature extraction characterization parameters based on the explicit features to set labels for the image data;

[0011] Construct an underwater image enhancement network, wherein the generator generates an output image after receiving image data, and the discriminator obtains the output image of the generator, traces the label corresponding to the output image, and determines whether to adjust the number of matrix grids based on the label, so as to combine the output image with a clear reference image through multiple sets of convolution operations to output a matrix corresponding to the number of matrix grids;

[0012] Determine the local area of ​​the output image according to the determination score of the matrix, determine the construction state of the underwater image enhancement network, and determine whether the output image needs to be generated again by the generator;

[0013] Using underwater images of a training set to train the underwater image enhancement network through a composite loss function until convergence;

[0014] The explicit feature information includes the number of difference pixels and contour density in the image data, and the elements of the matrix include judgment scores for several local areas of the output image.

[0015] Furthermore, the generator can decompose the first-level output features into frequency domain information through wavelet transform, and fuse them with the last-level output features.

[0016] Furthermore, the process of calculating feature extraction characterization parameters based on explicit features includes:

[0017] Determine difference pixels based on the chromaticity difference of each pixel in the image data, and record the number of difference pixels;

[0018] Identify contours in image data and calculate contour density;

[0019] Determine the ratio of the number of difference pixels to the number of reference difference pixels as a difference pixel influencing factor;

[0020] Determine the ratio of the contour density to the reference contour density as a contour density influencing factor;

[0021] Determine the weighted sum of the difference pixel influence factor and the contour density influence factor as the feature extraction characterization parameter;

[0022] The chromaticity difference between the difference pixels must be greater than a predetermined chromaticity difference threshold.

[0023] Furthermore, the label is set for the image data, wherein:

[0024] If the feature extraction characterization parameter is greater than a reference threshold, setting the label of the image data to a detail feature label;

[0025] If the feature extraction characterization parameter is less than or equal to the reference threshold, the label of the image data is set to a non-detail feature label.

[0026] Further, it is determined whether to adjust the number of matrix grids based on the label, wherein:

[0027] If the label is a detail feature label, the number of matrix grids is positively correlated with the feature extraction characterization parameter;

[0028] If the label is a non-detail feature label, the number of matrix grids is kept unchanged.

[0029] Furthermore, the composite loss function is expressed by formula (1):

[0030] L all =min G max D L WGAN +α1L1+α2L GD +α3L IC (1)

[0031] In formula (1), G represents the generator, D represents the discriminator, and L WAGN represents the adversarial loss to measure the difference between the generated data and the real data distribution through the Wasserstein distance, L1 represents the absolute difference between the generated image and the clear reference image, α1 represents the weight coefficient of the absolute difference between the generated image and the clear reference image, L GD represents the gradient difference loss, α2 represents the weight coefficient of the gradient difference loss, L IC It represents the image content loss which measures the similarity between the generated image and the clear reference image in terms of high-level semantic features through the pre-trained model of the VGG19 network. α3 represents the weight coefficient of the image content loss which measures the similarity between the generated image and the clear reference image in terms of high-level semantic features through the pre-trained model of the VGG19 network.

[0032] Furthermore, the adversarial loss measures the difference between the generated data and the real data distribution through the Wasserstein distance, which is expressed by formula (2):

[0033]

[0034] In formula (2), E represents the mathematical expectation, IC represents the undistorted underwater image, I D represents the distortion of underwater images, represents the samples along the straight line between the original data and the point pairs distributed by the generator, Represents the input content, α GP represents the weight coefficient of the gradient penalty term, express is from The sample obtained from Representation Discriminator For input The gradient represents the local rate of change at that point.

[0035] Furthermore, the absolute difference between the generated image and the real image is expressed by formula (3),

[0036] L1=E[||I C -(G(I D ))||1] (3)

[0037] In formula (3), G(I D ) represents the predicted image generated by the generator based on the distorted deformation of the underwater image.

[0038] Furthermore, the gradient difference loss is expressed by formula (4):

[0039]

[0040] In formula (4), I P To predict the image, I P =G(I D ), is the loss sensitivity coefficient, where and Represent the pixel values ​​of the predicted image and the real image at position (i, j), i, j∈[0,255].

[0041] Furthermore, the image content loss measures the similarity between the generated image and the real image in high-level semantic features through the pre-trained model of the VGG19 network, which is expressed by formula (5):

[0042]

[0043] In formula (5), Φ(·) represents the features of the image extracted using the VGG19 network.

[0044] Compared with the prior art, the present invention obtains underwater image data to construct an underwater image data set, analyzes the explicit feature information of the image data and calculates feature extraction characterization parameters to set labels for the image data, constructs an underwater image enhancement network, and the generator generates an output image after receiving the image data. The discriminator obtains the output image of the generator, determines whether to adjust the number of matrix grids based on the label, and outputs a matrix corresponding to the number of matrix grids. The local area of ​​the output image is judged according to the judgment score of the matrix, and the construction state of the underwater image enhancement network is determined to determine whether the output image needs to be generated again by the generator. The underwater image of the training set is used to train the underwater image enhancement network through a composite loss function until convergence. The present invention uses the generator and the discriminator to construct an underwater image enhancement network, enhances the degraded underwater image, and improves the image quality and the restoration of underwater details.

[0045] In particular, feature extraction representation parameters are calculated based on explicit features to set labels for image data. In actual situations, when underwater images are enhanced, there may be underwater images with rich detail features. At this time, if the preset number of matrix grids is still used, the enhanced underwater image will still be unclear. Based on this, consider analyzing the two more explicit features, the number of difference pixels and the contour density in the image data, to calculate feature extraction representation parameters to characterize the prominence of the detail features in the image data, and provide data support for the subsequent discriminator to generate the matrix. For image data with rich detail features, the matrix grid is adaptively adjusted to ensure the perception of local details. For image data with less detail features, the baseline matrix grid is maintained to save computing power.

[0046] In particular, the underwater image enhancement network includes a generator, which is used to supplement the features of the degraded underwater images and transmit the images after feature supplementation to the discriminator. In actual situations, when shooting underwater, unclear shots often occur. Based on this, the generator is based on the U-Net hybrid architecture that integrates wavelet transform and attention mechanism, and uses a five-layer U-shaped network to extract feature information and smooth transition of the input image. At the same time, jump connections are used between two layers with the same feature map to ensure cross-layer information transmission. Wavelet transform is specially introduced in the generator to decompose the first-level output features into frequency domain information, which is fused with the last level to supplement the feature set, further enhance the details and global information, and improve the image quality and the restoration of underwater details.

[0047] In particular, the underwater image enhancement network also includes a discriminator. The discriminator receives the output image of the generator and determines the output matrix in combination with the label to judge the authenticity of the image according to the judgment score of the matrix. In actual situations, the discriminator is mostly determined based on the adversarial training mechanism in GAN theory, but the training process of the adversarial training mechanism is relatively complex, and GAN tends to be unstable. Based on this, the PatchGAN architecture design based on the Markov model is considered and the cyclic residual convolution (RRC) module is embedded to construct the discriminator. Since the discrimination process of PatchGAN is based on local image blocks, the computational complexity can be reduced. The cyclic residual convolution (RRC) module can improve the discrimination accuracy of the image, while improving the image quality and the restoration of underwater details.

[0048] In particular, the convergence calculation of the underwater image enhancement network is performed according to the composite loss function to ensure the availability of the underwater image enhancement network. In actual situations, the convergence calculation of the underwater image enhancement network is mostly based on one function, which is limited and one-sided. Based on this, it is considered to construct a composite loss function for the convergence calculation of the underwater image enhancement network. Through multiple calculations, the underwater image enhancement network is made more accurate so that it can be applied to subsequent image enhancement in reality, thereby improving the image quality and the restoration of underwater details. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the method steps of an underwater image enhancement method with wavelet transform and fusion attention mechanism according to an embodiment of the invention;

[0050] Figure 2 A flowchart of setting labels for image data according to an embodiment of the invention;

[0051] Figure 3 A schematic diagram of the generator structure of an underwater image enhancement network with a fusion attention mechanism according to an embodiment of the invention;

[0052] Figure 4 A wavelet transform module embedded in the generator part upsampling of the embodiment of the invention;

[0053] Figure 5 A schematic diagram of the discriminator structure of an underwater image enhancement network based on a Markov model based PatchGAN architecture according to an embodiment of the invention. DETAILED DESCRIPTION

[0054] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0056] To facilitate reading of this plan, some of the terms involved in this plan are explained below:

[0057] Degraded underwater images: Images taken in an underwater environment may have color distortion, reduced contrast, blurred images, etc. due to the absorption and scattering of light by water. It is understandable that the images are partially degraded, and the greater the degree of degradation, the lower the image restoration quality.

[0058] Wavelet transform: A mathematical method that analyzes a signal by decomposing it into components of different scales (or frequencies). In practice, it is used to perform frequency domain analysis on the input image, thereby separating the information of different frequencies in order to better reconstruct and enhance the visual quality of underwater images.

[0059] Fusion attention mechanism: A technique widely used in deep learning that improves model performance by combining different types of attention mechanisms.

[0060] See also Figures 1 to 5 As shown, Figure 1 This is a schematic diagram of the method steps of an underwater image enhancement method with wavelet transform and fusion attention mechanism according to an embodiment of the invention. Figure 2 A flowchart of setting labels for image data according to an embodiment of the invention, Figure 3 A schematic diagram of the generator structure of an underwater image enhancement network with a fusion attention mechanism according to an embodiment of the invention is shown in FIG. Figure 4 The wavelet transform module embedded in the generator part upsampling of the embodiment of the invention, Figure 5 The discriminator structure diagram of the underwater image enhancement network based on the PatchGAN architecture of the Markov model of the embodiment of the invention. The underwater image enhancement method with wavelet transform and fusion attention mechanism of the present invention comprises:

[0061] Step S1, acquiring underwater image data to construct an underwater image dataset, including a degraded underwater image and its corresponding clear reference image, and dividing the dataset into a test set, a training set, and a validation set in proportion;

[0062] Step S2, extracting image data from the test set, analyzing explicit feature information of the image data, and calculating feature extraction characterization parameters based on the explicit features to set labels for the image data;

[0063] Construct an underwater image enhancement network, wherein the generator generates an output image after receiving image data, and the discriminator obtains the output image of the generator, traces the label corresponding to the output image, and determines whether to adjust the number of matrix grids based on the label, so as to combine the output image with a clear reference image through multiple sets of convolution operations to output a matrix corresponding to the number of matrix grids;

[0064] Step S3, judging the local area of ​​the output image according to the judgment score of the matrix, determining the construction state of the underwater image enhancement network, so as to determine whether the output image needs to be generated again by the generator;

[0065] Step S4, using the underwater images of the training set to train the underwater image enhancement network through a composite loss function until convergence;

[0066] The explicit feature information includes the number of difference pixels and contour density in the image data, and the elements of the matrix include judgment scores for several local areas of the output image.

[0067] Specifically, there is no limit on the proportion of the data set divided into proportions, as long as it is ensured that the network can be trained according to the data set. Preferably, according to the rule of thumb, for a large amount of data, it can be allocated according to a ratio of 70% training set, 15% validation set, and 15% test set. For a small amount of data or when the model requires more validation data to adjust parameters, it can be allocated according to a ratio of 60% training set, 20% validation set, and 20% test set. Those skilled in the art can also determine it based on actual conditions, which will not be repeated here.

[0068] Specifically, there is no limitation on the method of acquiring high-quality and clear reference images. For example, it can be an image taken in an underwater interference-free environment, which will not be elaborated here.

[0069] Specifically, in the implementation, the initial size of the image data received by the generator is 256×256×3, where 3 represents the number of RGB channels. The initial feature extraction in the image data is completed using a convolutional layer, the size of the convolution kernel is 3×3, and the step size is set to 1 to ensure that the detailed information is fully captured; the number of channels of the output feature map is set to 64, and the spatial size is kept at 256×256. After that, the feature map is input into the EMA attention mechanism, and the output feature map is downsampled by the maximum pooling operation; the kernel size of the maximum pooling is 2×2, and the step size is set to 2, so the feature map size is reduced from 256×256×64 to 1 28×128×64; the convolution kernel size of the second layer of convolution is 3×3, the number of output channels is 128, and the resulting feature map size is 128×128×128; the detail enhancement module introduces a direction-sensitive enhancement mechanism in the convolution operation to extract more detail information, and the enhanced feature map is sent to the EMA attention mechanism. The third and fourth layers of the network operate consistently with the second layer, with the feature map size of the third layer being 64×64×256 and the feature map size of the fourth layer being 32×32×512; the fifth level of the network uses pure convolution, and the output feature map size is 16×16×1024.

[0070] Specifically, the values ​​represented by the judgment scores of the matrix are not fixed values, but are dynamically adjusted through the gradient descent algorithm during the training process. These values ​​represent the judgment scores of the discriminator for whether each local area (i.e., the perception area) in the image is real or fake. These scores are continuous and can change continuously during the training process to better distinguish between real images and generated images.

[0071] Specifically, the output of the discriminator is a matrix of the same size as the input image, and the judgment score of each element usually ranges from 0 to 1, where 0 represents completely fake and 1 represents completely real.

[0072] Specifically, the generator can decompose the first-level output features into frequency domain information through wavelet transform and merge them with the last-level output features.

[0073] Specifically, the underwater image enhancement network includes a generator, which is used to supplement the features of the degraded underwater image and transmit the image after feature supplementation to the discriminator. In actual situations, when shooting underwater, the image is often unclear. Based on this, the generator is based on the U-Net hybrid architecture that integrates wavelet transform and attention mechanism, and uses a five-layer U-shaped network to extract feature information and smooth transition of the input image. At the same time, jump connections are used between two layers with the same feature map to ensure cross-layer information transmission. Wavelet transform is specially introduced in the generator to decompose the first-level output features into frequency domain information, which is fused with the last level to supplement the feature set, further enhance the details and global information, and improve the image quality and the restoration of underwater details.

[0074] Specifically, the five-layer U-type network refers to a U-Net network structure with five levels of encoders (downsampling paths) and five levels of decoders (upsampling paths). The functions of each layer are as follows:

[0075] The first layer consists of two branches. The first branch decomposes the original degraded underwater image into frequency domain features through wavelet transform. The frequency domain features include low-frequency features and a set of high-frequency features composed of horizontal, vertical and diagonal lines. The frequency domain feature information processed by wavelet transform is combined with the feature information of the last level of the network through fusion operation to further supplement the details and richness of the generated image.

[0076] The other branch of the first layer consists of a convolution block, an efficient multi-scale attention module (EMA), and a downsampling module. The convolution block contains a 3×3 convolution operation, a batch normalization operation, and a ReLU activation function to ensure efficient feature extraction and nonlinear mapping. The number of convolution kernels in this branch is 64. The downsampling module uses a maximum pooling operation to reduce the size of the feature map, thereby gradually reducing the spatial dimension.

[0077] Convolution blocks, detail enhancement convolution blocks, and EMA attention mechanisms are used in the second, third, and fourth layers of the network. The number of convolution kernels in the convolution blocks is 128, 256, and 512, respectively, gradually deepening the network to extract higher-level semantic information.

[0078] The number of convolution kernels in the fifth layer is increased to 1024 to capture richer and more complex features.

[0079] Specifically, the process of calculating feature extraction characterization parameters based on explicit features includes:

[0080] Determine difference pixels based on the chromaticity difference of each pixel in the image data, and record the number of difference pixels;

[0081] Identify contours in image data and calculate contour density;

[0082] Determine the ratio of the number of difference pixels to the number of reference difference pixels as a difference pixel influencing factor;

[0083] Determine the ratio of the contour density to the reference contour density as a contour density influencing factor;

[0084] Determine the weighted sum of the difference pixel influence factor and the contour density influence factor as the feature extraction characterization parameter;

[0085] The chromaticity difference between the difference pixels must be greater than a predetermined chromaticity difference threshold.

[0086] Specifically, the reference difference pixel number is calculated in advance, a number of clear reference pictures are obtained in advance, the average number of difference pixel points of the clear reference pictures is determined, and the average number of difference pixel points is set as the reference difference pixel number.

[0087] Specifically, the reference contour density is calculated in advance, a number of clear reference pictures are obtained in advance, the average contour density of the clear reference pictures is determined, and the average contour density is set as the reference contour density.

[0088] Specifically, the weight coefficient of the difference pixel influence factor is 0.54, and the weight coefficient of the contour density influence factor is 0.46.

[0089] Specifically, the predetermined chromaticity difference threshold is obtained by pre-calculation, a number of clear reference pictures are obtained in advance, the average chromaticity difference of the several clear reference pictures is determined, and 0.65 times of the average chromaticity difference is set as the chromaticity difference threshold.

[0090] Specifically, feature extraction representation parameters are calculated based on explicit features to set labels for image data. In actual situations, when underwater images are enhanced, there may be underwater images with rich detail features. At this time, if the preset number of matrix grids is still used, the enhanced underwater image will still be unclear. Based on this, consider analyzing the two more explicit features, the number of difference pixels and the contour density in the image data, to calculate feature extraction representation parameters to characterize the prominence of the detail features in the image data, and provide data support for the subsequent discriminator to generate the matrix. For image data with rich detail features, the matrix grid is adaptively adjusted to ensure the perception of local details. For image data with less detail features, the baseline matrix grid is maintained to save computing power.

[0091] Specifically, a label is set for the image data, where:

[0092] If the feature extraction characterization parameter is greater than a reference threshold, setting the label of the image data to a detail feature label;

[0093] If the feature extraction characterization parameter is less than or equal to the reference threshold, the label of the image data is set to a non-detail feature label.

[0094] Specifically, the reference threshold is selected within the interval (1.25, 1.5).

[0095] Specifically, the tag determines whether to adjust the number of matrix grids, where

[0096] If the label is a detail feature label, the number of matrix grids is positively correlated with the feature extraction characterization parameter;

[0097] If the label is a non-detail feature label, the number of matrix grids is kept unchanged.

[0098] It can be understood that the larger the feature extraction characterization parameter is, the richer the underwater image content is, and the more matrix grids are required.

[0099] In implementation, optionally,

[0100] If the benchmark threshold belongs to the interval [1.5, +∞], the number of grids is 64×64;

[0101] If the benchmark threshold belongs to the interval (1.25, 1.5), the number of grids is 32×32;

[0102] If the benchmark threshold is in the interval [0, 1.25], the number of grids is 16×16.

[0103] Specifically, the process of outputting a matrix through multiple sets of convolution operations includes:

[0104] Determining image data and corresponding data features of the generator output image;

[0105] Determine that the dot product operation result of the convolution kernel and each image data region of the output image is a single matrix;

[0106] stacking the single matrices to form a characteristic matrix;

[0107] Determining the characteristic matrix as an output matrix;

[0108] The convolution kernel is a single matrix, which is used to slide on the image data to extract features.

[0109] Specifically, the steps for calculating the dot product of the convolution kernel and each image data area of ​​the output image are as follows:

[0110] Step S01, determining the convolution kernel and each image data region, the convolution kernel and each image data region have the same size. It can be understood that the size of the image data region is determined by the number of matrix grids. The more matrix grids there are, the more image data regions need to be divided. Therefore, the more regions are measured, the more precise the determination result is, and it is suitable for output images with more features.

[0111] Step S02, multiplying each element of the convolution kernel by the element at the corresponding position in the underwater image data area;

[0112] Step S03, adding all the products obtained in step S02 to obtain a single value;

[0113] Step S04, placing the obtained single value at the corresponding position of the output feature map, sliding the convolution kernel one pixel to the right, and repeating the above process until the entire image area is covered;

[0114] Step S05: determine that the matrix composed of feature data corresponding to the entire image area covered is the feature matrix.

[0115] Specifically, the process of determining the construction status of the underwater image enhancement network includes,

[0116] Determine the judgment score of each matrix element;

[0117] Determine that the authenticity of the matrix element region image whose judgment score is greater than or equal to the judgment score threshold is true;

[0118] Determine the ratio of the number of region images with true authenticity to the number of all region images in the image;

[0119] Determine that the authenticity of the image whose ratio is greater than or equal to the reference ratio is true;

[0120] If the authenticity is true, the construction state of the underwater image enhancement network is complete, and the output image does not need to be generated again by the generator.

[0121] Or, the output image needs to be generated again by the generator;

[0122] Among them, each matrix element represents the judgment score of a perceptual area in the image output by the generator.

[0123] Specifically, the judgment score threshold is calculated in advance, a number of clear reference images are obtained in advance, the average value of each element in each clear reference image is calculated, and the average value of each element in each clear reference image is determined as the judgment score threshold.

[0124] Specifically, the benchmark ratio is calculated in advance, and a number of high-quality and clear reference images are obtained in advance, and the number of authentic reference area images in the number of high-quality and clear reference images is determined. 0.86 times the ratio of the number of authentic reference area images to the total number of reference area images in the number of high-quality and clear reference images is determined as the benchmark ratio.

[0125] Specifically, the underwater image enhancement network also includes a discriminator. The discriminator receives the output image of the generator and determines the output matrix in combination with a clear reference image to judge the authenticity of the image according to the judgment score of the matrix. In actual situations, the discriminator is mostly determined based on the adversarial training mechanism in GAN theory, but the training process of the adversarial training mechanism is relatively complex, and GAN tends to be unstable. Based on this, the PatchGAN architecture design based on the Markov model is considered and the cyclic residual convolution (RRC) module is embedded to construct the discriminator. Since the discrimination process of PatchGAN is based on local image blocks, the computational complexity can be reduced. The cyclic residual convolution (RRC) module can improve the discrimination accuracy of the image, while improving the image quality and the restoration of underwater details.

[0126] Specifically, the composite loss function is expressed by formula (1):

[0127] L all =min G max D L WGAN +α1L1+α2L GD +α3L IC (1)

[0128] In formula (1), G represents the generator, D represents the discriminator, and L WAGN represents the adversarial loss to measure the difference between the generated data and the real data distribution through the Wasserstein distance, L1 represents the absolute difference between the generated image and the clear reference image, α1 represents the weight coefficient of the absolute difference between the generated image and the clear reference image, L GD represents the gradient difference loss, α2 represents the weight coefficient of the gradient difference loss, L IC It represents the image content loss which measures the similarity between the generated image and the clear reference image in terms of high-level semantic features through the pre-trained model of the VGG19 network. α3 represents the weight coefficient of the image content loss which measures the similarity between the generated image and the clear reference image in terms of high-level semantic features through the pre-trained model of the VGG19 network.

[0129] Specifically, in the implementation, the value of α1 is 100, the value of α2 is 10, and the value of α3 is 3. It can be understood that the values ​​of α1, α2, and α3 are weight coefficients of hyperparameters adjusted empirically.

[0130] Specifically, the convergence calculation of the underwater image enhancement network is performed according to the composite loss function to ensure the availability of the underwater image enhancement network. In actual situations, the convergence calculation of the underwater image enhancement network is mostly based on one function, which is limited and one-sided. Based on this, it is considered to construct a composite loss function for the convergence calculation of the underwater image enhancement network. Through multiple calculations, the underwater image enhancement network is made more accurate, so as to improve the image quality and the restoration of underwater details in the subsequent image enhancement applied to reality.

[0131] Specifically, the adversarial loss measures the difference between the generated data and the real data distribution through the Wasserstein distance, which is expressed by formula (2):

[0132]

[0133] In formula (2), E represents the mathematical expectation, I C represents the undistorted underwater image, I D represents the distortion of underwater images, represents the samples along the straight line between the original data and the point pairs distributed by the generator, Represents the input content, α GP represents the weight coefficient of the gradient penalty term, express is from The sample obtained from Representation Discriminator For input The gradient represents the local rate of change at that point.

[0134] Specifically, the absolute difference between the generated image and the real image is expressed by formula (3),

[0135] L1=E[||I C -(G(I D ))||1] (3)

[0136] In formula (3), G(I D ) represents the predicted image generated by the generator based on the distorted deformation of the underwater image.

[0137] Furthermore, the gradient difference loss is expressed by formula (4):

[0138]

[0139] In formula (4), I P To predict the image, I P =G(I D ), is the loss sensitivity coefficient, where and Represent the pixel values ​​of the predicted image and the real image at position (i, j), i, j∈[0,255].

[0140] Specifically, the main function of the loss sensitivity coefficient is to control the weight or nonlinearity of the gradient error of the generated image, which can achieve different loss optimization objectives, thereby better balancing the edge details and global consistency of the generated image.

[0141] Specifically, the image content loss measures the similarity between the generated image and the real image in high-level semantic features through the pre-trained model of the VGG19 network, which is expressed by formula (5):

[0142]

[0143] In formula (5), Φ(·) represents the features of the image extracted using the VGG19 network.

[0144] It can be understood that the processes of calculating the expected E, summing and extracting image features are all performed inside the computer, and those skilled in the art only need to ensure that the data is correctly input, which will not be elaborated here.

[0145] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. An underwater image enhancement method with wavelet transform and fusion attention mechanism, characterized in that: include: Acquire underwater image data to construct an underwater image dataset, including degraded underwater images and their corresponding clear reference images, and divide the dataset into a test set, a training set, and a validation set in proportion; Extracting image data from a test set, analyzing explicit feature information of the image data, and calculating feature extraction characterization parameters based on the explicit features to set labels for the image data; Construct an underwater image enhancement network, wherein the generator generates an output image after receiving image data, and the discriminator obtains the output image of the generator, traces the label corresponding to the output image, and determines whether to adjust the number of matrix grids based on the label, so as to combine the output image with a clear reference image through multiple sets of convolution operations to output a matrix corresponding to the number of matrix grids; Determine the local area of ​​the output image according to the determination score of the matrix, determine the construction state of the underwater image enhancement network, and determine whether the output image needs to be generated again by the generator; Using underwater images of a training set to train the underwater image enhancement network through a composite loss function until convergence; The explicit feature information includes the number of difference pixels and contour density in the image data, and the elements of the matrix include judgment scores for several local areas of the output image.

2. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 1, characterized in that: The generator can decompose the first-level output features into frequency domain information through wavelet transform and merge them with the last-level output features.

3. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 2 is characterized in that: The process of calculating feature extraction characterization parameters based on explicit features includes: Determine difference pixels based on the chromaticity difference of each pixel in the image data, and record the number of difference pixels; Identify contours in image data and calculate contour density; Determine the ratio of the number of difference pixels to the number of reference difference pixels as a difference pixel influencing factor; Determine the ratio of the contour density to the reference contour density as a contour density influencing factor; Determine the weighted sum of the difference pixel influence factor and the contour density influence factor as the feature extraction characterization parameter; The chromaticity difference between the difference pixels must be greater than a predetermined chromaticity difference threshold.

4. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 1, characterized in that: The label is set for the image data, wherein: If the feature extraction characterization parameter is greater than a reference threshold, setting the label of the image data to a detail feature label; If the feature extraction characterization parameter is less than or equal to the reference threshold, the label of the image data is set to a non-detail feature label.

5. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 4, characterized in that: Based on the label, it is determined whether to adjust the number of matrix grids, wherein: If the label is a detail feature label, the number of matrix grids is positively correlated with the feature extraction characterization parameter; If the label is a non-detail feature label, the number of matrix grids is kept unchanged.

6. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 1, characterized in that: The composite loss function is expressed by formula (1): L all =min G max D L WGAN +α1L1+α2L GD +α3L IC (1) In formula (1), G represents the generator, D represents the discriminator, and L WAGN represents the adversarial loss to measure the difference between the generated data and the real data distribution through the Wasserstein distance, L1 represents the absolute difference between the generated image and the clear reference image, α1 represents the weight coefficient of the absolute difference between the generated image and the clear reference image, L GD represents the gradient difference loss, α2 represents the weight coefficient of the gradient difference loss, L IC It represents the image content loss which measures the similarity between the generated image and the clear reference image in terms of high-level semantic features through the pre-trained model of the VGG19 network. α3 represents the weight coefficient of the image content loss which measures the similarity between the generated image and the clear reference image in terms of high-level semantic features through the pre-trained model of the VGG19 network.

7. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 6, characterized in that: The adversarial loss measures the difference between the generated data and the real data distribution through the Wasserstein distance, which is expressed by formula (2): In formula (2), E represents the mathematical expectation, I C represents the undistorted underwater image, I D represents the distortion of underwater images, represents the samples along the straight line between the original data and the point pairs distributed by the generator, Represents the input content, α GP represents the weight coefficient of the gradient penalty term, express is from The sample obtained from Representation Discriminator For input The gradient represents the local rate of change at that point.

8. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 7, characterized in that: The absolute difference between the generated image and the real image is expressed by formula (3), L1 = E[||I C -(G(I D ))||1] (3) In formula (3), G(I D ) represents the predicted image generated by the generator based on the distorted deformation of the underwater image.

9. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 8, characterized in that: The gradient difference loss is expressed by formula (4): In formula (4), I P To predict the image, I P =G(I D ), is the loss sensitivity coefficient, where and Represent the pixel values ​​of the predicted image and the real image at position (i, j), i, j∈[0,255].

10. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 9, characterized in that: The image content loss measures the similarity between the generated image and the real image in high-level semantic features through the pre-trained model of the VGG19 network, which is expressed by formula (5): In formula (5), Φ(·) represents the features of the image extracted using the VGG19 network.

Images