An underwater image enhancement method with wavelet transform and fusion attention mechanism

Abstract

The present application relates to the field of image processing, especially to an underwater image enhancement method with wavelet transform and fusion attention mechanism, the present application obtains underwater image data, analyzes the explicit feature information of the image data and calculates the feature extraction characteristic parameter to set the label, constructs the underwater image enhancement network, the generator generates the output image after receiving the image data, the discriminator obtains the output image of the generator, judges whether to adjust the number of matrix grids based on the label, to output the corresponding matrix of the number of matrix grids, judges the local area of the output image, determines the construction state of the underwater image enhancement network, and trains the underwater image enhancement network by using the underwater image of the training set through the composite loss function until convergence. The present application constructs the underwater image enhancement network by using the generator and the discriminator, enhances the degraded underwater image, and improves the image quality and the restoration degree of underwater details.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and particularly relates to an underwater image enhancement method with wavelet transform and fusion attention mechanism. BACKGROUND

[0002] Underwater image enhancement is a key technology to solve the problem of underwater imaging. It mainly aims at the color difference and low visibility problems of underwater images caused by light absorption and scattering. Underwater optical images, as an important means of obtaining water information, not only play a key role in scientific research and detection, but also have important value in the field of national defense and military. It can intuitively display the underwater environment and provide reliable visual support for submarine navigation, military reconnaissance, sea monitoring and other tasks. However, the special medium properties and impurities of water make the light easily absorbed and scattered when propagating in water, resulting in image intensity attenuation and clarity reduction. The dynamic water environment further aggravates the complexity of image degradation. This degradation not only affects the visual effect, but also brings challenges to information extraction, reducing the accuracy of underwater military security, resource monitoring and submarine covert operation. Therefore, it is crucial to improve underwater image processing and enhancement technology.

[0003] Chinese patent publication No. CN117495687A discloses an underwater image enhancement method, comprising the following steps: S1, obtaining a high-definition underwater image dataset containing a reference image, establishing a training set and a test set; S2, establishing an adaptive histogram equalization model for image preprocessing to improve the brightness and contrast of the image; S3, establishing an image enhancement model based on an improved cGAN network; S4, training the improved cGAN network using the training set until convergence; S5, inputting the high-definition underwater image to be analyzed into the converged improved cGAN network, and outputting the enhanced high-quality and clear underwater image. The method can effectively avoid the degradation phenomena such as color cast, detail loss, target blur and low contrast of underwater images.

[0004] CN115034985A discloses an underwater image enhancement method, which comprises: performing reverse processing on the original image shot underwater to obtain a dark channel image; calculating the global background light and the underwater environment light according to the dark channel image; calculating the transmittance of the water medium to light; calculating the preliminary de-fogging processed underwater image according to the global background light and the transmittance of the water medium to light; performing image enhancement processing on the preliminary de-fogging processed underwater image by using a multi-scale Retinex algorithm; compensating for image distortion by using a color restoration function; balancing the proportion of multi-scale Retinex processing values of different color channels in the final processed image by using histogram interception, to obtain a secondary processed underwater image; and performing noise processing on the secondary processed underwater image by using a bilateral filter of Gaussian filtering and guided filtering, to obtain a final image. The present application improves the clarity of the original image shot underwater with fog and color deviation, and reduces the degree of color deviation and noise.

[0005] However, the prior art still has the following problems,

[0006] In actual situations, although the visual effect can be improved by adjusting the gray scale and contrast in the existing underwater image enhancement method, the effect is limited when dealing with color distortion and uneven illumination in complex underwater environments. The method based on the physical model has strong theoretical explanation, but its robustness depends on the model accuracy and input parameters. If the model accuracy and input parameters are not accurate, it will lead to low clarity of underwater images. Therefore, due to the insufficient consideration of underwater illumination characteristics and color distortion problems in the calculation process, the image quality is poor, it is difficult to accurately restore underwater details, and the actual application effect is affected. SUMMARY

[0007] Therefore, the present application provides an underwater image enhancement method with wavelet transform and fusion attention mechanism to solve the problem that in actual situations, although the visual effect can be improved by adjusting the gray scale and contrast in the existing underwater image enhancement method, the effect is limited when dealing with color distortion and uneven illumination in complex underwater environments. The method based on the physical model has strong theoretical explanation, but its robustness depends on the model accuracy and input parameters. If the model accuracy and input parameters are not accurate, it will lead to low clarity of underwater images. Therefore, due to the insufficient consideration of underwater illumination characteristics and color distortion problems in the calculation process, the image quality is poor, it is difficult to accurately restore underwater details, and the actual application effect is affected.

[0008] To achieve the above-mentioned purpose, the present application provides an underwater image enhancement method with wavelet transform and fusion attention mechanism, which comprises:

[0009] Acquiring underwater image data to construct an underwater image dataset including degraded underwater images and corresponding clear reference images, and dividing the dataset into a test set, a training set and a validation set in proportion;

[0010] Extracting image data from the test set, analyzing the explicit feature information of the image data, calculating feature extraction representation parameters based on the explicit features, and setting labels for the image data;

[0011] Constructing an underwater image enhancement network, the generator generates output images after receiving image data, the discriminator obtains the output images of the generator, traces the labels corresponding to the output images, and determines whether to adjust the number of matrix grids based on the labels, so that the output images combined with clear reference images are output through multiple convolution operations. The number of matrix grids corresponding to the matrix;

[0012] According to the judgment score of the matrix, the local area of the output image is judged, and the construction state of the underwater image enhancement network is determined to determine whether the output image needs to be generated again by the generator;

[0013] Using the underwater images of the training set to train the underwater image enhancement network through a composite loss function until convergence;

[0014] Wherein, the explicit feature information includes the number of difference pixels and the contour density in the image data, and the elements of the matrix include the judgment scores of several local areas of the output image.

[0015] Further, the generator can decompose the first-level output feature into frequency domain information through wavelet transform and fuse it with the last-level output feature.

[0016] Further, the process of calculating feature extraction representation parameters based on explicit features includes,

[0017] Determine the difference pixels based on the chroma difference of each pixel in the image data, and record the number of difference pixels;

[0018] Identify the contour in the image data and calculate the contour density;

[0019] Determine the ratio of the number of difference pixels to the reference number of difference pixels as the difference pixel influence factor;

[0020] Determine the ratio of the contour density to the reference contour density as the contour density influence factor;

[0021] Determine the weighted sum of the difference pixel influence factor and the contour density influence factor as the feature extraction representation parameter;

[0022] Wherein, the chroma difference between the difference pixels is greater than a predetermined chroma difference threshold.

[0023] Furthermore, the label is set for the image data, wherein,

[0024] If the feature extraction characterization parameter is greater than a reference threshold, setting the label of the image data to a detail feature label;

[0025] If the feature extraction characterization parameter is less than or equal to the reference threshold, the label of the image data is set to a non-detail feature label.

[0026] Furthermore, it is determined whether to adjust the number of matrix grids based on the label, wherein:

[0027] If the label is a detail feature label, the number of matrix grids is positively correlated with the feature extraction characterization parameter;

[0028] If the label is a non-detail feature label, the number of matrix grids remains unchanged.

[0029] Furthermore, the composite loss function is expressed by formula (1):

[0030] L all =min G max D L WGAN +α1L1+α2L GD +α3L IC (1)

[0031] In formula (1), G represents the generator, D represents the discriminator, and L WAGN Denotes the adversarial loss to measure the difference between the generated data and the real data distribution through the Wasserstein distance, L1 denotes the absolute difference between the generated image and the clear reference image, α1 denotes the weight coefficient of the absolute difference between the generated image and the clear reference image, L GD represents the gradient difference loss, α2 represents the weight coefficient of the gradient difference loss, L IC The image content loss measures the similarity between the generated image and the clear reference image in high-level semantic features through the pre-trained model of the VGG19 network. α3 represents the weight coefficient of the image content loss that measures the similarity between the generated image and the clear reference image in high-level semantic features through the pre-trained model of the VGG19 network.

[0032] Furthermore, the adversarial loss measures the difference between the generated data and the real data distribution through the Wasserstein distance, which is expressed by formula (2):

[0033]

[0034] In formula (2), E represents the mathematical expectation, IC denotes an undistorted underwater image, I D denotes a distorted deformation of the underwater image, denotes a sample along a straight line between a point of the original data and a point of the generator distribution, denotes an input content, a GP denotes a weight coefficient of the gradient penalty term, denotes is sampled from , denotes the gradient of the discriminator with respect to the input denotes the local change rate at the point.

[0035] Further, the absolute difference between the generated image and the real image is denoted by formula (3),

[0036] L1=E[||I C -(G(I D ))||1] (3)

[0037] In formula (3), G(I D ) denotes a predicted image generated by the generator based on the distorted deformation of the underwater image.

[0038] Further, the gradient difference loss is denoted by formula (4),

[0039]

[0040] In formula (4), I P is a predicted image, I P =G(I D ), is a loss sensitivity coefficient, where and respectively denote the pixel values of the predicted image and the real image at position (i,j), i,j∈[0,255].

[0041] Further, the image content loss, which measures the similarity between the generated image and the real image in high-level semantic features through a pre-trained model of the VGG19 network, is denoted by formula (5),

[0042]

[0043] In formula (5), Φ(·) denotes the features of the image extracted using the VGG19 network.

[0044] Compared with the prior art, the underwater image data is acquired to construct an underwater image data set, the explicit feature information of the image data is analyzed and the feature extraction representation parameter is calculated to set a label for the image data, an underwater image enhancement network is constructed, the generator generates an output image after receiving the image data, the discriminator obtains the output image of the generator, judges whether to adjust the number of matrix grids based on the label to output a matrix corresponding to the number of matrix grids, judges the local area of the output image according to the judgment score of the matrix, determines the construction state of the underwater image enhancement network, determines whether the output image needs to be generated again by the generator, and the underwater image enhancement network is trained by using the underwater image of the training set through a composite loss function until convergence. The underwater image enhancement network is constructed by using the generator and the discriminator, the degraded underwater image is enhanced, and the image quality and the restoration degree of underwater details are improved.

[0045] Especially, the feature extraction representation parameter is calculated based on the explicit features to set a label for the image data. In actual conditions, when the underwater image is enhanced, there may be an underwater image with rich detail features. At this time, if the preset number of matrix grids is still used, the enhanced underwater image may still be unclear. Therefore, the number of difference pixels and the contour density in the image data are analyzed as two relatively explicit features to calculate the feature extraction representation parameter to represent the prominence of the detail features in the image data, to provide data support for the matrix generated by the discriminator. For image data with rich detail features, the number of matrix grids is adaptively adjusted to ensure the perception of local details. For image data with few detail features, the baseline matrix grid is maintained to save computing power.

[0046] Especially, the underwater image enhancement network includes a generator for supplementing features of the degraded underwater image and delivering the image with supplemented features to a discriminator. In actual conditions, when underwater shooting is performed, shooting may not be clear. Therefore, the generator adopts a U-Net hybrid architecture based on wavelet transform and attention mechanism fusion, uses a five-layer U-shaped network to extract feature information of the input image and smoothly transition, uses a skip connection between two layers with the same feature map to ensure cross-layer information transmission, and particularly introduces wavelet transform in the generator to decompose the first-level output feature into frequency domain information, fuse the frequency domain information with the last level, supplement the feature set, further enhance the details and global information, and improve the image quality and the restoration degree of underwater details.

[0047] Especially, the underwater image enhancement network further comprises a discriminator, which determines an output matrix by receiving an output image of the generator combined with a label, so as to judge the authenticity of the image according to the judgment score of the matrix. In actual cases, the discriminator is determined based on the adversarial training mechanism in the GAN theory. However, the training process of the adversarial training mechanism is relatively complex, and the GAN tends to be unstable. Therefore, the PatchGAN architecture based on the Markov model is considered and the RRC (Residual Recurrent Convolution) module is embedded to construct the discriminator. Since the discrimination process of the PatchGAN is based on a local image block, the complexity of the calculation can be reduced. The RRC module can improve the discrimination accuracy of the image and improve the image quality and the restoration degree of underwater details.

[0048] Especially, the underwater image enhancement network is calculated for convergence based on a composite loss function to ensure the availability of the underwater image enhancement network. In actual cases, the underwater image enhancement network is mostly calculated for convergence based on a function, which has limitations and one-sidedness. Therefore, the composite loss function for the convergence calculation of the underwater image enhancement network is considered to make the underwater image enhancement network more accurate through multiple calculations, so as to improve the image quality and the restoration degree of underwater details in subsequent image enhancement in reality. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 Method steps of the underwater image enhancement method with wavelet transform and fusion attention mechanism of the embodiment of the application;

[0050] Figure 2 Flowchart for setting labels for image data of the embodiment of the application;

[0051] Figure 3 Generator structure diagram of the underwater image enhancement network with fusion attention mechanism of the embodiment of the application;

[0052] Figure 4 Wavelet transform module embedded in the up-sampling of the generator part of the embodiment of the application;

[0053] Figure 5 Discriminator structure diagram of the underwater image enhancement network based on the PatchGAN architecture of the Markov model of the embodiment of the application. DETAILED DESCRIPTION

[0054] In order to make the objects and advantages of the present application clearer, the present application will be further described below with examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0055] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art will understand that these embodiments are only used to explain the technical principles of the present application, and are not intended to limit the protection scope of the present application.

[0056] For the convenience of reading this scheme, some nouns related to this scheme will be explained below:

[0057] Degraded underwater image: an image taken in an underwater environment, due to the absorption and scattering of light by water, resulting in color distortion, reduced contrast, blurred image, etc. It can be understood that the image is partially degraded, and the greater the degree of degradation, the lower the image recovery quality.

[0058] Wavelet transform: a mathematical method that analyzes signals by decomposing them into components of different scales (or frequencies). In practice, it is used to analyze the frequency domain of input images, thereby separating information of different frequencies to better reconstruct and enhance the visual quality of underwater images.

[0059] Fusion attention mechanism: a widely used technique in deep learning that combines different types of attention mechanisms to improve model performance.

[0060] Please refer to Figures 1 to 5 as shown, Figure 1 the method steps of the underwater image enhancement method with wavelet transform and fusion attention mechanism of the invention embodiment, Figure 2 the flowchart for setting labels for image data of the invention embodiment, Figure 3 the generator structure diagram of the underwater image enhancement network with fusion attention mechanism of the invention embodiment, Figure 4 the wavelet transform module embedded in the up-sampling of the generator part of the invention embodiment, Figure 5 the discriminator structure diagram of the underwater image enhancement network based on the PatchGAN architecture of the Markov model of the invention embodiment. The underwater image enhancement method with wavelet transform and fusion attention mechanism of the present application comprises:

[0061] Step S1, obtain underwater image data to construct an underwater image dataset, including degraded underwater images and their corresponding clear reference images, and divide the dataset into test set, training set and validation set according to the proportion;

[0062] Step S2, extract image data from the test set, analyze the explicit feature information of the image data, calculate feature extraction representation parameters based on the explicit features, and set labels for the image data;

[0063] The underwater image enhancement network is constructed, the generator receives image data to generate an output image, the discriminator obtains the output image of the generator, traces a label corresponding to the output image, determines whether to adjust the number of matrix grids based on the label, and outputs a matrix corresponding to the number of matrix grids through a plurality of convolution operations by combining the output image with a clear reference image;

[0064] Step S3, determining the local area of the output image according to the judgment score of the matrix, determining the construction state of the underwater image enhancement network, and determining whether the output image needs to be generated again by the generator;

[0065] Step S4, training the underwater image enhancement network by using the underwater image of the training set through a composite loss function until convergence is achieved;

[0066] The explicit feature information includes the number of differential pixel points and the contour density in the image data, and the elements of the matrix include the judgment scores of the local areas of the output image.

[0067] Specifically, the proportion of the proportionally divided data set is not limited, as long as the network training can be ensured according to the data set, preferably, according to the experience rule, for the case of large data amount, the proportion of 70% training set, 15% verification set and 15% test set can be allocated, for the case of small data amount or the model needs more verification data to adjust the parameters, the proportion of 60% training set, 20% verification set and 20% test set can be allocated, and the person skilled in the art can also determine according to the actual situation, which will not be repeated here.

[0068] Specifically, the acquisition method of the high-quality clear reference image is not limited, for example, the image can be taken in an underwater non-interference environment, which will not be repeated here.

[0069] Specifically, in implementation, the initial size of the image data received by the generator is 256x256x3, where 3 represents the number of RGB channels, and the initial feature extraction in the image data is completed using a convolutional layer, the size of the convolution kernel is 3x3, and the step is set to 1 to ensure sufficient capture of detailed information; the number of channels of the output feature map is set to 64, and the spatial size is maintained at 256x256, then the feature map is input into the EMA attention mechanism, and the output feature map is down-sampled through a max-pooling operation; the kernel size of the max-pooling is 2x2, and the step is set to 2, so the feature map size is reduced from 256x256x64 to 128x128x64; the convolution kernel size of the second layer of convolution is 3x3, and the output channel number is 128, resulting in a feature map size of 128x128x128; a direction-sensitive enhancement mechanism is introduced in the convolution operation through the detail enhancement module, thereby extracting more detailed information, and the enhanced feature map is input into the EMA attention mechanism, the third and fourth layers of the network operate identically to the second layer, the feature map size of the third layer is 64x64x256, and the feature map size of the fourth layer is 32x32x512; the fifth level of the network uses pure convolution, and the output feature map size is 16x16x1024.

[0070] Specifically, the numerical values represented by the scores of the matrix are not fixed values, but are dynamically adjusted through the gradient descent algorithm in the training process. These values represent the discriminator's judgment score for whether each local region (i.e., the perception region) in the image is real or fake. These scores are continuous and can change during the training process to better distinguish between real and generated images.

[0071] Specifically, the output of the discriminator is a matrix with the same size as the input image, and the judgment score of each element is usually between 0 and 1, with 0 representing completely fake and 1 representing completely real.

[0072] Specifically, the generator can decompose the first-level output feature into frequency domain information through wavelet transform and fuse it with the last-level output feature.

[0073] Specifically, the underwater image enhancement network comprises a generator for feature supplementing of the degraded underwater image, and delivering the image after feature supplementing to a discriminator. In actual underwater shooting, there are often unclear shooting situations. Based on this, the generator based on the U-Net hybrid architecture of wavelet transform and attention mechanism fusion adopts a five-layer U-shaped network to extract feature information of the input image and smooth transition, and uses a skip connection between two layers with the same feature map to ensure cross-layer information transmission. The wavelet transform is particularly introduced in the generator, the first level output feature is decomposed into frequency domain information, fused with the last level, the feature set is supplemented, the details and global information are further enhanced, and the image quality and underwater detail restoration degree are improved.

[0074] Specifically, the five-layer U-shaped network refers to a U-Net network structure with five levels of encoders (down-sampling path) and five levels of decoders (up-sampling path), and the functions of each layer are as follows,

[0075] The first layer is composed of two branches, the first branch decomposes the original degraded underwater image into frequency domain features through wavelet transform, the frequency domain features include low frequency features and a group of high frequency features composed of horizontal, vertical and diagonal lines, and the frequency domain feature information processed by wavelet transform is combined with the feature information of the last level of the network through fusion operation, to further supplement the details and richness of the generated image;

[0076] The other branch of the first layer is composed of a convolution block, an efficient multi-scale attention module (EMA) and a down-sampling module, the convolution block includes a 3*3 convolution operation, a batch normalization operation and a ReLU activation function, which ensures efficient feature extraction and nonlinear mapping, the number of convolution kernels of this branch is 64, and the down-sampling module adopts a max-pooling operation to reduce the size of the feature map, thereby gradually reducing the spatial dimension;

[0077] In the networks of the second layer, the third layer and the fourth layer, a convolution block, a detail enhancement convolution block and an EMA attention mechanism are used, the number of convolution kernels of the convolution block is 128, 256 and 512 respectively, and the network is gradually deepened to extract higher level semantic information;

[0078] The number of convolution kernels of the fifth layer is increased to 1024 to capture more rich and complex features.

[0079] Specifically, the process of calculating feature extraction representation parameters based on explicit features includes,

[0080] Determine the difference pixel points based on the chroma difference of each pixel point in the image data, and record the number of difference pixel points;

[0081] Identify the contour in the image data, and calculate the contour density;

[0082] determining a ratio of the number of difference pixels to a reference number of difference pixels as a difference pixel influence factor;

[0083] determining a ratio of the contour density to a reference contour density as a contour density influence factor;

[0084] determining a weighted sum of the difference pixel influence factor and the contour density influence factor as a feature extraction representation parameter;

[0085] wherein the chrominance difference between the difference pixels is greater than a predetermined chrominance difference threshold.

[0086] Specifically, the reference number of difference pixels is obtained by prior calculation, a plurality of clear reference images are obtained in advance, an average number of difference pixels of the plurality of clear reference images is determined, and the average number of difference pixels is set as the reference number of difference pixels.

[0087] Specifically, the reference contour density is obtained by prior calculation, a plurality of clear reference images are obtained in advance, an average contour density of the plurality of clear reference images is determined, and the average contour density is set as the reference contour density.

[0088] Specifically, the weight coefficient of the difference pixel influence factor is 0.54, and the weight coefficient of the contour density influence factor is 0.46.

[0089] Specifically, the predetermined chrominance difference threshold is obtained by prior calculation, a plurality of clear reference images are obtained in advance, an average chrominance difference of the plurality of clear reference images is determined, and 0.65 times of the average chrominance difference is set as the chrominance difference threshold.

[0090] Specifically, the feature extraction representation parameter is calculated based on the explicit features, and the image data is labeled. In actual situations, when enhancing underwater images, there may be underwater images with rich detail features. At this time, if the preset number of matrix grids is still used, the enhanced underwater images may still be unclear. Therefore, by analyzing the number of difference pixels and the contour density, two relatively explicit features in the image data, the feature extraction representation parameter is calculated to represent the prominence of the detail features in the image data, to provide data support for the subsequent discriminator to generate a matrix. For image data with rich detail features, the matrix grid is adaptively adjusted to ensure the perception of local details. For image data with few detail features, the reference matrix grid is maintained to save computing power.

[0091] Specifically, the image data is labeled, and wherein

[0092] if the feature extraction representation parameter is greater than a reference threshold, the label of the image data is set as a detail feature label;

[0093] If the feature extraction representation parameter is less than or equal to a reference threshold, set the label of the image data as a non-detail feature label.

[0094] Specifically, the reference threshold is selected in the interval (1.25, 1.5).

[0095] Specifically, the label determines whether to adjust the number of matrix grids, wherein

[0096] If the label is a detail feature label, the number of matrix grids is positively correlated with the feature extraction representation parameter;

[0097] If the label is a non-detail feature label, the number of matrix grids remains unchanged.

[0098] It can be understood that the larger the feature extraction representation parameter is, the richer the underwater image content is, and the more matrix grids are required.

[0099] In implementation, optionally,

[0100] If the reference threshold belongs to the interval [1.5, +∞], the grid number is selected as 64x64;

[0101] If the reference threshold belongs to the interval (1.25, 1.5), the grid number is selected as 32x32;

[0102] If the reference threshold belongs to the interval [0, 1.25], the grid number is selected as 16x16.

[0103] Specifically, the process of outputting a matrix through multiple convolution operations includes,

[0104] Determine the image data of the generated output image and the corresponding data features;

[0105] Determine the dot product operation result of the convolution kernel and each image data region of the output image as a single matrix;

[0106] Stack each single matrix to form a feature matrix;

[0107] Determine the feature matrix as an output matrix;

[0108] Wherein, the convolution kernel is a single matrix, used to slide on the image data to extract features.

[0109] Specifically, the step of calculating the dot product of the convolution kernel and each image data region of the output image is as follows,

[0110] Step S01, determine the convolution kernel and each image data region, the convolution kernel is the same size as each image data region, it can be understood that the size of the image data region is determined by the number of matrix grids, the more the number of matrix grids, the more image data regions need to be divided, therefore, the more the determined regions, the more accurate the determination result, suitable for output images with more features;

[0111] Step S02, multiply each element of the convolution kernel with the element at the corresponding position in the underwater image data region;

[0112] Step S03, add all the products obtained in step S02 to obtain a single numerical value;

[0113] Step S04, place the obtained single numerical value at the corresponding position of the output feature map, slide the convolution kernel one pixel to the right, and repeat the above process until the entire image region is covered;

[0114] Step S05, determine that the matrix composed of the feature data corresponding to the entire image region covered is a feature matrix.

[0115] Specifically, the process of determining the construction state of the underwater image enhancement network includes,

[0116] determine the judgment score of each matrix element;

[0117] determine that the authenticity of the region image of the matrix element whose judgment score is greater than or equal to the judgment score threshold is true;

[0118] determine the ratio of the number of region images with true authenticity to the total number of region images in the image;

[0119] determine that the authenticity of the image is true when the ratio is greater than or equal to the reference ratio;

[0120] If the authenticity is true, the construction state of the underwater image enhancement network is completely constructed, and the output image does not need to be generated again by the generator,

[0121] or, the output image needs to be generated again by the generator;

[0122] wherein each matrix element represents the judgment score of a perception region in the image output by the generator.

[0123] Specifically, the judgment score threshold is calculated in advance, a plurality of clear reference images are obtained in advance, the average value of each element in each clear reference image is calculated, and the average value of each element in each clear reference image is determined as the judgment score threshold.

[0124] Specifically, the benchmark ratio is obtained by pre-computing, obtaining a plurality of high-quality clear reference images in advance, and determining the number of each reference region image in the plurality of high-quality clear reference images as true, and determining the benchmark ratio as 0.86 times of the ratio of the number of each reference region image determined as true to the total number of reference region images in the plurality of high-quality clear reference images.

[0125] Specifically, the underwater image enhancement network further comprises a discriminator, which determines the output matrix by receiving the output image of the generator and combining the clear reference image, so as to determine the authenticity of the image according to the judgment score of the matrix. In actual cases, the discriminator is determined based on the adversarial training mechanism in the GAN theory. However, the training process of the adversarial training mechanism is relatively complex, and the GAN tends to be unstable. Therefore, a PatchGAN architecture based on the Markov model is considered and a recurrent residual convolution (RRC) module is embedded to construct the discriminator. Since the discrimination process of the PatchGAN is based on a local image block, the complexity of the calculation can be reduced. The recurrent residual convolution (RRC) module can improve the discrimination accuracy of the image, and at the same time improve the image quality and the restoration degree of underwater details.

[0126] Specifically, the composite loss function is represented by formula (1),

[0127] L all =min G max D L WGAN +α1L1+α2L GD +α3L IC (1)

[0128] In formula (1), G represents the generator, D represents the discriminator, L WAGN represents the adversarial loss, which measures the difference between the generated data and the real data distribution by the Wasserstein distance, L1 represents the absolute difference between the generated image and the clear reference image, α1 represents the weight coefficient of the absolute difference between the generated image and the clear reference image, L GD represents the gradient difference loss, α2 represents the weight coefficient of the gradient difference loss, L IC represents the image content loss, which measures the similarity between the generated image and the clear reference image in the high-level semantic feature by the pre-trained model of the VGG19 network, and α3 represents the weight coefficient of the image content loss, which measures the similarity between the generated image and the clear reference image in the high-level semantic feature by the pre-trained model of the VGG19 network.

[0129] Specifically, in the implementation, the value of α1 is 100, the value of α2 is 10, and the value of α3 is 3. It can be understood that the values of α1, α2 and α3 are the weight coefficients of the experience adjustment to the hyperparameters.

[0130] Specifically, the underwater image enhancement network is calculated based on the composite loss function to ensure the availability of the underwater image enhancement network. In actual situations, the underwater image enhancement network is mostly calculated based on a function, which has limitations and one-sidedness. Therefore, the composite loss function for the underwater image enhancement network is constructed, and the underwater image enhancement network is calculated multiple times to make it more accurate, so as to improve the image quality and the restoration degree of underwater details in subsequent image enhancement in reality.

[0131] Specifically, the adversarial loss is measured by the Wasserstein distance between the generated data and the real data distribution, which is represented by formula (2).

[0132]

[0133] In formula (2), E represents the mathematical expectation, I C represents the underwater image without distortion, I D represents the distortion deformation of the underwater image, represents the sample along the straight line between the point pair of the original data and the generator distribution, represents the input content, and a GP represents the weight coefficient of the gradient penalty term, represents is sampled from , represents the gradient of the discriminator to the input , and represents the local change rate at the point.

[0134] Specifically, the absolute difference between the generated image and the real image is represented by formula (3).

[0135] L1=E[||I C -(G(I D ))||1] (3)

[0136] In formula (3), G(I D ) represents the predicted image generated by the generator based on the distortion deformation of the underwater image.

[0137] Further, the gradient difference loss is represented by formula (4).

[0138]

[0139] In formula (4), I P is the predicted image, I P =G(I D ), is the loss sensitivity coefficient, wherein and respectively represent the pixel value of the predicted image and the real image at position (i, j), i, j ∈ [0, 255].

[0140] Specifically, the main role of the loss sensitivity coefficient is to control the weight or nonlinearity degree of the generated image gradient error, which can achieve different loss optimization goals, so as to better balance the edge details and global consistency of the generated image.

[0141] Specifically, the image content loss measures the similarity between the generated image and the real image in the high-level semantic features through the pre-trained model of the VGG19 network, which is represented by formula (5),

[0142]

[0143] In formula (5), Φ(·) represents the feature of the image extracted using the VGG19 network.

[0144] It can be understood that the process of calculating the expectation E, summing and extracting the image features are all carried out in the computer, and the person skilled in the art only needs to ensure the correct input of data, which will not be repeated here.

[0145] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but the person skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. The person skilled in the art can make equivalent changes or replacements to the related technical features without deviating from the principles of the present application, and the technical solutions after these changes or replacements will all fall within the protection scope of the present application.

Claims

1. An underwater image enhancement method with wavelet transform and fusion attention mechanism, characterized in that: include: Acquire underwater image data to construct an underwater image dataset, including degraded underwater images and their corresponding clear reference images, and divide the dataset into a test set, a training set, and a validation set in proportion; Extracting image data from a test set, analyzing explicit feature information of the image data, and calculating feature extraction characterization parameters based on the explicit features to set labels for the image data; Construct an underwater image enhancement network. The generator generates an output image after receiving image data. The discriminator obtains the output image of the generator, traces the label corresponding to the output image, and determines whether to adjust the number of matrix grids based on the label. The output image is combined with a clear reference image through multiple sets of convolution operations to output a matrix corresponding to the number of matrix grids. Determining a local area of ​​the output image according to the determination score of the matrix, determining a construction state of the underwater image enhancement network, and determining whether the output image needs to be regenerated by the generator; Using underwater images from a training set, the underwater image enhancement network is trained using a composite loss function until convergence; The dominant feature information includes the number of difference pixels and contour density in the image data, and the elements of the matrix include judgment scores for several local areas of the output image; The composite loss function is expressed by formula (1), (1)， In formula (1), G represents the generator, D represents the discriminator, The adversarial loss measures the difference between the generated data and the real data distribution through the Wasserstein distance. represents the absolute difference between the generated image and the sharp reference image, The weight coefficient representing the absolute difference between the generated image and the sharp reference image, represents the gradient difference loss, represents the weight coefficient of gradient difference loss, The image content loss measures the similarity between the generated image and the clear reference image in terms of high-level semantic features through the pre-trained model of the VGG19 network. The image content loss is a weight coefficient that measures the similarity between the generated image and the clear reference image in high-level semantic features through the pre-trained model of the VGG19 network; The adversarial loss measures the difference between the generated data and the real data distribution through the Wasserstein distance, which is expressed by formula (2). (2)， In formula (2), E represents the mathematical expectation, , represents the undistorted underwater image, represents the distortion of underwater images, represents the samples along the straight line between the original data and the point pairs distributed by the generator, Indicates the input content. represents the weight coefficient of the gradient penalty term, express It is from The sample obtained from Representation Discriminator Input The gradient represents the local rate of change at that point; The absolute difference between the generated image and the real image is expressed by formula (3), (3)， In formula (3), represents the predicted image generated by the generator based on the distortion of the underwater image; The gradient difference loss is expressed by formula (4), (4)， In formula (4), To predict the image, , is the loss sensitivity coefficient, where , and Represents the predicted image and the real image at position The pixel value of ; The image content loss is expressed by formula (5) to measure the similarity between the generated image and the real image in high-level semantic features through the pre-trained model of the VGG19 network. （5）， In formula (5), Indicates that the features of the image are extracted using the VGG19 network.

2. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 1, characterized in that: The generator can decompose the first-level output features into frequency domain information through wavelet transform and fuse them with the last-level output features.

3. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 2, characterized in that: The process of calculating feature extraction characterization parameters based on explicit features includes: Determine difference pixels based on the chromaticity difference of each pixel in the image data, and record the number of difference pixels; Identify contours in image data and calculate contour density; Determine the ratio of the number of difference pixels to the number of reference difference pixels as a difference pixel influencing factor; Determining a ratio of the contour density to a reference contour density as a contour density influencing factor; Determine the weighted sum of the difference pixel influence factor and the contour density influence factor as the feature extraction characterization parameter; The chromaticity difference between the difference pixels must be greater than a predetermined chromaticity difference threshold.

4. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 1, characterized in that: The label is set for the image data, wherein: If the feature extraction characterization parameter is greater than a reference threshold, setting the label of the image data to a detail feature label; If the feature extraction characterization parameter is less than or equal to the reference threshold, the label of the image data is set to a non-detail feature label.

5. The underwater image enhancement method with wavelet transform and fusion attention mechanism according to claim 4, characterized in that: Determine whether to adjust the number of matrix grids based on the label, wherein: If the label is a detail feature label, the number of matrix grids is positively correlated with the feature extraction characterization parameter; If the label is a non-detail feature label, the number of matrix grids remains unchanged.

Images