An underwater image enhancement method based on multi-scale attention hybrid feature fusion

Abstract

The present invention discloses an underwater image enhancement method based on multi-scale attention hybrid feature fusion, which specifically includes the following steps: establishing an underwater image data set; performing data preprocessing on the data set; constructing an underwater image enhancement model based on multi-scale attention hybrid feature fusion; improving the loss function; training the model using a training set and a validation set; testing the model using a test set to meet the objective evaluation index requirements of image quality and obtain the final underwater image enhancement model. Compared with the prior art, the underwater image enhancement method based on multi-scale attention hybrid feature fusion disclosed by the present invention effectively solves the problems of underwater image detail loss, local fogging and image color distortion.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to an underwater image enhancement method based on multi-scale attention hybrid feature fusion. Background Art

[0002] The ocean is the largest ecosystem on Earth, and has a profound impact on global climate, biodiversity, and human activities. With the advancement of science and technology, ocean exploration activities are increasing, and underwater images have become an important means of obtaining ocean information. However, the particularity of the underwater environment, such as the absorption and scattering of light, and the dynamic changes of the water body, seriously affects the quality of underwater images, which poses a challenge to applications in scientific research, resource development, and environmental protection.

[0003] Traditional underwater image enhancement is divided into two categories. One is based on physical models to introduce visual priors to estimate underwater; the other is based on statistical methods, such as white balance algorithm, histogram stretching method, etc. When processing underwater images, traditional methods can often only handle specific categories of problems and cannot comprehensively deal with underwater fogging, color cast, and low brightness problems. With the rapid development of deep learning, deep learning models have been widely used in the field of image enhancement. The deep learning-based method uses the powerful learning ability of neural networks to build a strong deep learning model, which can automatically learn features from data, avoiding the tediousness and inaccuracy of manual feature extraction by traditional methods, such as UNET, UWGAN, DeepWaveNet, CastGAN and other underwater image enhancement deep learning models. However, the image enhancement model based on deep learning still has problems with underwater image detail loss, local fogging and image color distortion. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides an underwater image enhancement method based on multi-scale attention hybrid feature fusion. Through multi-scale feature extraction and attention mechanism, and multiple up- and down-sampling fusion methods, the underwater image enhancement model based on deep learning solves the problems of image detail loss, local fogging and image color distortion, and improves the overall quality of underwater images.

[0005] The technical solution provided by the present invention comprises the following steps:

[0006] Step 1: Acquire an underwater image data set to form a first data set; in the first data set, the underwater images can be taken by a professional underwater camera or collected from the Internet;

[0007] Step 2: preprocessing the images in the first data set to form a second data set, and dividing the second data set into a training set, a validation set, and a test set;

[0008] Step 3: Construct an underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion. The MAHFF model includes 6 MAH-FEB modules, 6 RESIZSE BLOCK modules, 1 Concatenation fusion module, and 1 Conv convolution module;

[0009] The MAHFF model takes the second data set as input;

[0010] The input image is input to the first MAH-FEB module, and the feature map output by the first MAH-FEB module is used as the input of the second MAH-FEB module; at the same time, the input image is input to the first RESIZE BLOCK module, and the output of the first RESIZE BLOCK module is sent to the Concatenation fusion module;

[0011] The feature map output by the first MAH-FEB module is used as the input of the second RESIZE BLOCK module; the output of the second RESIZE BLOCK module is sent to the Concatenation fusion module; and so on, the above process is repeated five times, the output of the fifth MAH-FEB module is sent to the sixth RESIZE BLOCK module, and the output of the sixth RESIZE BLOCK module is sent to the Concatenation fusion module; the output of the sixth MAH-FEB module is sent to the Concatenation fusion module;

[0012] The output of the Concatenation fusion module is sent to the Conv convolution module, and the output of the Conv convolution module is the final enhanced image;

[0013] Furthermore, the MAH-FEB module in step 3 specifically includes:

[0014] 4 3×3Conv convolution modules, 1 5×5Conv convolution module, 1 7×7Conv convolution module, 3 GAM attention mechanism modules, 5 BN batch normalization modules, 5 LeakyRelu activation function modules, 1 Resize module, 1 Maxpooling maximum pooling module, and 2 Concatenation fusion modules;

[0015] The MAH-FEB module, the internal process of the module further includes steps 3.1 to 3.11:

[0016] Step 3.1: Resize the original input image as a feature map;

[0017] Step 3.2: Take the feature map as input and send it to the RESIZE BLOCK module, 3×3, 5×5 and 7×7 convolution kernels for convolution operations to extract feature information of different scales and levels. The convolution formula is as follows:

[0018]

[0019] In formula (1), X out is the feature map after convolution, X in is the input feature map, K is the convolution kernel, [i, j] is the pixel position coordinate, and the feature map X in Multiply with the elements of the convolution kernel K and sum them. m and n are the indices used to traverse all elements of the convolution kernel K.

[0020] Step 3.3: Perform batch normalization on the output of each convolution branch to reduce the internal covariate shift during training. The batch normalization formula is as follows:

[0021]

[0022] In formula (2), μ is X out The mean value, σ 2 Yes X out The variance of ,∈ is a constant to prevent division by zero;

[0023] Step 3.4: After normalizing each convolution branch, use the LeakyRelu activation function to perform nonlinear transformation on the normalized features to avoid gradient disappearance. The calculation formula of the LeakyRelu activation function is as follows:

[0024]

[0025] In formula (3), X norm is the feature map after instance normalization, β is the parameter of the LeakyRelu function, indicating that X norm When it is less than 0, the slope of the function;

[0026] Step 3.5: Send the feature maps output by the three convolution branches of 3×3, 5×5, and 7×7 to the global attention mechanism GAM module respectively to output more detailed feature maps. The global attention mechanism formula is as follows:

[0027] X output =M c ×M s #(4)

[0028] In formula (4), X output is the output feature map, M c is the channel attention mechanism, M sIt is the spatial attention mechanism;

[0029] Among them, the channel attention mechanism M c The formula is:

[0030] M c =σ(FC(Relu(FC(X g ))))#(5)

[0031] In formula (5), X g is the feature map after global pooling, FC is the fully connected layer, and σ is the Sigmoid activation function;

[0032] Spatial Attention Mechanism M s The formula is:

[0033] M s =σ(Conv(Relu(Conc(X))))#(6)

[0034] In formula (6), X is the input feature map, Conv is the convolutional layer, and σ is the Sigmoid activation function;

[0035] Step 3.6: The output feature maps of the three GAM layers are fused along the channel dimension and sent to the Concatenation module to obtain a multi-scale feature map.

[0036] Step 3.7: Perform dimensionality reduction on the concatenated feature map through a 3×3 convolution block;

[0037] Step 3.8: Use the maximum pooling operation to downsample the reduced feature map. The formula for the maximum pooling is as follows:

[0038]

[0039] In formula (7), y kij Indicates that the rectangle associated with the kth feature map tends to R ij The maximum pooling output value, x kpq Represents the rectangular area R ij The element at (p,q) in

[0040] Step 3.9: Send the feature map output from step 3.8 to the Concatenation fusion module;

[0041] Step 3.10: The feature map of the original image input described in step 3.2 is resized to the same size as the downsampled feature map through an upsampling module RESIZEBLOCK Figure 1 Then, it is fused with the output feature map of step 3.8 to achieve feature fusion of different levels and scales;

[0042] Step 3.11: Perform a convolution operation on the fused feature map through a 3×3 convolution block and output the final feature map with more detailed features.

[0043] Furthermore, the RESIZE BLOCK module in step 3 includes a 3×3 Conv layer and a Resize layer; the Resize layer adjusts the size of the input feature map by upsampling or downsampling to achieve feature extraction at different scales, thereby capturing the global and local information of the image; the Conv layer further extracts spatial features from the feature map output by the Resize layer through a 3×3 convolution operation, thereby enhancing the expressiveness of the features;

[0044] Step 4: Use a new loss function. The loss function calculation formula is:

[0045] L newloss =(1-α)*L SSIM +α*L1#(8)

[0046] In formula (8), L new loss is the new loss function, α is the weight factor, α∈(0,1), and the L1 calculation formula is:

[0047]

[0048] In formula (9), N is the total number of samples in each training batch, x(p) is the original image, and y(p) is the enhanced image;

[0049] L SSIM The calculation formula is:

[0050] L SSIM (x,y)=L(x,y)×C(x,y)×S(x,y)#(10)

[0051]

[0052] c1=(k1l) 2 #(14)

[0053] c2=(k2l) 2 #(15)

[0054]

[0055] In formula (10), L(x, y) is brightness, C(x, y) is contrast, S(x, y) is structure, x represents the original image, y represents the enhanced image, μ x and μ y is expressed as the mean of x and y, and Expressed as the variance of x and y, σ xy It is expressed as the covariance between x and y, c1, c2, c3 are very small constants used to avoid the denominator being 0, k1 is 0.01, k2 is 0.03, and l is the image pixel value range, that is, l∈(0,255);

[0056] Step 5: Use the training set and the validation set to train the underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion, and save the model with the lowest MSE loss after the training as the optimal model; the MAHFF model training process further includes steps 5.1 to 5.4;

[0057] Step 5.1: Set the training parameters of the MAHFF model, including: batch size, number of training cycles, learning rate, optimizer, input image size;

[0058] Step 5.2: Input the training set and validation set images and corresponding labels into the MAHFF model, use the optimizer Adam to back-propagate and update the parameters, and then start training;

[0059] Step 5.3: Save the model after training as the optimal model;

[0060] Step 6: Use the test set to test the optimal model, and evaluate the test set test results by objective evaluation indicators to meet the objective evaluation indicator requirements, that is, to obtain the final underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion; the evaluation process further includes 6.1 to 6.3;

[0061] Step 6.1: Input the test set into the optimal model described in step 5, and output the enhanced test set image;

[0062] Step 6.2: Calculate the objective evaluation indicators of the original test set images and the enhanced test set images: peak signal-to-noise ratio (PNSR), structural similarity (SSIM) and visual information fidelity (VIF);

[0063] Step 6.3: When the objective evaluation indicators meet the requirements, the final underwater image enhancement model based on multi-scale attention hybrid feature fusion is obtained.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The present invention discloses an underwater image enhancement method based on multi-scale attention hybrid feature fusion. The method first extracts the features of underwater images at multiple scales, introduces the GAM attention mechanism to enhance the global interaction information between features, uses a multi-level RESIZE BLOCK module to adjust the image size information, increases the depth and expression ability of the network, and solves the problems of image detail loss, local fogging and image color distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of the underwater image enhancement method based on multi-scale attention hybrid feature fusion of the present invention;

[0067] Figure 2 It is a schematic diagram of the structure of the underwater image enhancement model based on multi-scale attention hybrid feature fusion of the present invention;

[0068] Figure 3 This is a schematic diagram of the structure of the MAH-FEB module of the present invention;

[0069] Figure 4 It is a schematic diagram of the structure of the RESIZE BLOCK module of the present invention;

[0070] Figure 5 This is the VIF flow chart of the evaluation index of the present invention DETAILED DESCRIPTION

[0071] In order to make the technical solution, structural features, achieved purposes and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and accompanying drawings. It should be noted that the specific embodiments described herein are only used to more clearly explain the present invention and are not used to limit the present invention.

[0072] Figure 1 This is a flow chart of an underwater image enhancement method based on multi-scale attention hybrid feature fusion disclosed by the present invention, and its implementation process is as follows:

[0073] Step 1: Establish an underwater image dataset to form a first dataset;

[0074] In the first data set, underwater images can be taken by a professional underwater camera and collected from the Internet;

[0075] In this embodiment, in order to better evaluate the enhancement effect of the underwater image enhancement method based on multi-scale attention hybrid feature fusion disclosed by the present invention, in addition to the underwater pictures collected by the user, a public data set UIEB (Underwater Image Enhancement Benchmark, UIEB) underwater image enhancement data set is also used to form a first data set;

[0076] Step 2: Preprocess the first data set to form a second data set, and divide the second data set into a training set, a validation set, and a test set.

[0077] Matlab software was used to preprocess the collected images, including using the imresize function to modify the image size, using the imreducehaze function to remove haze, using the imadjust function to increase the contrast, using the imadjust function to increase the brightness, and using the imsharpen function to perform color correction;

[0078] Since the public data set UIEB used in this embodiment has completed preprocessing, this step is skipped in this embodiment to form a second data set, and the second data set is divided into a training set, a validation set and a test set according to 8:1:1; the divided training set includes 960 pictures, the test set includes 120 pictures, and the validation set includes 120 pictures. In this embodiment, the image size of the second data set is 272×480;

[0079] Step 3: Construct an underwater image enhancement model based on multi-scale attention hybrid feature fusion;

[0080] The model structure of the underwater image enhancement model based on multi-scale attention hybrid feature fusion is as follows Figure 2 shown.

[0081] The MAHFF enhanced model takes the second data set as input;

[0082] The input image is input to the first MAH-FEB module, and the feature map output by the first MAH-FEB module is used as the input of the second MAH-FEB module; at the same time, the input image is input to the first RESIZE BLOCK module, and the output of the first RESIZE BLOCK module is sent to the Concatenation fusion module;

[0083] The feature map output by the first MAH-FEB module is used as the input of the second RESIZE BLOCK module; the output of the second RESIZE BLOCK module is sent to the Concatenation fusion module; and so on, the above process is repeated five times, the output of the fifth MAH-FEB module is sent to the sixth RESIZE BLOCK module, and the output of the sixth RESIZE BLOCK module is sent to the Concatenation fusion module; the output of the sixth MAH-FEB module is sent to the Concatenation fusion module;

[0084] The output of the Concatenation fusion module is sent to the Conv convolution module, and the output of the Conv convolution module is the final enhanced image;

[0085] like Figure 3 This is the internal structure of the MAH-FEB module constructed in step 3. This module needs to be called six times, each time with different inputs and outputs. The following takes the first call as an example.

[0086] The MAH-FEB module process further includes steps 3.1 to 3.11:

[0087] Step 3.1: The input image channel × image length × image width is 3 × 272 × 480, which is resized to image channel × image length × image width of 32 × 136 × 240 as the feature map.

[0088] Step 3.2: Input the feature map adjusted in step 3.1 into the convolution kernels of RESIZE BLOCK, 3×3, 5×5, and 7×7 for convolution operation to extract feature information of different scales and levels. The convolution formula is as follows:

[0089]

[0090] In formula (1), X out is the feature map after convolution, X in is the input feature map, K is the convolution kernel, [i, j] is the pixel position coordinate, and the feature map X in Multiply with the elements of the convolution kernel K and sum them. m and n are the indices used to traverse all elements of the convolution kernel K.

[0091] Step 3.3: Perform batch normalization (BN) on the output of each convolution branch to reduce internal covariate shift during training;

[0092] The batch normalization formula is as follows:

[0093]

[0094] In formula (2), μ is X out The mean value, σ 2 Yes X out The variance of ,∈ is a constant that prevents division by zero.

[0095] Step 3.4: Use the LeakyRelu activation function to perform nonlinear transformation on the normalized features to avoid gradient disappearance. The formula of LeakyRelu is as follows:

[0096] The activation function formula of LeakyRelu is as follows:

[0097]

[0098] In formula (3), X norm is the feature map after instance normalization, a is the parameter of the LeakyRelu function, indicating that X norm The slope of the function is less than 0;

[0099] Each convolution branch is divided into a feature map with a channel, image length, and image width of 32×136×240;

[0100] Step 3.5: Input the three feature maps with channel, image length and image width of 32×136×240 into their respective global attention mechanism GAM layers, and output them as feature maps of 32×136×240. The global attention mechanism formula is as follows:

[0101] X output =M c ×M s #(4)

[0102] In formula (4), X output is the output feature map, M c is the channel attention mechanism, M s It is the spatial attention mechanism;

[0103] Channel attention mechanism M c The formula is:

[0104] M c =σ(FC(Relu(FC(X g ))))#(5)

[0105] In formula (5), X g is the feature map after global pooling, FC is the fully connected layer, and σ is the Sigmoid activation function. In formula (5), X is firstly processed by a fully connected layer and a Relu activation function. g Transform, and then pass a fully connected layer and Sigmoid activation function to get the attention weight M of each channel c ;

[0106] Spatial Attention Mechanism M s The formula is:

[0107] M s =σ(Conv(Relu(Conv(X))))#(6)

[0108] In formula (6), X is the input feature map, Conv is the convolution layer, and σ is the Sigmoid activation function. In formula (6), X is first transformed through a convolution layer and a Relu activation function, and then through a convolution layer and a Sigmoid activation function to obtain the attention weight M of each position. s ;

[0109] Step 3.6: The feature maps output by the three GAM layers are fused along the channel dimension and sent to the Concatenation module to obtain a multi-scale feature map;

[0110] Step 3.7: Perform dimensionality reduction on the concatenated feature map through a 3×3 convolution kernel;

[0111] Step 3.8: Use the maximum pooling operation to downsample the reduced feature map. The maximum pooling formula is as follows:

[0112]

[0113] In formula (7), y kij Indicates that the rectangle associated with the kth feature map tends to R ij The maximum pooling output value, x kpq Represents the rectangular area R ij The element at (p,q) in

[0114] Step 3.9: Send the feature map output from step 3.8 to the Concatenation fusion module;

[0115] Step 3.10: The feature map of the original image input described in step 3.2 is resized to the same size as the downsampled feature map through an upsampling module RESIZEBLOCK Figure 1 Then, it is fused with the output feature map of step 3.8 to achieve feature fusion of different levels and scales;

[0116] Step 3.11: Perform a convolution operation on the fused feature map through a 3×3 convolution block and output the final feature map with more detailed features.

[0117] Here are the sizes and number of channels of the resized input and output feature maps for each call to the module:

[0118] The original input is 3×272×480;

[0119] First: downsampling, the input is adjusted to 32 channels, length and width are 32×136×240; the output is 32 channels, length and width are 68×120;

[0120] Second time: downsampling, the input is adjusted to 32 channels, length and width are 68×120; the output is 64 channels, length and width are 34×60;

[0121] The third time: downsampling, the input is adjusted to 64 channels, length and width are 34×60; the output is 128 channels, length and width are 17×30;

[0122] Fourth time: upsampling, the input is adjusted to 128 channels, length and width are 17×30; the output is 64 channels, length and width are 34×60;

[0123] Fifth: Upsampling, the input is adjusted to 64 channels, length and width are 34×60; the output is 32 channels, length and width are 68×120

[0124] Sixth: Upsampling, after adjustment, the input is 32 channels, the length and width are 68×120; the output is 20 channels, the length and width are 136×240

[0125] Figure 4 It is a schematic diagram of the structure of the RESIZE BLOCK module. In the MAHFF model, each RESIZE BLOCK module includes a Resize layer and a 3×3 Conv layer;

[0126] The function of the Resize layer is to adjust the size of the input feature map by upsampling or downsampling to achieve feature extraction at different scales, thereby capturing the global and local information of the image;

[0127] The function of the Conv layer is to further extract spatial features from the feature map output by the Resize layer through a 3×3 convolution operation, thereby enhancing the expressiveness of the features.

[0128] The following are the inputs and outputs of each RESIZE BLOCK module:

[0129] First time: input is 3 channels, length and width are 272×480; output is 32 channels, length and width are 136×240;

[0130] Second time: input is 32 channels, length and width are 272×480; output is 32 channels, length and width are 136×240;

[0131] The third time: the input is 64 channels, the length and width are 270×480; the output is 32 channels, the length and width are 136×240;

[0132] Fourth time: the input is 128 channels, the length and width are 270×480; the output is 32 channels, the length and width are 136×240;

[0133] Fifth time: input is 64 channels, length and width are 270×480; output is 32 channels, length and width are 136×240;

[0134] The sixth time: the input is 32 channels, the length and width are 270×480; the output is 32 channels, the length and width are 136×240;

[0135] Step 4: Improve the loss function; the loss function calculation formula is:

[0136] L newloss =(1-α)*L SSIM +a*L1#(8)

[0137] In formula (8), L new loss is the new loss function, α is the weight factor, α∈(0,1), and the L1 calculation formula is:

[0138]

[0139] In formula (9), N is the total number of samples in each training batch, x(p) is the original image, and y(p) is the enhanced image;

[0140] In formula (8), L SSIM The calculation formula is:

[0141] L SSIM (x,y)=L(x,y)×C(x,y)×S(x,y)#(10)

[0142]

[0143]

[0144] c1=(k1l) 2 #(14)

[0145] c2=(k2l) 2 #(15)

[0146]

[0147] In formula (10), L(x, y) is brightness, C(x, y) is contrast, S(x, y) is structure, x represents the original image, y represents the enhanced image, μ x and μ y is expressed as the mean of x and y, and Expressed as the variance of x and y, σ xyIt is expressed as the covariance between x and y, c1, c2, c3 are very small constants used to avoid the denominator being 0, k1 is 0.01, k2 is 0.03, and l is the image pixel value range, that is, l∈(0,255);

[0148] Step 5: Input the training set and validation set into the enhanced model for training;

[0149] The underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion is trained using the training set and the validation set, and the model after training is saved as the optimal model. The MAHFF model training process further includes steps 5.1 to 5.4;

[0150] Step 5.1: Setting the training parameters of the underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion. In this embodiment, the training parameters include: 16 batch size, 200 training cycles, 0.001 learning rate, Adam optimizer, 272×480 input image size;

[0151] Step 5.2: Input the training set and validation set images and corresponding labels into the underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion, and use the optimizer Adam to back-propagate and update the parameters; then start training;

[0152] Step 5.3: Save the trained model parameters as the optimal model;

[0153] Step 6: Input the test set into the enhancement model for testing to meet the objective evaluation index requirements of image quality and obtain the final model;

[0154] The optimal model is tested using the test set, and the test results of the test set are evaluated by objective evaluation indicators to meet the objective evaluation indicator requirements, that is, the final underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion is obtained; the evaluation process further includes steps 6.1 to 6.3:

[0155] Step 6.1: Input the test set into the optimal model described in step 5, and output the enhanced test set image;

[0156] Step 6.2: Calculate the objective evaluation indicators of the original test set images and the enhanced test set images: peak signal-to-noise ratio PNSR, structural similarity SSIM and visual information fidelity VIF. The specific calculation formula is as follows:

[0157] (1) Peak signal-to-noise ratio PNSR, the peak signal-to-noise ratio calculation formula is:

[0158]

[0159] In formula (16), MAX I is the maximum possible pixel value of the image, MSE is the average of the squares of the differences between the corresponding pixels of the two images, and the MSE calculation formula is as follows:

[0160]

[0161] In formula (17), MSE is the original image I and the enhanced image K of two m×n single channels.

[0162] (2) Structural similarity SSIM, the structural similarity calculation formula is:

[0163] SSIM(x,y)=L(x,y)×C(x,y)×S(x,y)#(18)

[0164]

[0165] c1=(k1l) 2 #(twenty two)

[0166] c2=(k2l) 2 #(twenty three)

[0167]

[0168] In formula (18), L(x, y) is brightness, C(x, y) is contrast, S(x, y) is structure, x represents the original image, y represents the enhanced image, μ x and μ y is expressed as the mean of x and y, and Expressed as the variance of x and y, σ xy It is expressed as the covariance between x and y, c1, c2, c3 are very small constants used to avoid the denominator being 0, k1 is 0.01, k2 is 0.03, and l is the image pixel value range, that is, l∈(0,255);

[0169] (3) Visual information fidelity VIF. The formula for visual information fidelity is:

[0170] Figure 5 It is the flow chart of evaluation index VIF. The image information is the common information of input and output images obtained through HVS (distortion-free), namely C and X. The other is distortion information. This method obtains the common information of input and output, namely C and Y, through the degenerate channel.

[0171] Step 6.3: When the objective evaluation indicators meet the requirements, the final underwater image enhancement model based on multi-scale attention hybrid feature fusion is obtained.

[0172] In order to verify the effectiveness of the enhancement of the MAHFF model, this paper uses CYCLEGAN, UGAN, DEEPWAVENET, IBLA, GB, DCP, and UDCP comparison algorithms to test on the UIEB dataset. The evaluation results are shown in Table 1. The MAHFF proposed in this paper is superior to other comparison algorithms in terms of PSNR, SSIM and VIF evaluation indicators.

[0173] Table 1 Comparative experimental results

[0174]

[0175] The above description is only one embodiment of the present invention and does not limit the patent scope of the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An underwater image enhancement method based on multi-scale attention hybrid feature fusion, characterized in that: The specific steps include: Step 1: Acquire an underwater image data set to form a first data set; in the first data set, the underwater images can be taken by a professional underwater camera or collected from the Internet; Step 2: preprocessing the images in the first data set to form a second data set, and dividing the second data set into a training set, a validation set, and a test set; Step 3: Construct an underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion. The MAHFF model includes 6 MAH-FEB modules, 6 RESIZSE BLOCK modules, 1 Concatenation fusion module, and 1 Conv convolution module; The MAHFF model takes the second data set as input; Input the image of the second data set into the first MAH-FEB module, and the feature map output by the first MAH-FEB module is used as the input of the second MAH-FEB module; at the same time, input the image of the second data set into the first RESIZE BLOCK module, and the output of the first RESIZE BLOCK module is sent to the Concatenation fusion module; The feature map output by the first MAH-FEB module is used as the input of the second RESIZE BLOCK module; the output of the second RESIZE BLOCK module is sent to the Concatenation fusion module; and so on, repeated five times, that is, the output of the fifth MAH-FEB module is sent to the sixth RESIZE BLOCK module, and the output of the sixth RESIZE BLOCK module is sent to the Concatenation fusion module; the output of the sixth MAH-FEB module is sent to the Concatenation fusion module; The output of the Concatenation fusion module is sent to the Conv convolution module, and the output of the Conv convolution module is the final enhanced image; Step 4: Use a new loss function. The loss function calculation formula is: ； In formula (1), L new loss is the new loss function, α is the weight factor, , the L1 calculation formula is: ； In formula (2), N is the total number of samples in each training batch, x(p) is the original image, and y(p) is the enhanced image; In formula (1), L SSIM The calculation formula is: ； ； ； ； ； ； ； In formula (3), L(x,y) is brightness, C(x,y) is contrast, S(x,y) is structure, x represents the original image, and y represents the enhanced image; In formula (4) to formula (9), μ x and μ y is expressed as the mean of x and y, and Expressed as the variance of x and y, σ xy It is expressed as the covariance between x and y, c1, c2, c3 are very small constants used to avoid the denominator being 0, k1 is 0.01, k2 is 0.03, and l is the image pixel value range, that is, ; Step 5: Use the training set and the validation set to train the underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion, and save the model after training as the optimal model. The training process of the MAHFF model further includes steps 5.1 to 5.4: Step 5.1: Set the training parameters of the MAHFF model, including batch size, number of training cycles, learning rate, optimizer, and input image size; Step 5.2: Input the training set and validation set images and corresponding labels into the MAHFF model, use the optimizer Adam to back-propagate and update the parameters, and then start training; Step 5.3: Save the model after training as the optimal model; Step 6: Use the test set to test the optimal model, and evaluate the test set test results by objective evaluation indicators to meet the objective evaluation indicator requirements, that is, to obtain the final underwater image enhancement model MAHFF based on multi-scale attention hybrid feature fusion; the evaluation process further includes steps 6.1 to 6.3: Step 6.1: Input the test set image into the optimal model described in step 5, and output the enhanced test set image; Step 6.2: Calculate the objective evaluation indicators of the original test set images and the enhanced test set images. The objective evaluation indicators include peak signal-to-noise ratio PNSR, structural similarity SSIM and visual information fidelity VIF; Step 6.3: When the objective evaluation indicators meet the requirements, the final underwater image enhancement model based on multi-scale attention hybrid feature fusion is obtained.

2. According to claim 1, the underwater image enhancement method based on multi-scale attention hybrid feature fusion is characterized in that: Each MAH-FEB module in step 3 specifically includes: 4 3×3Conv convolution modules, 1 5×5Conv convolution module, 1 7×7Conv convolution module, 3 GAM attention mechanism modules, 5 BN batch normalization modules, 5 LeakyRelu activation function modules, 1 Resize module, 1 Maxpooling maximum pooling module, and 2 Concatenation fusion modules; The MAH-FEB module, the internal process of the module further includes steps 3.1 to 3.11: Step 3.1: Resize the original input image as a feature map; Step 3.2: Take the feature map as input and send it to the RESIZE BLOCK module, 3×3, 5×5 and 7×7 convolution kernels for convolution operations to extract feature information of different scales and levels. The convolution formula is as follows: ； In formula (10), X out is the feature map after convolution, X in is the input feature map, K is the convolution kernel, [i, j] is the pixel position coordinate, and the feature map X in Multiply with the elements of the convolution kernel K and sum them. m and n are the indices used to traverse all elements of the convolution kernel K. Step 3.3: Perform batch normalization on the output of each convolution branch to reduce the internal covariate shift during training. The batch normalization formula is as follows: ； In formula (11), μ is X out The mean value, σ 2 Yes X out The variance of is a constant that prevents division by zero; Step 3.4: After normalizing each convolution branch, use the LeakyRelu activation function to perform nonlinear transformation on the normalized features to avoid gradient disappearance. The calculation formula of the LeakyRelu activation function is as follows: ； In formula (12), X norm is the feature map after instance normalization, β is the parameter of the LeakyRelu function, indicating that X norm When it is less than 0, the slope of the function; Step 3.5: Send the feature maps output by the three convolution branches of 3×3, 5×5, and 7×7 to the global attention mechanism GAM module respectively to output more detailed feature maps. The global attention mechanism formula is as follows: ； In formula (13), X output is the output feature map, M c is the channel attention mechanism, M s It is the spatial attention mechanism; Among them, the channel attention mechanism M c The formula is: ； In formula (14), X g is the feature map after global pooling, FC is the fully connected layer, and σ is the Sigmoid activation function; Spatial Attention Mechanism M s The formula is: ； In formula (15), X is the input feature map, Conv is the convolutional layer, and σ is the Sigmoid activation function; Step 3.6: The output feature maps of the three GAM layers are fused along the channel dimension and sent to the Concatenation module to obtain a multi-scale feature map. Step 3.7: Perform dimensionality reduction on the concatenated feature map through a 3×3 convolution block; Step 3.8: Use the maximum pooling operation to downsample the reduced feature map. The formula for the maximum pooling is as follows: ; In formula (16), y kij Indicates that the rectangle associated with the kth feature map tends to R ij The maximum pooling output value, x kpq Represents the rectangular area R ij The element at (p,q) in Step 3.9: Send the feature map output from step 3.8 to the Concatenation fusion module; Step 3.10: The feature map of the original image input described in step 3.2 is resized to be consistent with the feature map after downsampling through an upsampling module RESIZEBLOCK, and then fused with the output feature map of step 3.8 to achieve feature fusion of different levels and scales; Step 3.11: Perform a convolution operation on the fused feature map through a 3×3 convolution block and output the final feature map with more detailed features.

3. The underwater image enhancement method based on multi-scale attention hybrid feature fusion according to claim 1, characterized in that: Each RESIZE BLOCK module in step 3 includes a Resize layer and a 3×3 Conv layer; The function of the Resize layer is to adjust the size of the input feature map by upsampling or downsampling to achieve feature extraction at different scales, thereby capturing the global and local information of the image; The function of the Conv layer is to further extract spatial features from the feature map output by the Resize layer through a 3×3 convolution operation, thereby enhancing the expressiveness of the features.

Images