An underwater image enhancement method based on conditional convolution and channel attention mechanism

Abstract

The application discloses an underwater image enhancement method based on a conditional convolution and a channel attention mechanism, relates to the technical field of underwater image enhancement, and discloses an underwater image enhancement model used by the method, which comprises an encoder, a decoder, a bottleneck layer and a skip connection, wherein the encoder, the decoder and the bottleneck layer all adopt a residual SE conditional convolution module to extract feature information, the model uses a convolution downsampling layer and a deconvolution upsampling layer to respectively realize downsampling and upsampling of a feature map, the skip connection comprises a splicing operation and a conditional convolution layer, and the conditional convolution layer of the skip connection is used for feature dimension reduction and feature fusion of spliced feature maps between the encoder and the decoder. The application can enhance an underwater input image, and obtain an underwater enhanced result image with high image quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater image enhancement, in particular to an underwater image enhancement method based on conditional convolution and channel attention mechanism. BACKGROUND

[0002] At present, global land resources are scarce, and the population is growing rapidly, while underwater resources are abundant and diverse. Therefore, the research and development of underwater resources are rising worldwide, and the acquisition and processing of image information have become the first step in the development of marine resources. Under this background, underwater image processing technology has received more and more attention. Underwater image processing technology plays an important role in improving the quality of underwater images and provides important help for people to perform subsequent underwater visual tasks.

[0003] The existing underwater image processing technology still has unsatisfactory processing effect on underwater images, and the obtained underwater images still have some visual quality problems, such as image blur, color distortion, artifacts, etc.

[0004] At the same time, the traditional static convolution is a set of convolution kernels that are invariant to underwater input images, and cannot effectively dynamically cope with the diversity and complexity of underwater degradation conditions. Neural networks also cannot select feature map channels with important information. Therefore, neural network methods need to be further improved in these aspects. SUMMARY

[0005] Therefore, the purpose of the present application is to provide an underwater image enhancement method based on conditional convolution and channel attention mechanism, so as to obtain underwater images with high image quality. This method aims to solve the problem that the traditional static convolution cannot dynamically process underwater input images due to a fixed set of convolution kernels, and the problem that neural networks cannot filter important channel information, as well as the technical problem of low image quality in the existing underwater image processing technology.

[0006] To this end, the present application provides the following technical solutions:

[0007] The present application provides an underwater image enhancement method based on conditional convolution and channel attention mechanism, comprising the following steps:

[0008] Step 1, obtain distorted underwater images and their corresponding underwater reference images to form a training set and a test set;

[0009] Step 2, the underwater image enhancement model based on conditional convolution and channel attention mechanism is constructed; the underwater image enhancement model takes the distorted underwater image as input and outputs the enhanced high-quality underwater image, the underwater image enhancement model comprises an encoder, a decoder, a bottleneck layer and a skip connection for fusing two same-scale feature maps between the encoder and the decoder; the encoder comprises a residual SE conditional convolution module and a convolution downsampling layer, the convolution downsampling layer is used to realize the downsampling of the feature map; the decoder comprises a residual SE conditional convolution module and a deconvolution upsampling layer, the deconvolution upsampling layer is used to realize the upsampling of the feature map; the skip connection comprises a splicing operation and a conditional convolution layer, the feature dimension reduction and feature fusion of the spliced feature map between the encoder and the decoder are realized by using the conditional convolution layer; the bottleneck layer comprises a residual SE conditional convolution module; the residual SE conditional convolution modules in the encoder, the decoder and the bottleneck layer are used as basic processing units to process feature information; the residual SE conditional convolution module comprises a channel attention mechanism-based conditional squeeze-excitation module, a conditional convolution layer, a ReLU function and a residual connection;

[0010] Step 3, input the training set image into the constructed underwater image enhancement model for training;

[0011] Step 4, input the test set image into the trained underwater image enhancement model, and output the underwater enhancement result image processed by the model.

[0012] Advantages and positive effects of the present application:

[0013] Firstly, the conditional convolution layer adopted generates a different set of convolution kernels for underwater images in different degradation conditions, thereby realizing dynamic processing of underwater images and adapting to underwater degradation conditions with diversity and complexity; secondly, the channel attention mechanism-based conditional squeeze-excitation (SE) module can emphasize the information of important channels and suppress the information of redundant channels, and has a feature map channel-level screening capability; finally, the conditional convolution module, the conditional squeeze-excitation module and the residual connection are integrated to form a residual SE conditional convolution module; the residual SE conditional convolution module serves as a basic processing unit of the method, so that the method can effectively process feature information and finally obtain underwater images with good image quality; the underwater image enhancement technology in the present application will also promote the development of subsequent underwater visual tasks, such as underwater archaeology, underwater rescue, underwater resource exploration and other underwater application fields. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of the underwater image enhancement method based on conditional convolution and channel attention mechanism in the embodiment of the present application;

[0015] Figure 2 This is a model structure diagram of the underwater image enhancement method based on conditional convolution and channel attention mechanism in an embodiment of the present invention;

[0016] Figure 3 This is a schematic diagram of the skip connection structure in an embodiment of the present invention;

[0017] Figure 4 This is a schematic diagram of the structure of the conditional convolutional layer in an embodiment of the present invention;

[0018] Figure 5 This is a schematic diagram of the structure of the conditional squeeze-excitation module in an embodiment of the present invention;

[0019] Figure 6 This is a schematic diagram of the residual SE conditional convolution module in an embodiment of the present invention. Detailed Implementation

[0020] This invention addresses the color cast and blurring issues in underwater images by proposing an underwater image enhancement method based on conditional convolution and channel attention mechanisms. This method can be applied to tasks such as underwater archaeology, underwater target detection and recognition, and underwater resource exploration.

[0021] like Figure 1 As shown in the figure, an underwater image enhancement method based on conditional convolution and channel attention mechanism in an embodiment of the present invention includes the following specific steps:

[0022] Step 1: Obtain distorted underwater images and their corresponding underwater reference images to form training and testing sets;

[0023] Step 2: Construct an underwater image enhancement model based on conditional convolution and channel attention mechanisms; such as... Figure 2 As shown, the network model includes an encoder, a decoder, and a bottleneck layer, as well as skip connections used to connect the feature maps of the encoder and decoder; the encoder includes a residual SE conditional convolution module and a kernel size of [missing information]. The decoder includes a convolutional downsampling layer with a stride of 2; the decoder includes a residual SE conditional convolution module and a kernel size of [missing information]. The deconvolutional upsampling layer has a stride of 2; the bottleneck layer includes a residual SE conditional convolution module; the skip connection includes a concatenation operation and a kernel size of The conditional convolutional layer has a stride of 1; the residual SE conditional convolutional module includes a conditional squeeze and excitation (SE) module based on channel attention mechanism, with a kernel size of [missing information]. Conditional convolutional layers with a stride of 1, ReLU function, and residual connections;

[0024] Specifically, the network structure of the underwater image enhancement model includes:

[0025] The overall structure of the underwater image enhancement model consists of 5 layers, including an encoder, a bottleneck layer, a decoder, and skip connections.

[0026] The encoder includes four downsampling stages, each of which includes a kernel of size [missing information]. A convolutional downsampling layer with a stride of 2 and two residual SE conditional convolutional modules;

[0027] The decoder includes four upsampling stages, each upsampling stage comprising a kernel of size [missing information]. A deconvolutional upsampling layer with a stride of 2 and a residual SE conditional convolution module;

[0028] The bottleneck layer includes two residual SE conditional convolutional modules;

[0029] There are four hop connections in total, each hop connection includes a concatenation operation and a kernel size of [missing information]. A conditional convolutional layer with a stride of 1 is used, and the skip connections are used to fuse the decoder's upsampled feature map and the feature map from the encoder at the same scale. For example... Figure 3 As shown, the skip connection includes a splicing operation and a kernel size of The conditional convolutional layer has a stride of 1. In the skip connections, the feature map from the encoder and the upsampled feature map from the decoder at the same scale are first concatenated to obtain a concatenated feature map. Then, a kernel size of [missing information] is used... The conditional convolutional layer with a stride of 1 performs feature fusion and feature dimensionality reduction on the concatenated feature map, resulting in a fused feature map with the same feature dimension as the upsampled feature map.

[0030] The conditional convolutional layer includes a routing function and multiple experts (sub-convolutional kernels), such as Figure 4 As shown; the conditional convolutional layer can generate a corresponding convolutional kernel based on each specific underwater input image, avoiding the problem of traditional static convolutional kernels treating all samples equally, and effectively improving the network's dynamic processing capability;

[0031] The routing function consists of global average pooling, a sigmoid activation function, and a learned routing weight matrix; the specific calculation formula for the routing function is as follows:

[0032] ;

[0033] in, It is the input data. Indicates global average pooling. For activation function, It is a learnable weight matrix;

[0034] In the conditional convolutional layer, multiple weights generated by a routing function are used to weightedly fuse multiple sub-convolutional kernels (experts) to obtain the final convolutional kernel, which is then convolved with the input data to obtain the output; the convolutional kernel of the conditional convolutional layer is parameterized as follows:

[0035] ;

[0036] in, It's a convolution operation. Represents routing function The weights corresponding to the generated expert (sub-convolution kernel), For experts;

[0037] like Figure 5 As shown, the Squeeze and Excitation (SE) module based on the channel attention mechanism includes a squeezing operation, an excitation operation, and channel vector multiplication. The SE module can filter out information from important channels at the channel level while suppressing information from redundant channels, thereby effectively increasing the network's representational power. The squeezing operation of the SE module can be expressed as:

[0038] ;

[0039] in, These are the channels of the feature map. For input data, and These are the height and width of the feature map channels. It is the global average of the input data;

[0040] The activation operation is used to generate channel-level weights, calculated according to the following formula:

[0041] ;

[0042] in, It is the ReLU function. This represents the Sigmoid function. and Each of the two kernels has a size of [size missing]. A conditional convolutional layer with a stride of 1, set Compression ratio It is 16, which is The number of output channels is ;

[0043] Finally, the channel vector multiplication utilizes weights For the initial input Perform weight correction:

[0044] ;

[0045] in, It is a multiplication operation between channels and vectors. This indicates the output of the conditional squeeze-excitation module;

[0046] The residual SE conditional convolution module includes a conditional squeezing-activation module and two kernels with a size of [missing information]. A conditional convolutional layer with a stride of 1, a ReLU function, and a residual connection, such as Figure 6 As shown; in the residual SE conditional convolution module, the input data sequentially through a kernel of size A conditional convolutional layer with a stride of 1, a ReLU function, and a kernel size of [missing information]. The result is obtained by using a conditional convolutional layer with a stride of 1 and a conditional squeeze-excitation module. , Using the identity path and the initial input Add them together to get the final output result;

[0047] Step 3: Input the training set images into the constructed underwater image enhancement model for training;

[0048] Regarding network loss, this embodiment of the invention employs an L1 loss function and a perceptual loss function:

[0049] The L1 loss function is: ;

[0050] in, , These are a clear underwater reference image and the underwater output image of the underwater image enhancement model, respectively. This represents the total number of pixels in the image. For the pixel index of the image, This represents the total number of color channels in the image.

[0051] The perceptual loss function is: ;

[0052] in, For the VGG network layer, , and These are the number of channels, height, and width of the feature map, respectively.

[0053] The overall loss function is: ;

[0054] in, The underwater image enhancement model is set to use weight balancing coefficients. .

[0055] Step 4: Input the test set images into the trained underwater image enhancement model, and output high-quality underwater images after image enhancement;

[0056] The underwater image enhancement model in step 2 specifically processes the image, including the following steps:

[0057] Step 2.1: Obtain the input image I, and process the input image through a single kernel with a size of [missing information]. A conditional convolutional layer with a stride of 1 projects a feature map of arbitrary dimensions.

[0058] Step 2.2: The feature map of arbitrary dimensions is then input into the first layer of the encoder, which contains two residual SE conditional convolutional modules and one kernel with a size of [missing information]. A convolutional downsampling layer with a stride of 2; the residual SE conditional convolution module is used to learn feature information to obtain the first layer processed feature map of the encoder. The convolutional downsampling layer then modifies the feature map. By performing downsampling and feature dimension enhancement, a downsampled feature map is obtained. ;

[0059] Step 2.3, repeating step 2.2 three times, yields the second-layer processing feature map of the encoder. The third layer processes feature maps. and the fourth layer of feature map processing and downsampling feature maps of each layer , and ;

[0060] Step 2.4: Extract the feature map with the smallest spatial scale. The input is fed into a bottleneck layer, which includes two residual SE conditional convolutional modules; the bottleneck layer further learns feature information to obtain a feature map. ;

[0061] Step 2.5: Process the feature map Feature map expansion yields upsampled image and the same-scale feature map from the encoder. The concatenation process yields a concatenated feature map, which is then processed using a single kernel with a size of [missing information]. A conditional convolutional layer with a stride of 1 is used to process and concatenate the feature maps, resulting in a fused feature map. The fourth layer of the decoder uses a residual SE conditional convolutional module to extract feature maps. The feature information is used to obtain the fourth layer of processed feature map. ;

[0062] Step 2.6: Process the feature map Feature map expansion yields upsampled image and the same-scale feature map from the encoder. The concatenation process yields a concatenated feature map, which is then processed using a single kernel with a size of [missing information]. A conditional convolutional layer with a stride of 1 is used to process and concatenate the feature maps, resulting in a fused feature map. The feature map is extracted by a residual SE conditional convolutional module in the third layer of the decoder. The feature information is used to obtain the third-layer processing feature map. ;

[0063] Step 2.7 and step 2.6 are repeated twice to obtain the second-layer processing feature map of the decoder. The first layer processes the feature map. Feature maps after fusion Feature maps after fusion Upsampled feature map and upsampled feature map ;

[0064] Step 2.8: Process the feature map Through a kernel of size 1 After convolution processing with a conditional convolutional layer with a stride of 1, the final output image is obtained.

[0065] Finally, it should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. An underwater image enhancement method based on conditional convolution and channel attention mechanism, characterized in that, Includes the following steps: Step 1: Obtain distorted underwater images and their corresponding underwater reference images to form training and testing sets; Step 2: Construct an underwater image enhancement model based on conditional convolution and channel attention mechanisms. The underwater image enhancement model takes a distorted underwater image as input and outputs a high-quality enhanced underwater image. The model includes an encoder, a decoder, a bottleneck layer, and skip connections for fusing two feature maps of the same scale between the encoder and decoder. The encoder includes a residual SE conditional convolution module and a convolutional downsampling layer, which is used to downsample the feature map. The decoder includes a residual SE conditional convolution module and a deconvolutional upsampling layer, which is used to upsample the feature map. The skip connections include a concatenation operation and a conditional convolution layer, using the conditional convolution layer to achieve feature dimensionality reduction and feature fusion of the concatenated feature maps between the encoder and decoder. The bottleneck layer includes a residual SE conditional convolution module. The residual SE conditional convolution module in the encoder, decoder, and bottleneck layer all serve as basic processing units to process feature information. The residual SE conditional convolution module includes a conditional squeeze-activation module based on channel attention mechanism, a conditional convolutional layer, a ReLU function, and residual connections; Step 3: Input the training set images into the constructed underwater image enhancement model for training; Step 4: Input the test set images into the trained underwater image enhancement model, and output the underwater enhancement result image after model processing; In the conditional compression-activation module based on the channel attention mechanism, the input feature map first undergoes a global average pooling operation to obtain global statistics, and then passes through a compression ratio of 16 and a stride of 1. Conditional convolutional layer, a ReLU function, and a stride of 1 The conditional convolutional layer maps a set of weights for each channel, and then uses these weights to correct the input feature map, resulting in the final output of the conditional squeeze-excitation module.

2. The underwater image enhancement method based on conditional convolution and channel attention mechanism according to claim 1, characterized in that, The network structure of the underwater image enhancement model specifically includes: The overall structure of the underwater image enhancement model consists of 5 layers, including an encoder, a bottleneck layer, a decoder, and skip connections. The encoder includes four downsampling stages, each of which includes a kernel of size [missing information]. A convolutional downsampling layer with a stride of 2 and two residual SE conditional convolutional modules; The decoder includes four upsampling stages, each upsampling stage comprising a kernel of size [missing information]. A deconvolutional upsampling layer with a stride of 2 and a residual SE conditional convolution module; The bottleneck layer includes two residual SE conditional convolutional modules; There are four hop connections in total, each hop connection includes a concatenation operation and a kernel size of [missing information]. A conditional convolutional layer with a stride of 1, wherein the skip connections are used to fuse the decoder upsampled feature map and the feature map from the encoder at the same scale.

3. The underwater image enhancement method based on conditional convolution and channel attention mechanism according to claim 2, characterized in that, In the conditional convolutional layer, the input data is first processed by a routing function to generate weights for each expert. Then, the weights are used to perform weighted fusion on all experts to obtain the final convolutional kernel, which is then convolved with the input data to obtain the result.

4. The underwater image enhancement method based on conditional convolution and channel attention mechanism according to claim 3, characterized in that, In the residual SE conditional convolution module, the input feature map first passes through a single kernel of size 1. A conditional convolutional layer with a stride of 1, a ReLU function, and a single kernel of size 1. The calculation of a conditional convolutional layer with a stride of 1 and a conditional squeeze-excitation module yields an intermediate result. This intermediate result is then added to the initial input feature map through a residual connection to obtain the final output result of the residual SE conditional convolutional module.

5. The underwater image enhancement method based on conditional convolution and channel attention mechanism according to claim 4, characterized in that, The underwater image enhancement model uses two loss functions during training: L1 loss function and perceptual loss function. The L1 loss function is: ; in, , These are a clear underwater reference image and the underwater output image of the underwater image enhancement model, respectively. The total number of pixels in the image. For the pixel index of the image, This represents the total number of color channels in the image. The perceptual loss function is: ; in, For the VGG network layer, , and These are the number of channels, height, and width of the feature map, respectively. The overall loss function is: ; in, The underwater image enhancement model is set to use weight balancing coefficients. .

6. The underwater image enhancement method based on conditional convolution and channel attention mechanism according to claim 2, characterized in that, Step 2, the underwater image enhancement model, specifically processes the image, including the following steps: Step 2.1: Obtain the input image I, and process the input image through a single kernel with a size of [missing information]. A conditional convolutional layer with a stride of 1 projects a feature map of arbitrary dimensions. Step 2.2: The feature map of arbitrary dimensions is then input into the first layer of the encoder, which contains two residual SE conditional convolutional modules and one kernel with a size of [missing information]. A convolutional downsampling layer with a stride of 2; the residual SE conditional convolution module is used to learn feature information to obtain the first layer processed feature map of the encoder. The convolutional downsampling layer then modifies the feature map. By performing downsampling and feature dimension enhancement, a downsampled feature map is obtained. ; Step 2.3, repeating step 2.2 three times, yields the second-layer processing feature map of the encoder. The third layer processes feature maps. and the fourth layer of feature map processing and downsampling feature maps of each layer , and ; Step 2.4: Extract the feature map with the smallest spatial scale. The input is fed into a bottleneck layer, which includes two residual SE conditional convolutional modules; the bottleneck layer further learns feature information to obtain a feature map. ; Step 2.5: Process the feature map Feature map expansion yields upsampled image and the same-scale feature map from the encoder. The concatenation process yields a concatenated feature map, which is then processed using a single kernel with a size of [missing information]. A conditional convolutional layer with a stride of 1 is used to process and concatenate the feature maps, resulting in a fused feature map. ; The fourth layer of the decoder uses a residual SE conditional convolutional module to extract feature maps. The feature information is used to obtain the fourth layer of processed feature map. ; Step 2.6: Process the feature map Feature map expansion yields upsampled image and the same-scale feature map from the encoder. The concatenation process yields a concatenated feature map, which is then processed using a single kernel with a size of [missing information]. A conditional convolutional layer with a stride of 1 is used to process and concatenate the feature maps, resulting in a fused feature map. The feature map is extracted by a residual SE conditional convolutional module in the third layer of the decoder. The feature information is used to obtain the third-layer processing feature map. ; Step 2.7 and step 2.6 are repeated twice to obtain the second-layer processing feature map of the decoder. The first layer processes the feature map. Feature map after fusion Feature map after fusion Upsampled feature map and upsampled feature map ; Step 2.8: Process the feature map Through a kernel of size 1 After convolution processing with a conditional convolutional layer with a stride of 1, the final output image is obtained.

Images