An underwater image enhancement method based on semantic guidance and attention fusion

Abstract

This invention discloses an underwater image enhancement method based on semantic guidance and attention fusion, belonging to the fields of computer vision and machine learning. In underwater image enhancement tasks, this method utilizes semantic cues to improve the consistency of enhancement effects on semantically similar regions, improve image edge blurring, and provide prior knowledge to the network in semantically relevant scenarios that are rare in some types of image degradation, thus improving model performance and generalization ability. The introduction of a feature attention fusion mechanism avoids the loss of contextual information caused by direct fusion of cross-domain information, better combining semantic information with reconstructed features and maximizing the guiding role of semantic information. By combining spatial attention and channel attention, more reasonable weights can be assigned to severely degraded regions, improving the image enhancement effect.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and machine learning technology, specifically relating to an underwater image enhancement method based on semantic guidance and attention fusion. Background Technology

[0002] Underwater image enhancement is a crucial problem in the field of computer vision. Underwater image enhancement technology can correct underwater image degradation problems such as color shift, low contrast, and blurring, restoring the true colors of underwater images and reproducing realistic underwater scenes. Currently, underwater images have wide applications in marine environmental monitoring, underwater robotics, underwater engineering inspection, and search and rescue, promoting understanding of the marine environment and exploration of marine resources. However, limited by factors such as imaging equipment, water quality, and lighting conditions, underwater images obtained by existing imaging equipment often exhibit problems such as blurriness, low contrast, color distortion, and loss of detail, increasing the difficulty of extracting effective information from underwater images. Therefore, underwater image enhancement methods have significant practical significance and value.

[0003] Existing deep learning-based underwater image augmentation networks primarily utilize datasets that can be categorized into synthetic underwater datasets and real-world underwater datasets. For the first approach, the image quality of synthetic underwater datasets differs significantly from that of real underwater scenes, leading to a substantial decrease in the performance of underwater image augmentation networks trained on synthetic datasets in real-world scenarios. For the second approach, manual annotation is time-consuming and labor-intensive, and the selected ground truth is not always realistic; some datasets still exhibit color casts and blurriness in their annotated reference images.

[0004] Based on the above analysis, establishing a more effective learning mechanism is crucial for mining reliable supervised information and improving the model's generalization ability. While existing techniques can obtain more accurate semantic cues in low-level visual tasks by jointly training semantic segmentation and augmentation networks, this joint training approach results in an excessively large number of model parameters and difficulty in convergence, making it unsuitable for underwater image enhancement. Because underwater image degradation is complex, including color cast, low illumination, and blurring, the brightness consistency loss function used in existing low-illumination enhancement algorithms is insufficient to ensure that underwater image enhancement models obtain high-quality enhanced images. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an underwater image enhancement method based on semantic guidance and attention fusion.

[0006] The technical problem addressed by this invention is solved as follows:

[0007] An underwater image enhancement method based on semantic guidance and attention fusion, the specific process of which is as follows:

[0008] An underwater image enhancement network is constructed, including a generator, a semantic segmentation network, a semantic fusion module, and a discriminator; the generator includes an encoder and a decoder, both of which have N layers, where N is a positive integer;

[0009] Semantic segmentation networks are used to acquire semantic maps from input images;

[0010] The encoder is used to extract features from the input image. The encoded features output from the (N+1-i)th layer of the encoder are input to the next layer of the encoder and the i-th layer of the decoder, where 1≤i≤N. The i-th layer of the decoder upsamples the output of the previous layer of the decoder through transposed convolution to obtain the decoded features of the current layer of the decoder. Then, the encoded features output from the (N+1-i)th layer of the encoder and the decoded features of the current layer of the decoder are concatenated to obtain the concatenated features corresponding to the i-th layer of the decoder.

[0011] Each layer of the decoder has a corresponding semantic fusion module. The semantic fusion module is used to semantically fuse the semantic map with the concatenated features to obtain semantic fusion features, which are then output to the corresponding layer of the decoder. The semantic fusion module includes a convolutional layer, a downsampling layer, and a feature attention module connected in sequence. The semantic map is input to the convolutional layer and the downsampling layer connected in sequence to obtain semantically aware features. The semantically aware features and the concatenated features corresponding to the i-th layer of the decoder are input to the feature attention module to obtain semantic fusion features, which are then input to the i-th layer of the decoder.

[0012] The decoder is used to generate reconstructed images using semantic fusion features;

[0013] A discriminator is cascaded after the image enhancement network, and the discriminator adopts the PatchGAN discriminator; the underwater image enhancement network is then trained and validated.

[0014] Loss function L of underwater image enhancement network Total for:

[0015] L Total =λ1L Pix +λ2L Per +λ3L adv (G)

[0016] Among them, L Pix To reconstruct the loss function, L Per For the perceptual loss function, L adv (G) is the adversarial loss function of the generator, where λ1, λ2, and λ3 are respectively L... Pix L Per and L adv The weight of (G);

[0017] Real-time underwater images are used as input images and fed into an underwater image augmentation network that has been trained and validated to obtain augmented images.

[0018] Furthermore, the feature attention module includes an initial feature fusion module, a channel attention mechanism, a spatial attention mechanism, and a sigmoid function;

[0019] The initial feature fusion module is used to process the semantic-aware features F corresponding to the i-th layer of the decoder. i and splicing feature R i The features are added together to output the initial fused feature O. i Initial fusion feature O i The inputs are respectively fed into the channel attention mechanism and the spatial attention mechanism;

[0020] The channel attention mechanism includes a global average pooling layer, a global max pooling layer, a first convolutional layer with a 1×1 kernel, a first ReLU function, and a second convolutional layer with a 1×1 kernel; the global average pooling layer and the global max pooling layer respectively focus on the initial fused features O. i Global average pooling and global max pooling are performed, and the pooling results are summed with features. The summation is then fed into a cascaded first convolutional layer, a first ReLU function, and a second convolutional layer to obtain the channel attention weight matrix W. c (O i );

[0021] The spatial attention mechanism consists of a third convolutional layer with a 1×1 kernel, a second ReLU function, and a fourth convolutional layer with a 1×1 kernel, which are cascaded together to obtain the spatial attention weight matrix W. s (O i );

[0022] The outputs of the channel attention mechanism and the spatial attention mechanism are summed, and then input into the sigmoid function to obtain the semantically aware features F. i Attention weight matrix W(O) i );

[0023] Using the attention weight matrix W(O) i ) semantic perception features F i and splicing feature R i Weighted fusion is performed to obtain semantic fusion features H. i .

[0024] Furthermore, the formula for calculating the channel attention weight matrix is ​​as follows:

[0025]

[0026] Where GAP represents global average pooling, GMP represents global max pooling, conv1 and conv2 represent the first and second convolutional layers, respectively, and δ represents the first ReLU function. This indicates the addition of features;

[0027] The formula for calculating the spatial attention weight matrix is ​​as follows:

[0028] W s (O i )=conv4(δ'(conv3(O i )))

[0029] Where conv3 and conv4 represent the third and fourth convolutional layers, respectively, and δ' represents the second ReLU function;

[0030] Attention weight matrix W(O) i ) is represented as:

[0031]

[0032] Where σ represents the Sigmoid function;

[0033] Semantic fusion feature H i Represented as:

[0034]

[0035] Among them, I H Represents a matrix of all 1s. This represents element-wise multiplication.

[0036] Furthermore, the reconstruction loss function L Pix Using the L2 loss function, it can be expressed as:

[0037]

[0038] Where I represents the reconstructed image output by the generator, I GT Let (x, y) represent the reference image, (x, y) represent the pixel position index, 1 ≤ x ≤ H, 1 ≤ y ≤ W, and H and W represent the image size.

[0039] Perceptual loss function L Per The calculation formula is:

[0040]

[0041] Wherein, φ(I m ) represents the intermediate layer features of the m-th layer of the VGG network, which are input from the reconstructed image output by the generator. Let represent the intermediate layer features of the reference image input to the m-th layer of the VGG network, and let represent the distance function, 1≤m≤M, where M is the number of layers in the VGG network;

[0042] The adversarial loss function uses the LSGAN function.

[0043] Furthermore, the semantic segmentation network adopts SUIM-Net.

[0044] The beneficial effects of this invention are:

[0045] (1) The method described in this invention utilizes semantic cues in underwater image enhancement tasks to improve the consistency of the enhancement effect of the network on semantically similar regions, improve image edge blurring, and provide prior knowledge for the network in some image degradation types that are rare but semantically related, thereby improving model performance and enhancing the model's generalization ability.

[0046] (2) The method described in this invention introduces a feature attention fusion mechanism, which avoids the loss of contextual information caused by direct fusion of cross-domain information, and better combines semantic information with reconstructed features, maximizing the guiding role of semantic information. Considering the uneven degradation of underwater images, by combining spatial attention and channel attention, the network can assign more reasonable weights to severely degraded areas, thereby improving the image enhancement effect. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the underwater image enhancement network in the method described in this invention;

[0048] Figure 2 This is a schematic diagram of the semantic fusion process in the method described in this invention;

[0049] Figure 3 This is a schematic diagram of the feature attention module in the method of the present invention;

[0050] Figure 4 This is a schematic diagram comparing the underwater image enhancement effects of the method described in this invention with those of existing technologies. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0052] This embodiment provides an underwater image enhancement method based on semantic guidance and attention fusion. The specific process is as follows:

[0053] Construct underwater image enhancement networks, such as Figure 1 As shown, the underwater image enhancement network includes a generator, a semantic segmentation network, and a semantic fusion module;

[0054] The generator's overall architecture is a U-Net-like structure, including an encoder and a decoder. The encoder is used to extract features from the input image. Deep networks have a larger receptive field, but the downsampling process inevitably loses some edge features, which is detrimental to image restoration. The decoder is used to reconstruct high-quality images from the image features extracted by the encoder. However, the features lost during downsampling in the encoding process are permanent and cannot be recovered by upsampling. In the U-Net structure, edge features are recovered by concatenating the encoded and decoded features. The method described in this embodiment introduces semantic information after feature concatenation. At this time, the features used for reconstruction contain more texture features, and the introduction of corresponding semantic information guides the image reconstruction process of the decoder.

[0055] A semantic segmentation network is used to obtain the semantic map of the input image, and SUIM-Net is adopted.

[0056] S k The semantic graph information of the k-th instance category is represented by 1≤k≤K, where K is the total number of instance categories. The semantic graph information of all instance categories is connected on the channel to obtain the overall semantic graph S.

[0057] The semantic fusion module is used to semantically fuse the semantic map with the concatenated features to obtain semantically fused features, which are then output to the decoder. The semantic fusion module includes a convolutional layer, a downsampling layer, and a feature attention module connected in sequence. A schematic diagram of the semantic fusion process is shown below. Figure 2 As shown.

[0058] The semantic map is input into a convolutional layer to obtain semantic features. A downsampling layer downsamples the semantic features to adapt to multi-scale reconstruction features, resulting in semantically aware features F. i , is represented as;

[0059] F i =Down(Conv(S))

[0060] Among them, F i Let represent the semantic-aware features corresponding to the i-th layer of the decoder, 1≤i≤N, Conv represents the convolution operation, and Down represents the downsampling operation;

[0061] Semantic perceptual features F i The concatenated feature R corresponding to the i-th layer of the decoder i The input is fed into the feature attention module to obtain the semantic fusion feature H. i It is then input into the i-th layer of the decoder to achieve cross-domain information fusion.

[0062] Feature fusion can be performed directly using common operators such as concatenation, addition, or dot product. However, these operators often implicitly assume that the fused features originate from the same domain or source. Applying these methods directly to cross-domain information may overlook important contextual information. Therefore, the scheme described in this embodiment constructs a novel fusion mechanism by creating a feature attention module to fully utilize various characteristics.

[0063] The feature attention module includes a hybrid attention mechanism that combines channel attention and spatial attention, thereby acquiring more important information in both channel and spatial dimensions and realizing feature fusion based on the hybrid attention mechanism to fully utilize cross-domain features.

[0064] The distribution of features related to underwater image degradation within the feature channels is not uniform. A channel attention mechanism is used to discover features related to image enhancement and assign different attention weights to each channel. In addition, the degree of image degradation is spatially uneven as factors such as water depth, shooting distance, and microbial distribution change. By introducing a spatial attention mechanism, areas with more severe image degradation are identified and assigned more reasonable weights.

[0065] like Figure 3 As shown, the feature attention module includes an initial feature fusion module, a channel attention mechanism, a spatial attention mechanism, and a sigmoid function;

[0066] The initial feature fusion module is used to process semantically aware features F i and splicing feature R i The features are added together to output the initial fused feature O. i Initial fusion feature O i The inputs are respectively fed into the channel attention mechanism and the spatial attention mechanism;

[0067] The channel attention mechanism includes a global average pooling layer, a global max pooling layer, a first convolutional layer with a 1×1 kernel, a first ReLU function, and a second convolutional layer with a 1×1 kernel; the global average pooling layer and the global max pooling layer respectively focus on the initial fused features O. i Global average pooling and global max pooling are performed, and the pooling results are summed with features. The summation is then fed into a cascaded first convolutional layer, a first ReLU function, and a second convolutional layer to obtain the channel attention weight matrix W. c (O i This enables the aggregation of spatial information.

[0068] The formula for calculating the channel attention weight matrix is ​​as follows:

[0069]

[0070] Where GAP represents global average pooling, GMP represents global max pooling, conv1 and conv2 represent the first and second convolutional layers, respectively, and δ represents the first ReLU function. This indicates the addition of features.

[0071] The spatial attention mechanism consists of a third convolutional layer with a 1×1 kernel, a second ReLU function, and a fourth convolutional layer with a 1×1 kernel, which are cascaded together to obtain the spatial attention weight matrix W. s (O i );

[0072] The formula for calculating the spatial attention weight matrix is ​​as follows:

[0073] W s (O i )=conv4(δ'(conv3(O i )))

[0074] Where conv3 and conv4 represent the third and fourth convolutional layers, respectively, and δ' represents the second ReLU function.

[0075] The outputs of the channel attention mechanism and the spatial attention mechanism are summed, and then input into the sigmoid function to obtain the semantically aware features F. i Attention weight matrix W(O) i ):

[0076]

[0077] Where σ represents the Sigmoid function;

[0078] Using the attention weight matrix W(O) i ) semantic perception features F i and splicing feature R i Weighted fusion is performed to obtain semantic fusion features H. i :

[0079]

[0080] Among them, I H Represents a matrix of all 1s. This represents element-wise multiplication.

[0081] Training and validation of the underwater image enhancement network;

[0082] A discriminator is cascaded after the image augmentation network. The discriminator uses a PatchGAN discriminator, which outputs a binary matrix where each element represents a local region in the input image. The discriminator needs to determine the final authenticity based on the authenticity of multiple local regions, making it suitable for underwater image augmentation tasks that focus more on texture details.

[0083] The loss function of the underwater image enhancement network consists of a reconstruction loss function, a perceptual loss function, and an adversarial loss function.

[0084] The L2 loss function is used as the reconstruction loss function to minimize pixel-level error. Pix The calculation formula is:

[0085]

[0086] Where I represents the reconstructed image output by the generator, I GT Let (x, y) represent the reference image, (x, y) represent the pixel position index, 1 ≤ x ≤ H, 1 ≤ y ≤ W, and H and W represent the image size.

[0087] Compared to pixel-level reconstruction loss functions, which focus more on the similarity between pixels, perceptual loss functions place greater emphasis on semantic similarity. Perceptual loss function L Per The calculation formula is:

[0088]

[0089] Wherein, φ(I m ) represents the intermediate layer features of the m-th layer of the VGG network, which are input from the reconstructed image output by the generator. Let represent the intermediate layer features of the reference image input to the m-th layer of the VGG network, and let represent the distance function, 1≤m≤M, where M is the number of layers in the VGG network.

[0090] Use the LSGAN function as the adversarial loss function.

[0091] Loss function L of underwater image enhancement network Total for:

[0092] L Total =λ1L Pix +λ2L Per +λ3L adv (G)

[0093] Among them, L adv (G) is the adversarial loss function of the generator, where λ1, λ2, and λ3 are respectively L... Pix L Per and L adv The weight of (G).

[0094] Real-time underwater images are used as input images and fed into an underwater image enhancement network that has been trained and validated to obtain enhanced images.

[0095] The method described in this embodiment is trained and validated on the classic underwater pairwise datasets SUIM and UIEB. The restoration quality of the underwater image enhancement network is evaluated using full-reference metrics PSNR and SSIM combined with non-reference metrics UIQM and UCIQE. Full-reference metrics analyze the degree of distortion in the output image of the underwater image enhancement network by comparing the differences between the enhanced image and the reference image, thus obtaining a quality assessment of the network's enhancement effect. Non-reference metrics, based on the characteristics of the human visual system, quantitatively evaluate the uneven color difference, blurring, and low contrast of the enhanced image by combining chroma, saturation, and contrast.

[0096] Tables 1 and 2 present the quantitative evaluation results of different methods on the SUIM and UIEB datasets, respectively. A comprehensive analysis of various evaluation metrics shows that the method described in this embodiment generally achieves better results. From the perspective of full-reference metrics, the method described in this embodiment achieves the best results on both the SUIM and UIEB datasets, with PSNR scores improving by 13.1% and 9.1% respectively compared to the second-best method on both datasets. The high score on the Structural Similarity Index (SSIM) also demonstrates the superiority of the underwater image enhancement network described in this embodiment in restoring detailed textures. From the perspective of non-reference metrics, the method described in this embodiment also achieves good results in deep learning-based models, with scores similar to CLIIE-NET. Existing underwater image enhancement methods include white balance adjustments, image contrast enhancement, and color saturation enhancement during the restoration process. This aligns with the image evaluation processes of UIQM and UCIQE, thus leading to better scores on non-reference metrics. However, higher UIQM and UCIQE scores do not necessarily translate to better subjective visual experience.

[0097] Table 1 provides a quantitative comparison of the algorithm with existing underwater image enhancement algorithms on the SUIM dataset.

[0098]

[0099] Table 2 provides a quantitative comparison with existing underwater image enhancement algorithms on the UIEB dataset.

[0100]

[0101] Figure 4This chart compares the visual effects of the method described in this embodiment with different existing underwater image enhancement methods on the SUIM and UIEB datasets. The first column shows the original image, the second column shows the image generated using the CBF algorithm, the third column shows the image generated using the ULAP algorithm, the fourth column shows the image generated using Water-Net, the fifth column shows the image generated using the Ucolor algorithm, the sixth column shows the image generated using CLIIE-NET, the seventh column shows the image generated by the method described in this embodiment, and the eighth column shows a reference image. The CBF and ULAP algorithms, which are not based on deep learning, exhibit poor visual performance. The CBF algorithm introduces some non-existent colors in complex scenes, resulting in unrealistic color representation. The ULAP algorithm introduces too much red component, leading to a red bias in the restored image. This indicates that the underwater environment is complex, and with changes in the environment, the generalization ability and robustness are weak, resulting in poor image restoration effects. Deep learning-based methods generally perform better visually, but the Water-Net-restored image has an overall dark tone. Ucolor has good color representation but cannot effectively improve low contrast and blurriness issues. However, CLUIE-NET suffers from unstable repair performance, resulting in color distortion in some photos and an overall reddish or dark tone. In contrast, the method described in this embodiment provides stable color correction for underwater images with severe color deviations without introducing additional color bias. It also improves image contrast and alleviates the darkening caused by insufficient underwater lighting.

Claims

1. An underwater image enhancement method based on semantic guidance and attention fusion, characterized in that, The specific process is as follows: An underwater image enhancement network is constructed, including a generator, a semantic segmentation network, a semantic fusion module, and a discriminator; the generator includes an encoder and a decoder, both of which have N layers, where N is a positive integer; Semantic segmentation networks are used to acquire semantic maps from input images; The encoder is used to extract features from the input image. The encoded features output from the (N+1-i)th layer of the encoder are input to the next layer of the encoder and the i-th layer of the decoder, respectively, where 1≤i≤N; The i-th layer of the decoder upsamples the output of the previous layer to obtain the decoding features of the current layer of the decoder. Then, the encoding features output by the (N+1-i)-th layer of the encoder and the decoding features of the current layer of the decoder are concatenated to obtain the concatenated features corresponding to the i-th layer of the decoder. Each layer of the decoder has a corresponding semantic fusion module. The semantic fusion module is used to perform semantic fusion with the concatenated features to obtain semantic fusion features and output them to the corresponding layer of the decoder. The semantic fusion module includes a convolutional layer, a downsampling layer and a feature attention module connected in sequence. The semantic graph is input into a convolutional layer and a downsampling layer connected in sequence to obtain semantic-aware features; the semantic-aware features and the concatenated features corresponding to the i-th layer of the decoder are input into the feature attention module to obtain semantic fusion features, which are then input into the i-th layer of the decoder. The decoder is used to generate reconstructed images using semantic fusion features; A discriminator is cascaded after the image enhancement network, and the discriminator adopts the PatchGAN discriminator; the underwater image enhancement network is then trained and validated. Loss function L of underwater image enhancement network Total for: L Total =λ1L Pix +λ2L Per +λ3L adv (G) Among them, L Pix To reconstruct the loss function, L Per For the perceptual loss function, L adv (G) is the adversarial loss function of the generator, where λ1, λ2, and λ3 are respectively L... Pix L Per and L adv The weight of (G); Real-time underwater images are used as input images and fed into an underwater image augmentation network that has been trained and validated to obtain augmented images.

2. The underwater image enhancement method based on semantic guidance and attention fusion according to claim 1, characterized in that, The feature attention module includes an initial feature fusion module, channel attention mechanism and spatial attention mechanism, and a sigmoid function; The initial feature fusion module is used to process the semantic-aware features F corresponding to the i-th layer of the decoder. i and splicing feature R i The features are added together to output the initial fused feature O. i Initial fusion feature O i The inputs are respectively fed into the channel attention mechanism and the spatial attention mechanism; The channel attention mechanism includes a global average pooling layer, a global max pooling layer, a first convolutional layer with a 1×1 kernel, a first ReLU function, and a second convolutional layer with a 1×1 kernel; the global average pooling layer and the global max pooling layer respectively focus on the initial fused features O. i Global average pooling and global max pooling are performed, and the pooling results are summed with features. The summation is then fed into a cascaded first convolutional layer, a first ReLU function, and a second convolutional layer to obtain the channel attention weight matrix W. c (O i ); The spatial attention mechanism consists of a third convolutional layer with a 1×1 kernel, a second ReLU function, and a fourth convolutional layer with a 1×1 kernel, which are cascaded together to obtain the spatial attention weight matrix W. s (O i ); The outputs of the channel attention mechanism and the spatial attention mechanism are summed, and then input into the sigmoid function to obtain the semantically aware features F. i Attention weight matrix W(O) i ); Using the attention weight matrix W(O) i ) semantic perception features F i and splicing feature R i Weighted fusion is performed to obtain semantic fusion features H. i .

3. The underwater image enhancement method based on semantic guidance and attention fusion according to claim 2, characterized in that, The formula for calculating the channel attention weight matrix is ​​as follows: Where GAP represents global average pooling, GMP represents global max pooling, conv1 and conv2 represent the first and second convolutional layers, respectively, and δ represents the first ReLU function. This indicates the addition of features; The formula for calculating the spatial attention weight matrix is ​​as follows: W s (THE i )=conv4(δ'(conv3(O i ))) Where conv3 and conv4 represent the third and fourth convolutional layers, respectively, and δ' represents the second ReLU function; Attention weight matrix W(O) i ) is represented as: Where σ represents the Sigmoid function; Semantic fusion feature H i Represented as: Among them, I H Represents a matrix of all 1s. This represents element-wise multiplication.

4. The underwater image enhancement method based on semantic guidance and attention fusion according to claim 1, characterized in that, Reconstruction loss function L Pix Using the L2 loss function, it can be expressed as: Where I represents the reconstructed image output by the generator, I GT Let (x, y) represent the reference image, (x, y) represent the pixel position index, 1 ≤ x ≤ H, 1 ≤ y ≤ W, and H and W represent the image size. Perceptual loss function L Per The calculation formula is: Wherein, φ(I m ) represents the intermediate layer features of the m-th layer of the VGG network, which are input from the reconstructed image output by the generator. Let represent the intermediate layer features of the reference image input to the m-th layer of the VGG network, and let represent the distance function, 1≤m≤M, where M is the number of layers in the VGG network; The adversarial loss function uses the LSGAN function.

5. The underwater image enhancement method based on semantic guidance and attention fusion according to claim 1, characterized in that, The semantic segmentation network used is SUIM-Net.

Images